SULU 56a

Summary

* Added a new version of the aria2c site mover (code from Rodney Walker)

* Added a new version of the JEM module that fixes a problem when running
with multi-trfs (code from Frank Volkmer)

* Now setting permission 0770 explicitly in mkdir call for job recovery
directories external to the WN, avoiding world readable directories (on
non-US sites)

* Added recognition for SLURM batch system (requested by Andrej Filipcic)

* Created new error code 1215, Failed to open TCP connection to localhost
(worker node network problem). Now aborting pilot if TCP connection could
not be opened

* Created new error code 1207, Job killed by signal: SIGUSR1 (note: pilot
wrappers need to be updated for proper signal handling, work in progress)

* Refactored stage-out method in preparation for stage-out to alternative
SE (which currently is in testing); number of normally transferred files
vs number of files transferred to alternative SE is reported with
filesNormalStageOut and filesAltStageOut in job metrics (will be sent with
a later pilot version when everything is ready)

* Updated xrdcp site mover for new CERN setup (previously a complicated
xrdcp setup had to be used which had to be hardcoded in the pilot, now
using a simplified setup from schedconfig) (requested by Alessandro Di
Girolamo)

* Added -f overwrite option for xrdcp stage-out (requested by Alessandro
Di Girolamo/Guidone Negri)

* Added support for queuedata overwrite via job parameters; syntax:
--overwriteQueuedata={field=value,...} [used by HC] NOTE: The corresponding prun option is called --queueData. It will be added to pathena as soon as possible

* Added FAX transfer counters; number of successful FAX transfers reported
as filesWithFAX in job metrics, number of normal transfers [when at least
one FAX transfer] are reported with filesWithoutFAX

* Now possible to switch off FAX with --disableFAX in job parameters [used
by HC; note: could also be achieved with
--overwriteQueuedata={allowfax=false}. --disableFAX has precedence over
--overwriteQueuedata]

* Added direct i/o support to FAX site mover to be used when FAX is used
as primary site mover (requested by Johannes Elmsheuser)


A later pilot (to be released as soon as possible) will address FAX usage
by runGen/runAthena in direct i/o mode.

The EGI pilot wrapper has been updated by Peter Love to launch the Release
Candidate pilot with python 2.6 as well as setting up DQ2Client 2.3.0 on
CVMFS sites (the production version of the pilot is still run by python <
2.6 and with an older DQ2Client ­ which is not FAX compatible). The US
pilot wrapper update is in progress (by John Hover and Jose Caballero).
 
Version info

Corrected missing experiment argument in FAX get_data() call in mover_get_data() (Mover)
Added a new version of JEMstub that fixes a problem when running with multi-trfs (JEMstub)
Removed wrong useExtension option in getQueuedataFileName() call (atlasProdPilot)
Now setting permission 0770 explicitly in mkdir call in moveToExternal() (pilot) [Alert Fernando/Ramon]
Corrected wrong variable name in getXMLAndWorkdir() (JobLog)
Created getTrfExitInfo() and extractDictionaryObject() (runJob)
Simplified sitemover_get_all_data() and it's call in mover_get_data() (Mover)
Removed unnecessary _id variable from getBatchSystemJobID() (pUtil)
Added recognition for SLURM batch system in getBatchSystemJobID() (pUtil)
Added FAX transfer counters to mover_get_data() (Mover)
mover_get_data() now returns number of FAX transfers to get_data() (Mover)
get_data() now receives and returns number of FAX transfers (Mover)
stageIn() now receives and sets number of FAX transfers (runJob)
Added data members filesWithoutFAX and filesWithFAX to Job class (Job)
Added filesWithoutFAX and filesWithFAX to msgdic in updateJobInfo() (RunJobUtilities)
Now receiving the filesWithFAX and filesWithoutFAX from the pilot server in updateHandler::handle() (pilot) [Alert Fernando/Ramon]
Created extractQueuedataOverwrite() (SiteInformation)
Created updateQueuedataFromJobParameters() (SiteInformation)
Added si object to getJob() and its call in runmain() (pilot) [Alert Fernando/Ramon]
getJob() now calls updateQueuedataFromJobParameters() if queuedata overwrite command found in job parameters (pilot) [Alert Fernando/Ramon]
Now possible to switch off FAX with --disableFAX in job parameters, in updateQueuedataFromJobParameters() (SiteInformation)
Added protoype for getTier1Queue() (SiteInformation)
Created new error code 1215, Failed to open TCP connection to localhost (worker node network problem) (PilotErrors)
Now aborting pilot if TCP connection could not be opened (pilot) [Alert Fernando/Ramon]
Added alt option for readpar() definition (pUtil)
Corrected resetting of PilotHomeDir in getQueuedata() (SiteInformation)
Created sitemover_put_data() (Mover)
mover_put_data() now keeps track of number of normal vs alt staged out files (Mover)
Added handling for N_filesNormalStageOut, N_filesAltStageOut to transferLogFile(), transferAdditionalFile() (JobLog)
Added handling for job.filesNormalStageOut, job.filesAltStageOut to moveLostOutputFiles() [not saved in recovery mode] (pilot) [Alert Fernando/Ramon]
Added handling for job.filesNormalStageOut, job.filesAltStageOut to stageOut() (runJob)
Added filesNormalStageOut, filesAltStageOut to Job class (Job)
Added handling for job.filesNormalStageOut, job.filesAltStageOut to getJobMetrics() (PandaServerClient)
Added handling for job.filesNormalStageOut, job.filesAltStageOut to updateJobInfo() (RunJobUtilities)
Added handling for job.filesNormalStageOut, job.filesAltStageOut to handle() (pilot) [Alert Fernando/Ramon]
Now sending alt option to put_data() from sitemover_put_data() (Mover)
Receiving alt option in put_data() (lcgcp2SiteMover)
Sending alt option from put_data() to getProperPaths() (lcgcp2SiteMover)
Receiving alt option in getProperPaths() (ATLASSiteInformation)
Sending alt option from getProperPaths() to getPreDestination() (ATLASSiteInformation)
Added alt option to getPreDestination() (ATLASSiteInformation)
Using alt option in readpar() calls in getPreDestination() (ATLASSiteInformation)
Sending alt option to getMatchingDestinationPath() from getPreDestination() (ATLASSiteInformation)
Added alt option to getMatchingDestinationPath() (SiteMover)
Using alt option in readpar() call in getMatchingDestinationPath() (SiteMover)
Sending alt option to getLFCPath() from getProperPaths() (ATLASSiteInformation)
Receiving alt option in getLFCPath() (SiteMover)
Sending alt option to getProperSE() in getProperPaths() (ATLASSiteInformation)
Receiving and using alt option in getProperSE() (SiteMover)
Sending alt option to getEnvsetup() from put_data() (lcgcp2SiteMover)
Receiving and using alt option in getEnvsetup() (SiteMover)
Removed useless cache variable in getCPUmodel() (pUtil)
Corrected wrong variable, path -> appdir, in verifySwbase() (pUtil)
Renamed _token to tokenList in getSpaceTokenList(), mover_put_data() (Mover)
Created alternative space token list in mover_put_data() (Mover)
Sending alt option to getSpaceTokenList() from mover_put_data() (Mover)
Added alt option to getSpaceTokenList() (Mover)
Sending alt option from getSpaceTokenList() to getProperSpaceTokenList() (Mover)
Added alt option to getProperSpaceTokenList() (Mover)
Created prepareAlternativeStageOut(), used from mover_put_data() (Mover)
Added new error code 1207, ERR_SIGUSR1 (PilotErrors)
Using new error code ERR_SIGUSR1 in signal handlers (pilot, runJob) [Alert Fernando/Ramon]
Now using getSetup() properly in get_data(), put_data() (xrdcpSiteMover)
Removed the forced setting of xrdcp setup in mover_get_data(), mover_put_data() (Mover)
Added direct i/o checks in get_data() (FAXSiteMover)
Now allowing FAX for values "true" or "retry" in mover_get_data() (Mover)
Created checkForDirectAccess() (SiteMover)
get_data() is now using checkForDirectAccess() (xrdcpSiteMover, FAXSiteMover)
Now sending si from getJob() to getNewJob() (pilot) [Alert Fernando/Ramon]
Moved updateQueuedataFromJobParameters() call from getJob() to getNewJob() (pilot) [Alert Fernando/Ramon]
Using copyprefixin before copyprefix in getLFCFileList() (Mover)
Added -f overwrite option for xrdcp in put_data() (xrdcpSiteMover)
Simplied findGlobalFilePath() (FAXSiteMover)
Created getGlobalPathsFileName() used by getGlobalFilePaths() (FAXSiteMover)
Added new version of aria2c site mover from Rodney Walker