SULU 44a

Summary

* runAthena/Athena setup has been revised for user analysis jobs after it was discovered that old style setup was incompatible with new CMT version v1r21. The problem resulted in user jobs using a patched release (15.6.9.4) to wrongly use the normal release (15.6.9). Discovered by Andrea Bocci et al.

* glExec has been integrated with the pilot. Testing through PanDA will commence shortly. Thanks to Jose Caballero.

* Limit of user workDir has been increased from 5 GB to 7 GB. Requested by Kaushik De, Rodney Walker.

* prun stdout[err] is now redirected to prun_stdout[err].txt. Previously it was redirected to athena_stdout[err].txt.

* Revamped copysetup[in] handling in pilot related to file stager. New format for copysetup[in] is:
copysetup = setup_string^oldPrefix^newPrefix^useFileStager^directIn
where:
setup_string = standard copy setup string, same as before
oldPrefix = prefix to replace in direct access mode, same as before
newPrefix = prefix for direct access, same as before
useFileStager = True for file stager mode, new Boolean
directIn = True for direct access mode and file stager mode, same as before but now required for file stager mode
TRIUMF queues were updated today in relation to this since useCopyTool was set to True (now replaced with useFileStager).
The tedious setting of old/newPrefix will be replaced by automatic method a la Ganga in one of the next pilot versions. Thanks to Johannes Elmsheuser.

* CPU conversion factor now set to constant 1.0 since method is completely outdated.

* A difference in how exit codes are handled between pyfactory and autopilot prevented multi-jobs from running on EGI resources. Now corrected.

* Now grabbing checksum from lcg-ls output if available and using this for file integrity verification. Defaults to alternative methods and file size check (lcg-cp/cr site movers) if not available. Requested by Wensheng Deng et al.

* Pilot is now querying ToA on file location prior to stage-in to avoid file replica located on tape to be chosen instead of disk replica (relevant when file path does not explicitly contain “tape” string). Feature has been used on CERN-* with special pilot version since problem was discovered there last month. Requested by Simone Campana et al.

* First part of new recovery algorithm has been added. A report of current file states (not created/created/not transferred/transferred/not registered/registered) dumped to pilot log whenever relevant. Other part of new algorithm is not activated yet pending further implementation and testing.

* Old job recovery code has been strengthened with a more secure locking mechanism of job state files that are probed prior to a recovery attempt, to avoid racing condition. It was discovered at MWT2_UC that two pilots started on the same WN within 1 second of each other, recovered the same output file. After it was successfully transferred and registered, the second lsm-put command removed what it believed was a failed transfer. A stronger lock file will prevent this from happening. Thanks to Charles Waldman.

* Time out limit in local site mover has been increased from 1800 s to 5400 s to allow lsm command to use internal time-out mechanism. Requested by Charles Waldman.

* Avoiding sending dummy adler32 checksums to lsm-get. Requested by Pedro Salgado.

* Converting lsm command exit codes using os.WEXITSTATUS(). Requested by Pedro Salgado.

* Added std::bad_alloc to memory error identification. Requested by Rodney Walker.

Version info

Created isFileOnTape(), isTapeSite() (SiteMover)
Replica sorting algorithms (randomizeReplicas(), sortReplicas()) now check ToA if replica is on tape (Mover)
Updated getDirectAccessDic() to handle new copysetup format (pUtil)
Removed useless info text in chdir() (pilot)
Added conversion factor minutes to seconds in getTimeFloor() (pUtil)
Moved isTier3() from SiteMover to pUtil
Updated code for isTier3() move (mvSiteMover, SiteMover, Mover)
Now adding envsetup to special_command_setup on tier-3 site (RunJobUtilities)
Now adding special_command_setup to run command on tier-3 site (RunJobUtilities)
Created global jobIds list, displayed in job report (pilot)
Removed useless debugLevel messages in pilot and runJob
Corrected misuse of find command leading to misinterpretation of athena error output (runJob)
Created getInstallDir(), checkCMTCONFIG(), getProdCmd2(), updateAnalyCmd2(), getSpecialSetupCommand() (RunJobUtilities)
Renamed getCmd2() to getAnalyCmd2() (RunJobUtilities)
Added error handling to getAnalyCmd2(), getProdCmd2() (RunJobUtilities)
Removed useless getTag call for analysis jobs in setupTrf() (RunJobUtilities)
Renamed getCmd3WithReleasePy() to getProdCmd3() (RunJobUtilities)
Setting CERNVM site status by hand in postProcessQueuedata() (pUtil)
Removed obsolete dq2_poolFCJobO call in getPoolFileCatalogDQ2() (Mover)
Created removeTestFiles() (pilot)
Added correction code for test jobState files in RecoverLostJobs() and moveToExternal() (pilot)
Created updateJobStateTest() (pUtil)
Added jobState and fileStateDictionary to Job class (Job)
Added testing code for job state handling (pilot, runJob)
Clarified confusing "job is recoverable" log message (RunJobUtilities)
Moved global dispatcher error codes to local getNewJob() (pilot)
Created createAtomicLockFile(), removeAtomicLockFile() (pilot)
Old job recovery code is now using atomic lock file to avoid possible race-condition (pilot)
Added new job recovery algorithm test function (pilot, runJob)
Changed confusing log message "Final job state set to" to "Final payload state set to" (RunJobUtilities)
Added jobState and fileStateDictionary to updateJobInfo() and class updateHandler (RunJobUtilities, pilot)
Created FileStateClient module
Added workDir as argument to mover_put_data(), needed by updateFileState() (Mover)
stageOut() is now sending workDir to mover_put_data() (runJob)
moveLostOutputFiles() and transferLogFile() are now sending workDir to mover_put_data() (pilot)
Added std::bad_alloc to memory error identification in isOutOfMemory() (RunJobUtilities)
Code cleanup in setJobDef() (Job)
Added file open exception handling in createPoolFileCatalog() (pUtil)
Created backupPFC4Mover(), createPFC4TRF() (Mover)
Old PFC file now backed up before being overwritten by mover_get_data() (Mover)
Began refactoring of mover_get_data() (Mover)
Added prun to list in getPayloadName() (pUtil)
Improved exception handling for ls of direct access files that were not transferred in get_data() (Mover)
setup() is now sending pilot_initdir to setupTrf() (runJob, RunJobUtilities)
setupTrf() is now sending pilot_initdir to setupNordugridTrf() (RunJobUtilities)
setupNordugridTrf() and setupTrf() are now sending pilot_initdir to getAnalysisTrf() (RunJobUtilities)
getAnalysisTrf() is now copying trf from pilot_initdir if it exists, defaults to downloading it (RunJobUtilities)
Set the CPU consumption factor to 1.0 in setTimeConsumed() (pUtil)
Removed extra log message in getTimeFloor() (pUtil)
Avoiding sending dummy adler to lsm-get (LocalSiteMover)
Converting exit code with os.WEXITSTATUS() in check_space(), get_data(), put_data() (LocalSiteMover)
Added atlasProdPilot.py to pilot (temporarily?)
Replaced all fsize_remote with remote_fsize (SiteMover)
Created getRemoteFileInfo(), getLCGLS(), getRemoteChecksumLCGLS(), getRemoteFileSizeLCGLS(), _getRemoteChecksumLCGLS(),
_getRemoteFileSizeLCGLS() (SiteMover)
put_data() now using getRemoteFileInfo() to extract checksum and file size using lcg-ls (lcgcpSiteMover, lcgcp2SiteMover)
Ignoring dummy checksums in _getRemoteChecksumLCGLS() (SiteMover)
Created getSetupFromCopysetup() (RunJobUtilities)
Got rid off useless code in getAnalyCmd3() (RunJobUtilities)
Excluding log file from warning message in verifyOutputFileSizes() (pilot)
Removed a bunch of old out-commented code (RunJobUtilities)
Moved setting of prod tags to getTag() (RunJobUtilities)
Added runtime to tags in getTag() (RunJobUtilities)
Turning off proxy verification for ptest jobs in argParser() (pilot)
Created isSetupCommandOk() (RunJobUtilities)
Now detecting CMT warnings (ignored) in setup command output (RunJobUtilities) [corrected comment in v 44b]
Removed monitor warning from check_syserr() (futil)
Removed monitor warning from check_space_df() (SiteMover)
A difference in how exit codes are handled between pyfactory and autopilot prevented multi-jobs from running
on EGI resources. Fixed with in runMain(), sending wrflag to True in cleanup() call (pilot)
Increased time-out limit from 1800 s to 5400 s (90 minutes) in LocalSiteMover
Removed misleading default get time-out log message in get_data() (Mover)
Reverting swbase to appdir if fallback software dir does not exist (problem seen on one WN at CERN causing
pilots to abort before asking for a job. The WN has afs access problems. Error message is not clearer.) (pilot)

glExec merge:
credname and myproxy added to class definition and to displayJob() (Job)
Added execution code in executePayload() (runJob)
Created executePayloadGLExec() (runJob)
Added module myproxyUtils
Added getProxyKey to jNode dictionary in getNewJob() (pilot)