SULU 46a

Summary

* The pilot will now skip stage-in of DBRelease file if the file is available locally ([$VO_ATLAS_SW_DIR|$OSG_APP]/database/DBRelease; pilot skips stage-in and creates a skeleton setup file used by the trf). It has been successfully tested with standard test jobs, RC test jobs and user analysis jobs. In case of unforeseen complications, the feature can be turned off locally by defining the ATLAS_DBREL_DWNLD environmental variable.

* In preparation for LFC registration by the Panda server, the pilot is now sending SURL metadata to the Panda server after a successful job. LFC registration by the pilot can be turned off with the new pilot option, -B False.

* Looping job limit has been lowered to three hours for user analysis jobs, as discussed during the Napoli workshop.

* The pilot now stores the list of identical replicas. In case one replica is not available for stage-in, the pilot will retry using an alternative replica (maximum three replicas).

* Corrected checksum calculation in the pilot which caused the adler32 to be calculated two times (Savannah ticket 71792).

* In case a TURL based PFC cannot be created for production jobs, the pilot will fail immediately and not attempt copy-to-scratch as it did before.

* Added additional flush calls at the end of the job to prevent stdout batch log to be cut short (seen in the IT cloud). Requested by David Rebatto.

* Pilot is is using JSON files (queuedata, replica dictionary, etc) instead of pickle files when available.

* Added JEM module developed by Tim dos Santos. JEM monitoring can be enabled in user jobs with the --enable-jem pathena/prun option. Initial testing performed at Wuppertalprod. Testing will now commence on more sites. A separate announcement with instructions on how to use JEM will be sent at a later time.

* Several new classes have been added to the pilot related to new job recovery algorithm. Testing will commence in the next couple of weeks on selected US sites.

* Added site exclusion list to prevent rfio from being used if it is not supported in TURL based PFC creation (ANALY_TW-FTT_TEST).

* Pilot is no longer sending countryGroup to server since it is not needed.

Version info

Added path variable (job.workdir) to getFileInfo() call in mover_get_data() (Mover)
Added workdir variable to getFileInfo(), getPoolFileCatalog() (Mover)
Added workdir variable to getPoolFileCatalog() call in getFileInfo() (Mover)
Added workdir variable to getLFCFileList() call in getPoolFileCatalog() (Mover)
Created storeMatchedReplicas() (Mover)
Corrected variable name checksum -> fchecksum in put_data() call in mover_put_data() (Mover)
Restriced xrdcp -DIReadCacheSize option to CERN sites in getSpecialSetupCommand() (RunJobUtilities)
Added allowdirectaccess to shouldPFC4TURLsBeCreated() (Mover)
In case TURL based PFC cannot be created for prod jobs, pilot will fail immediately and not attempt copy-to-scratch (Mover)
Added flush() calls at the end of cleanup() and fastCleanup() (pilot)
Added MB control variable to getMaxInputSize() (pUtil)
getNewJob() is now sending min(space, maxInputSize) to dispatcher and is using getMaxInputSize() (pilot)
Updated get_RETRY handling in mover_get_data(), now using the min of pilot option retry variable and max allowed (3) (Mover)
alt. if number of retries is set by replica dictionary, the min of the replica list and max allowed (3) (Mover)
Updated verifyQueuedata(), new version requires queuedata to contain appdir and copytool fields but unordered (pUtil)
Removed misleading 'release too old' message in useAtlasSetup() (RunJobUtilities)
Created getQueuedataFileName() (pUtil)
copytool() is now using getQueuedataFileName() (Mover)
readpar() is now using getQueuedataFileName() (pUtil)
Added containsJson control varible to getpar() (pUtil)
Created getExtension() (pUtil)
getQueuedata() now has json handling (pUtil)
get(), put(), rename(), getFilename(), remove() now use json if possible (JobState)
getMatchedReplicasFileName() is now using pUtil.getExtension() (Mover)
storeMatchedReplicas(), getReplicaDictionary() now use json if possible (Mover)
Added JEMstub module to pilot distribution
Created updateRunCommand4JEM() (RunJobUtilities)
Added a call to updateRunCommand4JEM() in setupTrf() (RunJobUtilities)
Created createAtomicLockFile() and releaseAtomicLockFile(), used by run() (JobRecovery)
Added pinitdir to "No space left on device" error message in get_data(), requested by Wensheng/Hiro (Mover)
Removed duplicated error assignment in updatePandaServer() (pilot)
Removed duplicated error assignment in updatePandaServer() (pilot)
Moved updateJobState(), isLogfileCopied() and isLogfileRegistered() from pilot to pUtil
Corrected missing variable name (sitename) for CERNVM site check in updatePandaServer() (pilot)
Created PandaServerClient and JobLog classes
Moved chdir from pilot to pUtil
Created DBReleaseHandler class
Created processDBRelease() (pUtil)
Added SURL metadata tag to XML in PFCxml() (pUtil)
Created getSURLDictionaryFilename(), getSURLDictionary(), putSURLDictionary(), updateSURLDictionary() (SiteMover)
Now storing guid and surl in dictionary in mover_put_data() (Mover)
Added workDir to core_get_data() calls in get_data() (lcgcpSiteMover, stormSiteMover)
Created createJobSetupScript(), getJobSetupScriptName(workdir)() and addToJobSetupScript() (pUtil)
Now writing command to job setup script in executePayload() (runJob)
Now writing command to job setup script in core_get_data() (lcgcpSiteMover, stormSiteMover)
Now writing command to job setup script in get_data() (xrdcpSiteMover)
Now writing command to job setup script in runMain() (pilot)
Added sitename to getLoopingLimit() (pilot)
Replaced loopingLimitDefault with loopingLimitDefaultUser and loopingLimitDefaultProd (pilot)
Added exception handling in core_get_data() (stormSiteMover)
Now using job workdir instead of site workdir in moveLostOutputFiles(), transferLogFile(), transferAdditional() (pilot)
Now using job workdir instead of site workdir in stageOut (runJob)
Changed workDir to jobWorkDir, created new local variable workDir in mover_put_data() (Mover)
Removed XML printouts in updatePandaServer(), added new XML printout (pilot)
Removed useless "XML will be sent message" in updatePandaServer() (pilot)
Copying surlDictionary to datadir/ in cleanup() (runJob)
Added lfcreg to mover_put_data() (Mover)
Now only performing LFC registration if lfcreg==True in mover_put_data() (Mover)
Added new pilot option B to control LFC registration in argParser() (pilot, runJob)
Created global variable lfcRegistration (pilot, runJob)
Sending lfcRegistration from pilot to runJob (pilot, runJob)
Sending lfcRegistration from moveLostOutputFiles(), transferLogFile(), transferAdditional() to mover_put_data() (pilot)
Preventing pilot LFC registration for lfcRegistration==False in put_data() (Mover)
Added sitename to getTURLFileInfoDic(), getTURLs(), createPFC4TURLs() (Mover)
Added site exclusion list to prevent rfio from being used if it is not supported in getTURLFileInfoDic() (Mover)
Changed FAILED to WARNING in "Error in copying"-error in mover_get_data() (not failed if secondary replica can be transferred) (Mover)
Added surl to tag dictionary in convertMetadata4NG() (RunJobUtilities)
Removed countryGroup in node structure in getNewJob() (pilot)
Added 'jobDefinitionID' to Job class and setJobDef() (Job)
Removed jobId log message from updateFileState() (FileStateClient)
Instantiating DBReleaseHandler in get_data() (Mover)
Now sending dbh and jobPars from get_data() to mover_get_data() (Mover)
Setting file state to "no_transfer" for skipped DBRelease file in mover_get_data() (Mover)