SULU 58k

Summary

* Switching to FAX site mover for transferType xrd/fax (used in
Hammercloud tests). Requested by Ilija Vukotic

* Updated replica loop over multiple catalogs. Previously, all input files
needed to be registered in the same catalog. Now pilot can grab replicas
from different catalogs. Requested by Ilija Vukotic

* Updated process killer to alleviate problem with lingering orphans from
AthenaMP. Requested by Alessandro Di Girolamo et al. os.setpgrp() set in
runJob subprocess, pgrp value returned to pilot which uses it to kill the
entire process group

* Only setting ATHENA_PROC_NUMBER when nCores > 1 to prevent AthenaMP jobs

using 1 process only. Also, sending nCores back to server during job

updates as well as with job metrics (coreCount). Requested by Rodney

Walker et al

* Now using --cksum if available in xrdcp command (from xrdcp v 3.3.6)

* Updated the PanDA Logger URL. Requested by Ilija Vukotic

* Corrected variable ERR_NOATHENAMETADATA, should be

ERR_NOPAYLOADMETADATA. Requested by Andrej Filipcic

Version info

Always extending file catalog host list in getFileCatalogHosts() (Mover)    [added to 59a]
Switching to FAX site mover for transferType xrd/fax in getNewJob() (pilot)    [from 59a]
Added coreCount plus handling to job definition. Requested by Rodney Walker et al (Job)    [added to 59a]
Created postGetJobActions() (Experiment, ATLASExperiment)    [added to 59a]
Added handling for postGetJobActions() in runMain() (pilot)    [added to 59a]
Now sending the coreCount back to the server in getNodeStructure(). Requested by Wolfgang Ehrenfeld (PandaServerClient)    [added to 59a]
Only setting ATHENA_PROC_NUMBER when nCores > 1 to prevent AthenaMP jobs using 1 process only, requested by Rodney Walker et al (Node)    [added to 59a]
Created verifyNCoresSettings() used by postGetJobActions() (ATLASExperiment)    [added to 59a]
Added new error code 1217, ERR_CORECOUNTMISMATCH (PilotErrors)    [added to 59a]
Added coreCount to jobMetrics in getJobMetrics() (PandaServerClient)    [added to 59a]
Removed unused error object from getReplicaDictionaryRucio() (Mover)    [added to 59a]
Created jobMetric() used by getJobMetrics() (PandaServerClient)    [added to 59a]
Added rstrip() to final jobMetrics string. Also changed lstrip(' ') to lstrip() (PandaServerClient)    [added to 59a]
Removal of duplicate replicas in getPoolFileCatalog() (Mover)    [added to 59a]
Updates to getPoolFileCatalog(), getCatalogFileList(), getReplicaDictionary(), getReplicaDictionaryRucio(), verifySURLGUIDDictionary(), getFileInfo() related
to replica loop which now iterates over several hosts in case of incomplete replica list (Mover)    [added to 59a]
Created verifyReplicasDictionary() used by getPoolFileCatalog() (Mover)    [added to 59a]
Cleaned up outdated md5 sum comments in put_data() (xrdcpSiteMover)    [added to 59a]
Now using --cksum if available in xrdcp command in put_data() (xrdcpSiteMover)    [added to 59a]

Added subprocess imports (pilot)    [added to 59a]

Added functions developed by Graeme Stewart for process killing: getAncestry(), listChildren(), infanticide() (pilot)    [added to 59a]

Added multi-core support to cmd2 command in getJobExecutionCommand() (especially important in case coreCount=1 to limit parallel make), requested by Andrej Filicic (ATLA\

SExperiment)    [added to 59a]

Renamed old killProcesses() to killProcessesOld() (pilot)    [added to 59a, removed again]

killProcesses() now uses infanticide() (pilot)    [added to 59a, removed again]

Renamed killProcess() to killProcessNew(), took back old pilot code since new code does not handle orphans (pilot) [removed?]

Corrected ERR_NOATHENAMETADATA -> ERR_NOPAYLOADMETADATA in updatePandaServer() (PandaServerClient)    [added to 59a]

Added os.setpgrp() to __main__() (runJob)    [added to 59a, also to runEvent]

Setting "setup" job state in __main__() prior to setup stage (runJob)    [added to 59a, also to runEvent]

Calling updatePilotServer() from __main__() prior to setup stage to send (runJob)    [added to 59a, also to runEvent]

Updated the PanDA Logger URL in toPandaLogger(). Requested by Valeri Fine and Ilija Vukotic (pUtil)]    [added to 59a]

Removed usage of pgrp in if-statement in updateHandler::handle() (pilot)    [added to 59a]

Removed printout of cmtconfig value in updateHandler::handle() (pilot)    [added to 59a]

Now receiving pgrp in updateHandler::handle() (pilot)    [added to 59a]

Rewrote killProcesses() to use os.killpg() (pilot)    [added to 59a]

Removed killOrphans() from killLoopingJob() (pilot)    [added to 59a]

Removed useless "pass 1/2" log messages from cleanUpEndedJobs() (pilot)    [added to 59a]

Setting and receiving transferType from updateQueuedataFromJobParameters() in getNewJob() (pilot)    [added to 59a]

Added special handling for transferType in updateQueuedataFromJobParameters() which now also returns transferType (SiteInformation)    [added to 59a]

Added a warning message in updateMetadata() in case of empty xml (JobLog)    [added to 59a]

Added missing variable strXML in updateMetadata() (JobLog)    [was already added to 59a]