SULU 47b

Summary

* The stage-out functions in the lcgcp site movers (lcgcpSiteMover,
lcgcp2SiteMover) are now using lcg-get-checksum for file verification. The
command will also set the checksum in case it is not already set. Since
testing revealed that the command is not returning correct checksum values
on some STORM sites, it is not used there for the time being.

* The special setup for Nordugrid and CERNVM has been updated to make it
compatible and use the latest round of refactoring in the pilot introduced
in the previous pilot version.

* Corrected problem with a file remove function not using the correct file
extension (json instead of pickle). Problem seen in one prod job that
failed due to an unrelated error; problem reading queuedata file which
triggered the unpredictable error with file extension followed by error
'SiteMover' has no attribute 'getSiteMover' (Savannah report 81472).

* Corrected XML sent to server after a successful job which previously did
not always contain SURL information (needed later for LFC registration by
server).

* Now selecting glExec mode for schedconfig.glexec = 'true' or 'uid'.
Requested by Jose Caballero.

* It is now possible to set the timefloor with a new pilot option -C
. This will override any schedconfig setting. It was added to
simplify pilot testing (for dev pilot and glExec testing).

* Updated exception handling in get_data/put_data() for dCache site mover.
Requested by Charles Waldman.

* Rewrote legacy code in getModTime() to use getmtime() instead of using
the find command (which is heavy on especially Lustre file systems). (code
from Jose Caballero).

Version info

Added missing self to copyNodeStruct4NG() in updatePandaServer() (PandaServerClient)
Created updateOutputFilesXMLWithSURLs4NG() (PandaServerClient)
updatePandaServer() is now updating OutputFiles.xml with SURLs (PandaServerClient)
Now importing time.localtime instead of time.time in JobLog
Corrected outputDir handling in transferAdditionalFile() (JobLog)
Corrected problem with remove() not using the correct file extension (json instead of forced pickle) (JobState)
Updated updateXMLWithSURLs() to handle SURLs for NG (pUtil)
Avoiding removing jobState file for CERNVM and Nordugrid in postJobTask() (postJobTask)
Skipped dumping of preliminary XML for CERNVM / Nordugrid in convertMetadata4NG() (RunJobUtlities)
Updated node structure XML with SURLs prior to copy for Nordugrid / CERNVM in updatePandaServer() (PandaServerClient)
Added format option to updateXMLWithSURLs() (pUtil)
Updating SURLs in metadata-.xml file in getXMLAndWorkdir() (JobLog)
Created putMetadata() (pUtil)
Now writing updated metadata-.xml file with SURLs in postJobTask() (JobLog)
Now selecting glExec mode for schedconfig.glexec = true or uid, in executePayload() and getNewJob() (runJob, pilot)
Added new pilot option -C in argParser() (pilot)
Created global variable timefloor_default used by getTimeFloor() (pilot)
Added argument timefloor_default in getTimeFloor() (pUtil)
Updated getrunpars() for timefloor deactivation for dev pilot (atlasProdPilot)
Created lcgGetChecksum() (SiteMover)
put_data() is now using lcg-get-checksum via lcgGetChecksum() (lcgcpSiteMover)
Added storm to consistency server exception list in mover_put_data() (Mover)
Updated exception handling in get_data/put_data() (dCacheSiteMover)
Now recognizing csumtype AD as well as adler32 in checksum verification in put_data() (lcgcpSiteMover)
Added log warning message when checksums do not match in put_data() (lcgcpSiteMover, lcgcp2SiteMover)
Rewrote getModTime() to use getmtime() instead of find (code from Jose Caballero) (SiteMover)
Avoiding lcg-get-checksum on storm sites (lcgcpSiteMover)