SULU 48b

Summary

* Updated the get retry mechanism. The original get retry was basically replaced with the multiple replica retrials some time ago, but is now working in combination with multiple replicas. If stage-in fails, the pilot will take a one minute nap and then try again. If however there's another replica available, it will try that instead. If that replica also fails to stage in, the pilot will make a second attempt of the first replica, and so on. The default number of get retries is set to 2 (can be changed with pilot option –x ).

* Fixed a problem with mover jobs. A mover job previously threw an error ("Unknown pilot error code") due to a bad exit code in a pilot function, even though the mover succeeded. Now fixed.

* New internal pilot time-out function (timed_command) by Charles Waldman.

* Support for comma separated copyprefix lists for mixed file sources (supported by the xrdcp site mover), to alleviate a problem with Castor at CERN. Requested by Rod Walker, Alessandro Di Girolamo. Currently under testing.

* Moved file corruption reporting from stage-out to stage-in. [In the multiple get retry scenario described above, a file corruption (SE -> WN) will only be reported if all get operations fail].

Version info

Resetting ec = -1 to zero in getOutputFileInfo() to prevent prun jobs with no output files from failing (pUtil)
New timed_command.py version from Charles Waldman (timed_command)
Added support for comma separated old/newPrefix list in updateCopysetup() (pUtil)
Changed to_native_lfn to to_native_lfn2 in put_data() (CastorSiteMover))
Removed re-imports of sys module in get_data() and put_data() (dCacheSiteMover)
Removed re-imports of re module (SiteMover)
Removed some unused variables (dCacheSiteMover, SiteMover, pUtil)
Cleanup usage of urllib (SiteMover)
Created getCopyprefixLists(), foundMatchedCopyprefixReplica() (Mover)
Added support for copyprefix lists in getLFCFileList() (Mover)
Cleaned up pUtil with pychecker (pUtil)
Moved getCopyprefix*() from Mover to pUtil (pUtil, Mover)
Updated getProdSourceLabel() to make it easier to test panda mover jobs (pilot)
Corrected collectZombieJob() call in cleanup() (pilot)
Changed stageInTries from 1 to 2, global (pilot)
Changed _attempt to replica_number in get_data() (Mover)
Added a loop with get retries in mover_get_data() (Mover)
Out-commented reportFileCorruption() in put_data() / mover_put_data() (lcgcpSiteMover, Mover)
Added call to reportFileCorruption() in get_data() / mover_get_data() (lcgcpSiteMover, Mover)
Corrected fchecksum handling in get_data() (xrdcpSiteMover)
Added taskID to Job class (Job)
Now setting task ID environmental variable in getJob() (pilot)