SULU 52a

Summary

* Number of events is now primarily read from metadata.xml. Requested by
Graeme Stewart

* File sizes and checksums are now forwarded to the pilot from the
dispatcher, in an effort to greatly reduce the load on the DQ2 server. In
case the information is not delivered by the dispatcher the pilot falls
back to the old method as a backup. Requested by Vincent Garonne

* Improved get-retry. Previously, some failed stage-in attempts (e.g. in
the case of corrupted files) the second stage-in attempt failed since the
local file from the previous attempt was not removed

* Automatic switch to alternative CMTCONFIG for user jobs. If a user has
specified the wrong CMTCONFIG the pilot will attempt to use an alternative
CMTCONFIG (found in the local software directory). Requested by Asoka de
Silva et al

* Improved handling of hanging commands during setup. Previously such jobs
would fail with ³Exception caught by runJob² (due to bad signal handling
in python commands module for python v < 2.6). Now properly labeled with
new error code 1195, ³Command timed out². Requested by Alexandre Vaniachine

* Removed tmp directory from pilot log

Version info

Created processMetadata() (RunJobUtilities)
Added call to processMetadata() in getNumberOfEvents() (RunJobUtilities)
Added filesizeIn and checksumIn to Job class (Job)
Printing filesizeIn and checksumIn in displayJob() (Job)
Grabbing filesizeIn and checksumIn in setJobDef() (Job)
Now sending filesizeIn and checksumIn from get_data() to mover_get_data() (Mover)
Now sending filesizeIn and checksumIn from mover_get_data() to getFileInfo() (Mover)
Added arguments filesizeIn and checksumIn to getFileInfo() (Mover)
Updated prepareInFiles() to handle changes in input file, file size and checksum lists in case of inproper file names (RunJobUtilities)
Updating filesizeIn and checksumIn using prepareInFiles() in stageIn() (runJob)
Created removeLocal() (SiteMover)
Using removeLocal() in get_data() (xrdcpSiteMover, xrootdSiteMover, lcgcpSiteMover, lcgcp2SiteMover, castorSvcClassSiteMover, CastorSiteMover,
LocalSiteMover, BNLdCacheSiteMover, dCacheSiteMover, rfcpSiteMover)
Renamed final_dest_file to dest_path in get_data() (CastorSiteMover)
Added variable dest_path in get_data() (BNLdCacheSiteMover)
Removed unnecessary variable loc_filename and renamed dest_file to dest_path in get_data() (BNLdCacheSiteMover, dCacheSiteMover)
Created getFileInfoFromDispatcher() and getFileInfoDictionaryFromDispatcher() (Mover)
Now using getFileInfoFromDispatcher() and getFileInfoDictionaryFromDispatcher() in getFileInfo() (Mover)
Added tmp directory to dir list in rmRedundants() (JobLog)
Created getCmtconfigAlternatives() (pUtil)
getSiteroot() now returns the cmtconfig that is confirmed to work (pUtil)
Rewrote getSiteroot() to loop over cmtconfig
Renamed analJob to analysisJob (RunJobUtilities)
Added call to getCmtconfigAlternatives() in setupTrf() (RunJobUtilities)
Added cmtconfig alternatives to getSiteroot() call in setupTrf() (RunJobUtilities)
Corrected bad return statement in getAnalyCmd2() (RunJobUtilities)
Added call to getSiteroot() primarily in order to get a good cmtconfig value in setupTrf() (RunJobUtilities)
User analysis setup now loops over alternative cmtconfig values in setupTrf() (RunJobUtilities)
Added timed_command() to verifyCMTCONFIG() (pUtil)
verifyCmtsiteCmd() can now identify time outs (pUtil)
Added new error code 1195, ERR_COMMANDTIMEOUT [retryable] (PilotErrors)
Added handling of ERR_COMMANDTIMEOUT in checkCMTCONFIG() (RunJobUtilities)
Avoiding testing alternative cmtconfig in case of time out in setupTrf() (RunJobUtilities)
Added additional time-out handling in timedCommand() (pUtil)