SULU 51a

Summary

* Removal of the _sub bit from the dataset names (internal to the pilot),
to prevent it from ending up in the destination SE paths. Requested by
Cedric Serfon et al

* Major update to the current Job Recovery algorithm; e.g. allowing
external recovery directories to be properly used (using
schedconfig.recoverdir), and preventing job recovery for gangarobot jobs
(note: only preventing jobs from going into holding state - it cannot
prevent the actual job recovery of previously failed jobs since the
processingType (gangarobot*) is not known until the job has been
downloaded and the job rec algo is run at the beginning of the pilot).
Requested by Alastair Dewhurst and Dan van der Ster

* Added an additional recognition of missing DBRelease files ('not yet
transferred' error). Requested by Dan van der Ster

* The curl site mover is now using lcg-gt to get the https path (on dCache
systems). Code by Jhen-Wei Huang

* The xrdcp site mover can now handle copyprefix lists, restoring the
possibility to write to ATLASMCTAPE. Space token and path matching is done
using schedconfig.seopt. Requested by Alessandro Di Girolamo and Rodney
Walker

* The castorSvcClassSiteMover, rfcpLFCSiteMover and xrootdSiteMovers are
now using the pilots internal time-out function to protect against handing
commands

Version info

Removed remove_sub() (SiteMover)
Uncommented calls to remove_sub() in site movers (lcgcpSiteMover, CastorSiteMover, curlSiteMover, dCacheLFCSiteMover, rfcpLFCSiteMover)
Created removeSubFromResponse() (pUtil)
Can call removeSubFromResponse() from parseDispatcherResponse() - will not be used until later pilot version, if at all (pUtil)
parseDispatcherResponse() is now returning 'response' (pUtil)
Updated getLFCFileList() to recognise missing DBReleases (Mover)
Using lcg-gt to get https path (curlSiteMover)
Created getSEFromToken() (SiteMover)
put_data() can now use copyprefix lists, restoring the possibility of writing to tape/castor (xrdcpSiteMover)
Removed unnecessary timed_command imports (lcgcpSiteMover, lcgcp2SiteMover, dCacheLFCSiteMover)
Added timed_command import and to get_data(), plus error handling (castorSvcClassSiteMover, rfcpLFCSiteMover)
Added timed_command import and to get_data() and put_data(), plus error handling (xrootdSiteMover)
Now updating SURL dictionary whenever possible (not only for successful transfers) in mover_put_data() (Mover)
Added jobrec argument to updateXMLWithSURLs() (pUtil)
Allowing surltobeset line to remain in metadata for job recovery mode in updateXMLWithSURLs() (pUtil)
Sending self.__jobrec to updateXMLWithSURLs() in postJobTask() (JobLog)
Sending self.__jobrec to updateXMLWithSURLs() in updateOutputFilesXMLWithSURLs4NG(), updatePandaServer() (PandaServerClient)
Added surlDictionary to file list in moveToExternal() (pilot)
Corrected job.workdir update to external recovery directory in moveToExternal() (pilot)
Created getFileState() (FileState, FileStateClient)
Cleanup is now querying the log file state and may remove the log if necessary (pilot)
Added argument recoveryWorkDir to mover_put_data() (Mover)
moveLostOutputFiles() is now sending recoveryWorkDir to mover_put_data() (pilot)
Now sending recoveryWorkDir to mover_put_data() in transferLogFile() (JobLog)
Switching off further job recovery for gangarobot jobs in getNewJob() (pilot)
Now setting tracing report dataset in mover_put_data() (Mover)
Removed setting of tracing report dataset in put_data() (lcgcpSiteMover, lcgcp2SiteMover, xrootdSiteMover, xrdcpSiteMover, castorSvcClassSiteMover,
rfcpLFCSiteMover, LocalSiteMover, dCacheSiteMover, dCacheLFCSiteMover, BNLdCacheSiteMover, GOSiteMover, HUSiteMover, CastorSiteMover, ChirpSiteMover,
curlSiteMover, SRMSiteMover)
Removed duplicated dsname line in put_data() (BNLdCacheSiteMover)
Created removeSubFromDatasetName() (SiteMover)
Now calling removeSubFromDatasetName() prior to calling put_data() in mover_put_data() (Mover)