UHURA 37a

Summary

There have been many internal changes to the code, e.g. the internal error code system has been completely rewritten, and further parts of the code have been refactored in preparation for the glExec integration with the pilot. Of more interest perhaps are the following new features:

* LFC get replicas bulk requests. Previously each input file generated a request to the LFC causing unnecessary strain. This has now been replaced with a single bulk call to lfc.lfc_getreplicas() for all input files.

* HammerCloud metrics. The pilot is now reporting the number of processed events to the dispatcher. It is not yet reporting the number of actual input files processed by athena. Will be added soon when the required new field in the DB has been made (the request was made yesterday by Tadashi).

* DQ2 tracing. With the help of Mario, tracing has been added to a first round of site movers (lcgcp, lcgcp2, dCache, xrootd, BNLdCache, castorSvcClass, rfcpLFC). Mario has also updated the remaining site movers which are pending tests and will be added as soon as possible. It is also being implemented in the ARC control tower by Andrej.

* Support for the pilot release candidate testing scheme. A release candidate of the pilot is downloaded by the autopilot wrapper / pilot factories 1% of the time (both autopilot and pilot factories have been updated by Jose and Graeme). These pilots only pick up special jobs defined by a modified version of Douglas' FT job script and are evaluated using a new error reporting page provided by Aaron.

Some of the minor updates include:

* In the case the pilot wrapper failed to download the queuedata, the ATLAS_CONDDB env variable would not be set. Now set in the pilot as well.

* Changed lcg-cp timeout from 18000 s to 1800 s (lcgcp2SiteMover).

* Get function in dCacheSiteMover now recognizes one error where dccp gives a zero exit code even though it failed:

"Failed to create a control line
Failed open file in the dCache.
Can't open source file : Unable to connect to server
System error: No route to host"

* Additional corrections to the dq2 site mover (dataset name was not propagated from the dispatcher to the site mover, lfc registration was not disabled when dq2-put was used in the US). Under testing at OU_OSCER_ATLAS and HEPHY-UIBK (Paul, Horst, Mario).


Version info
Introduced a pilot version tag, RC for release candidates, PR for production pilot (pilot)
Now sending pilot version tag with pilotID (pilot)
Removed unused _test_errors() function (PilotErrors)
Renumbered internal error codes (PilotErrors)
Removed error code matching in get_data() (Mover - 3kB less code)
Renamed and replaced all occurances of ERR_MOVEFAILED with ERR_STAGEOUTFAILED
Renamed and replaced all occurances of ERR_FAILEDCP with ERR_STAGE[INOUT]FAILED
Removed ERR_FAILEDCP (PilotErrors)
Introduced ERR_STAGEINFAILED for error code 1099 (Mover, PilotErrors)
Corrected wrong error code in get function (ERR_DQ2GETFAIL -> ERR_STAGEINFAILED) (LocalSiteMover)
Rewrote removeDuplicates() which is now order preserving (pUtil)
Removed second sorting pass which is no longer necessary since removeDuplicates() is order preserving (Mover)
Simplified sorting algoritm (sortReplicas()) (Mover)
Now only testing file sizes in put_data() if checksum could not be verified (dCacheSiteMover)
Some cleanups done in dCacheSiteMover, GSIftpSiteMover, COGSiteMover, dCacheLFCSiteMover,
castorSvcClassSiteMover, CastorSiteMover, lcgcpSiteMover, lcgcp2SiteMover, LocalSiteMover
rfcpLFCSiteMover, SiteMover, SRMSiteMover, xrdcpSiteMover, xrootdSiteMover, UberftpSiteMover
Put function only returned MD5 mismatch error in case of checksum failure, now also adler32 (xrdcpSiteMover)
Replaced ERR_WRONGAD/MD5 with ERR_[GETPUT][ADMD5]MISMATCH everywhere
Removed unnecessary verifyPilotWorkir() (pUtil)
Corrected error code handling in installPyJobTransforms() (RunJobUtilities)
Removed getErrorStrVerbose() and error_dic (PilotErrors)
Using getErrorStr() instead of removed getErrorStrVerbose() (Mover, pUtil)
Removed use of replica.host in getPoolFileCatalog() since it will not be available when lfc_getreplicas
is used instead of lfc_getreplica (Mover)
getProdSourceLabel() now sets prodSourceLabel to rc_test for RC pilots (pilot)
Added removeFile() call in put_data (lcgcpSiteMover)
Removed unused status variable (runJob)
SystemError exception handling removed from put_data operations (pilot, runJob)
put_data() operations simplified (pilot, runJob)
Removed unnecessary call to updatePandaServer() in put_data() exception handling in moveLostOutputFiles() (pilot)
Created FileRegistration class responsible for LRC registrations (LFC later)
Removed redundant LRC file registration code (pilot, runJob)
Moved fileRegistration() from pUtil to FileRegistration and renamed it to performLRCFileRegistration (pUtil, FileRegistration)
Introduced new pilot option -i to switch between PR and RC pilots (pilot)
Argument list now alphabetical in argParser() (pilot)
Created setCONDDBVar(). If the pilot wrapper fails to download the queuedata, ATLAS_CONDDB will not
be set (RunJobUtilities)
Renamed file_list to file_dic (Mover)
Created getLFCFileList(), getReplicas() for bulk lookup in LFC (Mover)
Created replica support class, getReplicasDictionary(), getGuids() (Mover)
Cleaned up getPoolFileCatalog() (Mover)
getPoolFileCatalog() is now using getLFCFileList() (Mover)
mover_get_data() is now using getReplicas() (Mover)
Replaced monitor WARNING messages with simple Warning messages in removeFile(), getRemoteFileSize(),
getdCacheChecksum() (SiteMover)
Created isPNFSVisible() (SiteMover)
getdCacheChecksum() and getRemoteFileSize() are now checking if PNFS is visible on the worker node
before attempting to get the checksum/file size (SiteMover)
Changed lcg-cp timeout from 18000 to 1800 (lcgcp2SiteMover)
Created isOutOfMemory(), getNumberOfEvents() (RunJobUtilities)
Added nEvents to Job class (Job)
Pilot is now extracting and reporting the total number of processed events (pilot, runJob)
get_data() now recognizes some dccp failures with zero exit code (dCacheSiteMover)
Doing an ls -l on in/output files (RunJobUtilities, Mover)
Now only importing urlretrieve() from urllib instead of whole module (RunJobUtilities)
Added Mario's code for DQ2 tracing with lcgcpSiteMover (Mover, lcgcpSiteMover)
DQ2 tracing reporting implemented in get/put_data() (lcgcpSiteMover, lcgcp2SiteMover, dCacheSiteMover,
xrootdSiteMover, BNLdCacheSiteMover, castorSvcClassSiteMover, rfcpLFCSiteMover)
[Also being implemented in the ARC control tower separately]
sendTrace() added to SiteMover
Created evaluate() (pUtil)
Created getInitialTracingReport() (Mover)
Skipping LFC registration in the pilot when copytool is dq2 (Mover)
Renamed toDispatcher() to toServer() (pUtil)
Rephrased less serious warning in addPaths() which is irrelevant for e.g. OU_OSCER_ATLAS
which is still using dq2url (SiteMover)
Created realDatasetsIn data member in Job class (Job)
Now using realDatasetsIn instead of dispatchDblock (Mover)
Now always send dsname to get_data(), needed by dq2-get (Mover)
Changed errordiag of ec 1170/440311 (removed "Get error:" in proddb from errordiag)
Changed errordiag of ec 1199/240901 (removed "job work" in proddb from errordiag)
Renamed fchecksum/fsize to _fchecksum/_fsize and lrc_fsize/lrc_checksum to fsize/fchecksum (Mover)
Removed redundant variable lrc_metadata_dom (Mover)