SULU 58h

Summary

* LFC function calls replaced with corresponding DQ2 code. Stage-in:

replica look ups now rely on DQ2 bulkFindReplicas() instead of

lfc.lfc_getreplicas() etc. Stage-out: file registration now using DQ2

bulkRegisterFiles() instead of corresponding LFC code. Thanks to Vincent

Garonne and Cedric Serfon for code examples and debugging.

* Bug fix in JEM code related to JEM download URL. Thanks to Frank Volkmer

for code update.

* Pilot now downloads all schedconfig fields. Previously only a selection

of the fields were downloaded since not all are relevant for pilot,

however this forces Graeme Stewart to update the code every time a new

field needs to be sent to the pilot. Now this is no longer needed. Thanks

Graeme.

* Bug fix for scope handling of log files. Tomas Kouba discovered a case

with a file in the catalog from a merge job with a missing scope in the

SURL.

* Updated PanDA Logger URL (used for storing FAX info). Requested by Ilija

Vukotic and Valeri Fine.

* Additional cleanup of AthenaMP processes to address a problem with jobs

leaving output files in the AthenaMP worker directories which then end up

in the job log. Requested by Wolfgang Ehrenfeld et al (code provided by

Wolfgang and Graeme Stewart).

Version info

Created getRucioReplicaDictionary() and getRucioFileDictionary() (Mover)    [added to 59a]

Created getFileCatalog() (Experiment, ATLASExperiment)    [added to 59a]

Renamed getReplicas() to getReplicasLFC() (Mover)    [added to 59a]

Renamed getReplicasDictionary() to getReplicaDictionaryLFC() (Mover)    [added to 59a]

Renamed old getReplicaDictionary() to getReplicaDictionaryFile() (Mover)    [added to 59a]

Created getReplicaDictionary(), used in getLFCFileList() (Mover)    [added to 59a]

Renamed getLFCFileList() to getCatalogFileList (Mover)    [added to 59a]

Renamed getLFCHosts() to getFileCatalogHosts (Mover, Experiment, ATLASExperiment)    [added to 59a]

Now using getFileCatalog() in getFileCatalogHosts() instead of readpar('lfchost') (Mover)    [added to 59a]

Added lfchost option to getReplicasLFC() (Mover)    [added to 59a]

Added host argument to getReplicaDictionary(), sending it to getReplicasLFC() (Mover)    [added to 59a]

Sending scope_dict to getCatalogFileList() in getPoolFileCatalog() (Mover)    [added to 59a]

Added scope_dict argument to getCatalogFileList() (Mover)    [added to 59a]

Sending scope_dict to getReplicaDictionary() in getCatalogFileList() (Mover)    [added to 59a]

Added scope_dict argument to getReplicaDictionary() (Mover)    [added to 59a]

Sending scope_dict to getReplicaDictionaryRucio() in getReplicaDictionary() (Mover)    [added to 59a]

Updated logger URL in toPandaLogger(). Requested by Ilija Vukotic and Valeri Fine (pUtil)    [added to 59a]

Added new version on JEMstub. Requested by Frank Volkmer (JEMstub)    [added to 59a]

Renamed registerFileLFC() to registerFileInCatalog() (Mover, SiteMover)    [added to 59a]

Created bulkRegisterFiles() used by registerFileInCatalog() (SiteMover)    [added to 59a]

Added scope to registerFileInCatalog() call in mover_put_data() (Mover)    [added to 59a]

Added scope argument to registerFileInCatalog() (SiteMover)    [added to 59a]

Now using bulkRegisterFiles() for registering files using Rucio methods (SiteMover)    [added to 59a]

Now sending logFile to getScope() to prevent problem with merged log files which contain .log. (Mover)    [added to 59a]

Using logFile in getScope() (Mover)    [added to 59a]

Sending thisExperiment.getFileCatalog() to registerFileInCatalog() from mover_put_data() (Mover)    [added to 59a]

Now getting full schedconfig dump, using ".all.json" instead of ".pilot.json" in getQueuedata() (SiteInformation)    [added to 59a]

Minor cleanup in if statement for lfc file registration in put_data() (Mover)    [added to 59a]

Sending thisExperiment to getCatalogFileList() from getPoolFileCatalog() (Mover)    [added to 59a]

Added thisExperiment argument to getCatalogFileList(), getReplicaDictionary() (Mover)    [added to 59a]

Sending thisExperiment to getReplicaDictionary() from getCatalogFileList() (Mover)    [added to 59a]

Created willDoAlternativeFileLookups(), used by getReplicaDictionary() (Experiment, ATLASExperiment)    [added to 59a]

Sending thisExperiment.getFileCatalog() instead of lfc host to getReplicaDictionary() (Mover)    [added to 59a ??? not explicitly]

Removed Exception in exception handling in getRucioReplicaDictionary() (Mover)    [added to 59a]

Removed Exception in exception handling in isTapeSite(), getDQ2SEType(), reportFileCorruption(), bulkRegisterFiles() (SiteMover)[added to 59a]

Removed RUCIO_ACCOUNT warning from checkSpecialEnvVars() (pUtil)    [added to 59a]

Added RUCIO_ACCOUNT to payload setup in addEnvVars2Cmd() (ATLASExperiment)    [added to 59a]

Created cleanupAthenaMP(). Code from Graeme Stewart and Wolfgang Ehrenfeld (ATLASExperiment)    [added to 59a]

Calling cleanupAthenaMP() from removeRedundantFiles() (ATLASExperiment)    [added to 59a]