SULU 57a

Summary

* Input/output paths now using Rucio format (if desired). For activation

and for larger scale testing, finish se[prod]path's with "/rucio" (e.g.

seprodpath="/pnfs/desy.de/atlas/dq2/atlasdatadisk/rucio"). If it is

preferred, the activation can eventually also be through a new schedconfig

boolean (not created yet).  Tested at DESY-HH and LRZ-LMU.

* Added CMS experiment classes to pilot. Code from Valentina Mancinelli.

The pilot has also been additionally refactored and many ATLAS specific

methods have been moved to the ATLAS experiment classes (most notably the

payload setup methods)

* Added Nordugrid classes to pilot. Code contribution from Andrej Filipcic

(thanks for helping with the tests). Probably the special Nordugrid

classes will be removed with the next pilot release, but are currently

needed due to the special job setup for jobs on NG and the separation of

ATLAS specifics from the general pilot code

* Now looping over LFC hosts to handle a special cross-Atlantic FAX case

(assuming schedconfig.allowfax = True). Thanks to Johannes Elmsheuser for

helping with the HC tests. Apologies for the long delay of deploying this

code

* Added http://atlas-install.roma1.infn.it and http://classis01.roma1.infn.it as valid URL's for downloading code. Requested by Alessandro de Salvo

* Added loop over all stdout files (including new log.*) in file size

verification performed every ten minutes. Requested by Graeme Stewart

* Updated queuedata overwrites to handle comma separated key values (e.g.

--overwriteQueuedata={lfcpath='/grid/atlas/dq2',sepath='/pnfs/usatlas.bnl.g

ov/BNLT0D1/,/pnfs/usatlas.bnl.gov/MCTAPE/,/pnfs/usatlas.bnl.gov/BNLT1D0/'}

). Requested by Simone Campana

* Added special timeout case for BNL_ATLAS_RCF to getProperTimeout(),

requested by Xin Zhao

* Removed usage of http_proxy for BNL. Requested by John Hover

* Added tmp*, *.tmp, *.TMP to list of files/dirs to be removed before log

is created. Requested by Graeme Stewart et al

* Now converting short surls into full surls for input files in storm site

mover. Code by David Rebatto

* Removed hardcoded ATLASPRODDISK from pilot. Previously it would be

forced for cross-cloud jobs. Now the default space token will be selected.

Approved by Rodney Walker

Version info


Removed dataset name update from updateReport(), as well as arguments dsdict, lfn and number_of_files, and its call in mover_get_data() (Mover)
Introduced __error as new data member in SiteInformation class, used in processQueuedata(),  (SiteInformation)
Removed error argument from processQueuedata(), extractAppdir() (SiteInformation)

Removed error argument from call to processQueuedata() in prepareAlternativeStageOut(), handleQueuedata() (Mover, pilot)

Removed error argument from prepareAlternativeStageOut() and from its call in mover_put_data() (Mover)

Removed legacy from related to LYON and dq2 sites in getSpecialSetupCommand() (RunJobUtilities)

Sending thisExperiment to setup() (runJob)

Using thisExperiment to call getJobExecutionCommand() instead of setupTrf() from RunJobUtilities (runJob)

Moved setupTrf() to ATLASExperiment, and renamed it to getJobExecutionCommand() (RunJobUtilities, ATLASExperiment)

Moved setINDS() from RunJobUtilities to Experiment (RunJobUtilities, Experiment)

Moved verifyReleaseInTagsFile(), getInstallDir(), installPyJobTransforms(), installPacman(), getCmtsiteCmd, getCacheInfo(), getProdCmd2()

getTag(), isCMTCONFIGOutcommented(), checkCMTCONFIG(), getSpecialSetupCommand(), getAnalyCmd2(), updateAnalyCmd2()

from RunJobUtilities to ATLASExperiment (RunJobUtilities, ATLASExperiment)

Renamed getSiteroot() to getProperSiterootAndCmtconfig() (pUtil)

Moved verifyCMTCONFIG(), verifyCmtsiteCmd() from pUtil to ATLASExperiment (pUtil, ATLASExperiment)

Removed unused arguments from updateAnalyCmd2(), getSpecialSetupCommand(), updateAnalyCmd2() (ATLASExperiment)

Removed old fashion trf setup from getJobExecutionCommand() (ATLASExperiment)

Moved getValidBaseURLs(), downloadTrf(), getAnalysisTrf(), getAnalyCmd3(), getFileTransferInfo(), getSetupFromCopysetup()

from RunJobUtilities to Experiment (RunJobUtilities, Experiment)

Removed unnecessary usages of variable experiment in setPython(), findPythonInRelease(), getSpecialSetupCommand() and their calls (ATLASExperiment)

Removed outdated ATLAS release checks from getAnalyCmd3() (Experiment)

Renamed getAnalyCmd3() to getAnalysisRunCommand(), renamed cmd3 to run_command within method (Experiment)

Added Rucio scopes to Job class, and setJobDef() (Job)

Created getPathFromScope(), getFullPath() (SiteMover)

Created createZippedDictionary(), used by get_data() (Mover)

Creating scopedict in get_data(), sending it to mover_get_data() (Mover)

Added scopedict argument to mover_get_data() (Mover)

Sending scopedict and sitemover object to getFileInfo() (Mover)

Added scopedict and sitemover arguments to getFileInfo() (Mover)

Now using scope for tier 3 sites in getFileInfo() (Mover)

Sending scopedict to getPoolFileCatalog() from getFileInfo() (Mover)

Added argument scopedict to getPoolFileCatalog() (Mover)

Renamed createFileDicsForLFC() to createFileDictionaries() (Mover)

Updated willDoFileLookups() and created doFileLookups, and added __doFileLookups (Experiment)

Updated willDoFileLookups() for __doLookups boolean (ATLASExperiment)

Renamed all accessdict to access_dict, scopedict to scope_dict, filename_dic to lfn_dict (Mover)

Sending sitemover to getPoolFileCatalog() from getFileInfo(), added sitemover argument to getPoolFileCatalog() (Mover)

Renamed thisfile to lfn in parts of getFileInfo() (Mover)

Removed unused xml_from_PFC from getLFCFileList() (Mover)

Created getFileLookups() (Experiment)

Using getFileLookups() in willDoFileLookups() (ATLASExperiment)

Using doFileLookups() in specialChecks() (ATLASExperiment)

Added arguments scopeOut and scopeLog to mover_put_data() (Mover)

Sending scopeOut list to mover_put_data() from stageOut() (runJob)

Sending scopeLog (list) to mover_put_data() from transferLogFile() (JobLog)

Sending scopeLog (list) and scopeOut to mover_put_data() from moveLostOutputFiles() (pilot)

Renamed tokenList to token_list in mover_put_data(), and getSpaceTokenList() (Mover)

Renamed fileList to file_list in mover_put_data() (Mover)

Created getScope() used by mover_put_data() (Mover)

Sending scope to sitemover_put_data(), two places, from mover_put_data() (Mover)

Added scope argument to sitemover_put_data() (Mover)

Added experiment to Job class (Job)

Setting job.experiment in getNewJob(), __main__() (pilot, runJob)

Removed experiment argument from interpretPayload() and from its call in __main__() (ErrorDiagnosis, runJob)

Removed experiment argument from interpretPayloadStdout() and extractJobInformation(), and from their calls in interpretPayload() (ErrorDiagnosis)

Removed experiment argument from get_data() and from its call in stageIn() (Mover, runJob)

Moved getPayloadName() from pUtil to *Experiment, created prototype method in Experiment (pUtil, *Experiment)

Now using getPayloadName() from *Experiment instead of from pUtil, in runMain(), __main__() (pilot, runJob)

Removed thisExperiment verification from __main__() (runJob)

Created getMetadataForRegistration() (*Experiment)

Updated PFCxml() to use getMetadataForRegistration() (pUtil)

Added experiment as option to PFCxml() (pUtil)

Sending experiment to PFCxml() from transferLogFile(), transferAdditionalFile() (JobLog)

Sending experiment to PFCxml() from getXML() (PandaServerClient)

Sending experiment to PFCxml() from moveLostOutputFiles(), OK to send "unknown" (pilot)

Protected against unset job.experiment in moveLostOutputFiles() (pilot)

Sending experiment to PFCxml() from createFileMetadata(), stageOut() (runJob)

Created getAttrForRegistration() (Experiment, CMSExperiment; implementation not needed in ATLASExperiment)

updateXMLWithSURLs() is now using getAttrForRegistration() (pUtil)

Added experiment argument to updateXMLWithSURLs() (pUtil)

Now calling updateXMLWithSURLs() with experiment argument, from updateOutputFilesXMLWithSURLs4NG(), updatePandaServer() (PandaServerClient)

Added experiment argument to updateOutputFilesXMLWithSURLs4NG() (PandaServerClient)

Calling updateOutputFilesXMLWithSURLs4NG() with experiment argument (PandaServerClient)

Created getExpSpecificMetadata() (Experiment, CMSExperiment)

Using getExpSpecificMetadata() in postJobTask() (JobLog)

Sending expSpecificMetadata to updatePandaServer() (JobLog)

Added additionalMetadata argument to updatePandaServer() (JobLog, PandaServerClient)

Sending additionalMetadata to PandaServerClient::updatePandaServer() in updatePandaServer() (JobLog)

Added additionalMetadata to metaData dictionary in updatePandaServer() (PandaServerClient)

Updated getDirList() to handle the rucio identifier in se[prod]path (SiteMover)

Sending scope to put_data() from sitemover_put_data() (Mover)

Extracting scope from pdict in put_data() and sending scope to getProperPaths() (lcgcpSiteMover, lcgcp2SiteMover, xrdcpSiteMover, aria2cSiteMover,

curlSiteMover, LocalSiteMover)

Added **pdict argument to getProperPaths() (SiteInformation)

Replaced JobData and alt arguments with **pdict in getProperPaths() (CMSSiteInformation)

Added scope to getFinalLCGPaths() (SiteMover)

Calling getFinalLCGPaths() with scope from getProperPaths() (ATLASSiteInformation)

Created isFAXAllowed(), used by mover_get_data() and getPoolFileCatalog() (Mover)

Looping over LFC hosts while calling getLFCFileList() in getPoolFileCatalog() (Mover)

Added http://atlas-install.roma1.infn.it and http://classis01.roma1.infn.it to getValidBaseURLs() (Experiment)

Added loop over all stdout files (including new log.*) in checkPayloadStdout() (pilot)

Renamed filenameAthenaXML to filenamePayloadXML, athenaXMLProblem to payloadXMLProblem,

AthenaXML to payloadXML in updatePandaServer() (PandaServerClient)

Renamed metadata-%s.xml.ATHENA to metadata-%s.xml.PAYLOAD in updatePandaServer() (PandaServerClient)

Renamed fname + ".ATHENA" to fname + ".PAYLOAD" in updatePandaServer() (pUtil)

Renamed "metadata-%s.xml.ATHENA" % (repr(jobId)) to "metadata-%s.xml.PAYLOAD" % (repr(jobId)) in moveTrfMetadata() (runJob)

Renamed BAK = ".ATHENA" to BAK = ".PAYLOAD" in getMetadata() (pUtil)

Renamed "metadata-%d.xml.ATHENA" to "metadata-%d.xml.PAYLOAD" in moveToExternal() (pilot)

Renamed ERR_NOATHENAMETADATA to ERR_NOPAYLOADMETADATA, ERR_ATHENAOUTOFMEMORY to ERR_PAYLOADOUTOFMEMORY (PilotErrors)

Renamed "Athena metadata is not available" to "Payload metadata is not available" (PilotErrors)

Renamed "Athena ran out of memory" to "Payload ran out of memory" (PilotErrors)

Created getCopytoolIn() used by reportFileCorruption(), getPoolFileCatalog() (Mover)

Will explicitly not use Rucio path generation during stage-in for aria2c, in getPoolFileCatalog(), since aria2c needs

full replica information from the LFC (Mover)

Updated extractQueuedataOverwrite() to handle comma separated key values, requested by Simone Campana (SiteInformation)

Added special timeout case for BNL_ATLAS_RCF to getProperTimeout(), requested by Xin Zhao (pUtil)

Added tmp*, *.tmp, *.TMP, removed tmp, in rmRedundants() (JobLog)

Created getRucioPath(), called from getFileInfo() and ... (SiteMover, Mover)

Added optional scope, token arguments to getProperPaths() (SiteMover)

Calling getProperPaths() with scope, token from put_data() (curlSiteMover, aria2cSiteMover, lcgcpSiteMover, lcgcp2SiteMover, LocalSiteMover)

Removed usage of http_proxy for BNL, in __main__() and setHTTPProxies(). Requested by John Hover (runJob, pUtil)

Added filesizeIn, checksumIn arguments to getPoolFileCatalog(), and its call from getFileInfo() (Mover)

Added filesizeIn, checksumIn arguments to createFileDictionaries(), and its call from getPoolFileCatalog() (Mover)

createFileDictionaries() now return filesize and checksum dictionaries to getPoolFileCatalog() (Mover)

Created verifyFileInfo() used by getRucioFileList() (Mover)

Renamed ERR_ATHENAOUTOFMEMORY to ERR_PAYLOADOUTOFMEMORY in interpretPayloadStdout() (ErrorDiagnosis)

Ignoring empty VO_ATLAS_SW_DIR in checkSpecialEnvVars() (pilot)

Added Nordugrid to experiment factory. Code from Andrej Filipcic (ExperimentFactory)

Added testImportLFCModule(). Code from Andrej Filipcic (NordugridATLASExperiment)

Removed region variable from getJobExecutionCommand() (ATLASExperiment)

Moved setupNordugridTrf() from RunJobUtilities to NordugridATLASExperiment

Created NordugridATLASSiteInformation class

Added CMS and Nordugrid-ATLAS to SiteInformationFactory

Now converting short surls into full surls for input files in core_get_data(). Code by David Rebatto (stormSiteMover)

Removed ATLASPRODDISK hardcoding from getProperSpaceTokenList(), requested by Ikuo Uedo et al. (Mover)