SULU 58g

Summary

* FAX updates; Added missing stage-in error "No such file or directory" (error code 1103) to FAX retry. Now following Rucio file name convention in global file path generation (instead of using old to_native_lfn() to generate path) when SURL contain "rucio" sub string (to_native_lfn() is still used if SURL does not contain the rucio sub string). Requested by Johannes Elmsheuser and Ilija Vukotic

* Now assuming SITEROOT to follow convention software_path/cmtconfig/release. Previous pilot version would source setup file to get SITEROOT. Requested by Alessandro de Salvo

* Fixed a problem with core file removal (leading to core file ending up in the log). Problem discovered by Wolfgang Ehrenfeld

* Displaying the IP number of the WN in the log. Requested by Alessandro Di Girolamo

* Now allowing for JEM monitoring if schedconfig.allowjem exists and is set to True. Currently this new field is not delivered with queuedata yet. Until it is available in queuedata pilot will only allow JEM in DE cloud [as before]. Requested by JEM team

* Added '2> /dev/null' to all uuidgen usages, since command threw stderr on some NG analysis jobs on slc6. Requested by Andrej Filipcic

* Getting appdir from new env variable VO_ATLAS_RELEASE_DIR for AtlasP1HLT and AtlasHLT jobs. Discussed at https://its.cern.ch/jira/browse/PRODSYS-77 (Migrate HLT reprocessing to SLC6)

* Simplification of production payload setup [similar to NG job setup]. First removed python localization (previously command contained full path to python executable), then removed explicit python altogether. Removed explicit path to trf. Requested by Rodney Walker

Version info

Note: "added to 59a" means that the change has been applied to the other dev pilot in development, PICARD 59a.

Created PilotFAXErrorCodes and isPilotFAXErrorCode() (PilotErrors)    [added to 59a]
now uses error.isPilotFAXErrorCode instead of error.isPilotResubmissionErrorCode in mover_get_data() (1103 was missing) (Mover)    [added to 59a]
Added surl argument to sitemover.getGlobalFilePaths() call in convertSURLtoTURLUsingDataset() (Mover)    [added to 59a]
Added surl argument to getGlobalFilePaths() (FAXSiteMover)    [added to 59a]
Added surl argument to self.getGlobalFilePaths() call in findGlobalFilePath() (FAXSiteMover)    [added to 59a]
Sending gpfn instead to lfn to self.findGlobalFilePath() in get_data() (FAXSiteMover)    [added to 59a]
Change filename to surl argument in findGlobalFilePath() (FAXSiteMover)    [added to 59a]
Extracting filename from surl in findGlobalFilePath() (FAXSiteMover)    [added to 59a]
Created extractPattern() (pUtil)    [added to 59a]
Updated getGlobalFilePaths() to use extractPattern() for generating a global Rucio path using new convension (FAXSiteMover)    [added to 59a]
Redefined SITEROOT in getProperSiterootAndCmtconfig(), requested by Alessandro de Salvo (ATLASExperiment)    [added to 59a]
Created a new version of getProperSiterootAndCmtconfig(), still verifying cmtconfig but using redefined SITEROOT (ATLASExperiment)    [added to 59a]
Replaced call to findPythonInRelease() with "python" ["which python"] in setPython(), requested by Rodney Walker (ATLASExperiment)    [added to 59a]
Simplified getProdCmd3() using no path for the trf ["which "], requested by Rodney Walker (ATLASExperiment)    [added to 59a]
Fixed a problem with core file removal in removeCoreDumps(), wrong type, requested by Wolfgang Ehrenfeld (JobLog)    [added to 59a]
Displaying IP number of WN in collectMachineFeatures() (Node)    [added to 59a]
Created isJEMAllowed(), used from getJobExecutionCommand() (ATLASExperiment)    [added to 59a]
Created getSpecialAppdir(), used by extractAppdir() (ATLASSiteInformation)    [added to 59a]
Nightlies appdir extraction now uses getSpecialAppdir() (ATLASSiteInformation)     [added to 59a]
Getting appdir from VO_ATLAS_RELEASE_DIR for AtlasP1HLT and AtlasHLT jobs, using getSpecialAppdir in extractAppdir() (ATLASSiteInformation)    [added to 59a]
Added '2> /dev/null' to all uuidgen usages, since command threw stderr on some NG analysis jobs on slc6, requested by Andrej
Filipcic (Job, JobLog, Mover, pUtil, pilot)     [added to 59a]
Corrected findGlobalFilePath() and convertSURLtoTURLUsingDataset() for new Rucio file convention containing ":" (FAXSiteMover, Mover)    [added to 59a]
Created getVerifiedAtlasSetupPath(), used by useAtlasSetup() (ATLASExperiment)    [added to 59a]
Updated getProperSiterootAndCmtconfig() for special SITEROOT on AFS (needed for HLT) (ATLASExperiment)    [added to 59a]
Added HLT special case in getProdCmd2() (ATLASExperiment)    [added to 59a]
Added special setup for HLT jobs in getProperASetup() (ATLASExperiment)    [added to 59a]
Replaced 'dummy' with 'atlasProject' after checkCMTCONFIG() call in getJobExecutionCommand() (ATLASExperiment)    [added to 59a]
Using updateCmd1WithProject() in getJobExecutionCommand() (ATLASExperiment)    [added to 59a]
Created updateCmd1WithProject() (ATLASExperiment)    [added to 59a]
Extended extractFilePaths() to handle unevaluated env variables (pUtil)
Created new setup options for HLT jobs in getProperASetup() (ATLASExperiment)
Corrected LFNs containing rucio :-separators in createPoolFileCatalog() (pUtil)