SULU 58a

Summary

* Prevention of additional LFC lookups in runAthena/runGen by doing SURL

to TURL conversion already in the pilot, to alleviate problems on SL6

queues due to mangled LD_LIBRARY_PATHs. Note that this has revealed

another problem seen on US sites because of missing libdcap.so (issue

currently being discussed in US ProdSys list ­ using direct access on such

queues [using dcap door] will cause payload error 40 "Athena crash ­

consult log file"; error opening file)

* Added --file $X509_USER_PROXY to voms-proxy-info to avoid problems on

SL6 queues with this command, requested by Alessandra Forti

* Converting "HH:MM:SS" format to "SS" for EMI-3 systems to sort out a

problem with "voms-proxy-info ­actimeleft" (returning a time string with

unexpected format)

* Solved a problem with file names containing the __DQ2-bit (sub string

removed from LFN prior to TURL based PFC creation in the pilot which

previously led to  payload error 40 "Athena crash ­ consult log file";

Tadashi Maeno updated runGen correspondingly)

* Optimization of replica lookup in LFC host loop (recently implemented

for FAX needs). In previous pilot versions this loop was not aborted

properly even when a proper replica was found leading to unnecessary LFC

lookups (in the case FAX was enabled).

* Pilot now runs a CVMFS diagnostics tool (checkValidity.sh),  requested

by Asoka De Silva

* Correcting for potentially missing sepaths in Rucio paths

* Now identifying _CONDOR_JOB_AD in getBatchSystemJobID(). Requested by

David Rebatto

* Removed --ciphers options to curl in core_get_data(). Requested by

Jhen-Wei Huang

* Added exceptions list used with list of files and directories to be

removed prior to log file creation. Requested by Wolfgang Ehrenfeld. Also

added "geant4" to list of directories to be removed

* Further refactoring for Generic PanDA Pilot (queue data handling,

software directory verification, interpretation of payload stdout errors,

and merging with latest CMS classes), after requests from CMS (Valentina

Mancinelli)

Version info

Forcing usePFCTurl in getFileTransferInfo() (Experiment)
Updated usage of usePFCTurl in getAnalysisRunCommand() (Experiment)
Added fax_mode to getLFCFileList() (Mover)
Now using fax_mode in getLFCFileList() (Mover)
Added fax_mode logic to getPoolFileCatalog() (Mover)
Updated shouldPFC4TURLsBeCreated() to always create TURL based PFC for direct access (and empty old/newPrefix) (Mover)
Removing "?svcClass" etc from TURLs/PFNs in updatePFN() if present to prevent problem in runGen/runAthena (Mover)
Added --file $X509_USER_PROXY to verifyProxy(), dumpExtendedProxy() (SiteMover)
Converting from "HH:MM:SS" format to SS for EMI-3 systems in verifyProxy() (SiteMover)
Removed the allowdirectaccess requirement in shouldPFC4TURLsBeCreated() (Mover)
Forced usePFCTurl in getFileTransferInfo(), reset old/newPrefix (Experiment)
Testing CVMFS in specialChecks(), requested by Asoka De Silva (ATLASExperiment)
Created testCVMFS() (ATLASExperiment)
Changed "debugon" to "debug" in updateJobs() (pilot)
No setup string in getTURLs() for lcg-getturls (Mover)
Correcting for potentially missing sepaths in dst_gpfn, in getFinalLCGPaths() (SiteMover)
Now identifying _CONDOR_JOB_AD in getBatchSytemJobID(), requested by David Rebatto (pUtil)
Removed --ciphers options to curl in core_get_data(), requested by Jhen-Wei Huang (curlSiteMover)
Relaxed warning message in getMatchingDestinationPath() (SiteMover)
Added exception to exeErrorDiag "OK" message in postJobTask() (JobLog)
Added file exceptions list to removeRedundantFiles(), ie files that will not be removed during workdir cleanup. Requested by
Wolfgang Ehrenfeld (ATLASExperiment)
Moved stripDQ2FromLFN() from Mover to pUtil
Now removing any __DQ2-parts from the LFN prior to TURL based PFC creation in createPoolFileCatalog() (pUtil)
Created convertSURLtoTURL() (Mover)
Calling convertSURLtoTURL() from getTURLs() (alternative to using lcg-getturls) (Mover)
Added -b T srmv2 to lcg-getturls in getTURLs() (Mover)
Created updatePandaLogger(), code from Ilija Vukotic (PandaServerClient)
Calling updatePandaLogger() from updatePandaServer() (PandaServerClient)
Moved getQueuedata() to ATLASSiteInformation (SiteInformation, ATLASSiteInformation)
Changed processQueuedata() call in prepareAlternativeStageOut() to getQueuedata() (Mover)
Changed processQueuedata() call in handleQueuedata() to getQueuedata() (pilot)
Merged processQueuedata() with getQueuedata() (ATLASSiteInformation)
Removed processQueuedata() since it is no longer needed (SiteInformation)
Added geant4 to directory list in removeRedundantFiles() (ATLASExperiment)
Updated to the latest versions of CMSExperiment and CMSSiteInformation
Renamed rmRedundants() to removeRedundantFiles() (CMSExperiment)
Now using getSiteInformation() in displayChangeLog() (ATLASExperiment)
Using verifySwbase() from experiment class in handleQueuedata() (pilot)
Calling interpretPayloadStdout() from interpretPayload() using experiment object (ErrorDiagnosis)
Moved interpretPayloadStdout() from ErrorDiagnosis to *Experiment classes