* Report to consistency server now also includes missing files, i.e. in
case of "No such file" error during stage-in. Requested by Cedric Serfon
* Added support for updating space check frequency (remaining disk space +user workdir) via env variables NON_LOCAL_ATLAS_SCRATCH and
NON_LOCAL_ATLAS_SCRATCH_SPACE to alleviate problems with pilot's use of
"du" and "find" when there are many files generated by the payload. Problem seen at MWT2 when even jobs generated ~20k files (Wolfgang patched the jobs). Requested by David Lesny
- NON_LOCAL_ATLAS_SCRATCH=True
- NON_LOCAL_ATLAS_SCRATCH_SPACE=[1,10] n multiplier, "n" means that the
pilot will check the space every 10*n minutes
* Removed setting of env variables http[s]_proxy from the pilot (used
at SLAC, no longer needed). Requested by Wei Yang
* Added time-out to CVMFS test function. Requested by Rodney Walker
* Updated proxy validation function to primarily use arcproxy instead
of voms-proxy-info. Requested by Alessandra Forti et al. Several problems recently with voms-proxy-info (java under SL6 related). Decision from ADC Weekly meeting on Sep 17 to replace the command
* Pilot now uses arc proxy to get remaining proxy time, in case of command failure (other than out of time) the pilot defaults to voms-proxy-info. Currently investigating library issues (command fails on multiple sites)
* Added support for env variables in schedconfig.appdir. E.g. appdir =
"$VO_ATLAS_SW_DIR|nightlies^$VO_ATLAS_SW_DIR/../../../
atlas-nightlies.cern.ch/repo/sw/nightlies²
* Added support for new env variable VO_ATLAS_NIGHTLIES_DIR which
is/will be the default location for nightlies. Requested by Alexander
Undrus et al
* Added new pilot option G . Requested by
Fredrick Luehring et al. Pilot can now keep running asking for
jobs even if there are no available jobs at the moment.
Note: setting is on pilot wrapper level, not schedconfig
* Importing JobLog early in pilot to prevent later problem with missing
module seen on some sites (e.g. at EELA-UTFSM). Pilot modules gets deleted by some unknown process during running, a late import would trigger a crash leading to Lost Heartbeat. Fix tested with 100s of long (12h+) running jobs
* Corrected a problem with Rucio paths in complex se[prod]paths (seen
during testing at CERN) leading to stage-out failure. Requested by
Rodney Walker
Note: the "added to 59a" indicates that the same change has been added to the next major pilot version, PICARD 59a, also in development.
Added test jobState file to removeRedundantFiles() list (ATLASExperiment) [added to 59a]
Added ERR_NOSUCHFILE to file corruption report in mover_get_data(), requested by Cedric Serfon (Mover) [added to 59a]
Added support for updating space check frequency via env variable in setUpdateFrequencies(), requested by David Lesny (pilot) [added to 59a]
Removed call to setHTTPProxies (pilot) [added to 59a]
Removed function setHTTPProxies (pUtil) [added to 59a]
Removed unneeded code and comments about http[s]_proxy (runJob) [added to 59a]
Removed http[s]_proxy from env variable list (atlasProdPilot) [added to 59a]
Added timedCommand() to testCVMFS(), requested by Rodney Walker (ATLASExperiment) [added to 59a]
Created interpretProxyInfo() (SiteMover) [added to 59a]
Updated verifyProxy() to primarily use arcproxy instead of voms-proxy-info (SiteMover) [added to 59a]
Renamed all s11 to exitcode and o11 to output (SiteMover) [added to 59a]
Created getDN() (pilot) [added to 59a]
Trying to use arcproxy before voms-proxy-info in getDN() used in getDispatcherDictionary() (pilot) [added to 59a]
Added appdir to fields list in evaluateQueuedata() (SiteInformation) [added to 59a]
Now handling |-signs in appdir, in evaluateQueuedata() (SiteInformation) [added to 59a]
Added hardcoded setup path for arcproxy in verifyProxy() (move to wrapper) (SiteMover) [added to 59a]
Added new global variable getjobmaxtime (pilot) [added to 59a]
Added new pilot option -G
Reading -G option in argParser() (pilot) [added to 59a]
Rewrote getJob() to download jobs for a maximum time instead of maximum number of trials (pilot) [added to 59a]
Importing JobLog early in pilot to prevent later problem on some sites (e.g. at EELA-UTFSM) (pilot) [added to 59a]
Removed getProperPaths() (SiteMover) [added to 59a]
Added experiment variable in put_data() (aria2cSiteMover) [added to 59a]
Added si object in put_data() (aria2cSiteMover) [added to 59a]
Now using si object instead of self when calling getProperPaths() (aria2cSiteMover) [added to 59a]
Added token, prodSourceLabel arguments to getFinalLCGPaths() call in getProperPaths() (ATLASSiteInformation) [added to 59a]
Added token, prodSourceLabel arguments to getFinalLCGPaths() (SiteMover) [added to 59a]
Now getting the proper sepath using getPreDestination() in getFinalLCGPaths() (SiteMover) [added to 59a]
Changed the return error code in case of value exception in interpretProxyInfo() (SiteMover) [added to 59a]
Improved exit code handling from interpretProxyInfo() in verifyProxy() (SiteMover) [added to 59a]
Adding missing /CN=proxy to DN string if necessary in getDN() (pilot) [added to 59a]
Added "runwrapper", "jobReport", "log." to exclusion list in removeRedundantFiles(), requested by Graeme Stewart (ATLASExperiment) [added to 59a]
Added extractAppdir() to ATLASSiteInformation [added to 59a]
Now using new environmental variable VO_ATLAS_NIGHTLIES_DIR for nightlies in extractAppdir() (ATLASSiteInformation) [added to 59a]
No comments:
Post a Comment