* Removed LFC dependence in skipping of locally available DBRelease. Pilot
uses DBRelease file from CVMFS area when available, but previously the
file still needed to be registered in LFC. If it wasn¹t, the job would
fail with an irrelevant LFC error. Now fixed
* New convention for user job LFC directories. Old:
/grid/atlas/users/pathena/user.username/datasetname -> New:
/grid/atlas/users/pathena/user/username/[field3]/datasetname as suggested
by Ikuo Ueda et al.
* Stage-out retry with a second attempt after 60 s. Default number of
retries = 2, can be changed with pilot option E
* Server-side retries for user analysis jobs, triggered by retry error
from pilot. Same pilot error code as before but negated (server receives
negative pilot error code -> retry job on same site). The current list of
retry errors (see presentation from software week) currently contains 22
error types. We are testing lowering the priority (with 10) of the retried
jobs to lower the probability that the retried job will fail due to the
same problem as the initial attempt. The first tests showed that the
retried job were executed too quickly after the first attempt, and thus
suffered the same problem. In collaboration with Gabriele Compostella and
Tadashi Maeno
* Left-over RAW file in pilot log now removed. Requested by Alexander
Vaniachine. Also removed 'home' and '*.data*' from the log
* Using 1.33 instead of 6 as default throttle parameter for CERN cloud.
Added max throttle time protection. Requested by Rodney Walker
* Increased max file size from 50 MB to 200 MB in the Chirp site mover.
Requested by Rodney Walker
* Major code cleanup using pychecker of roughly half of the pilot code
Version info
Using local variable _uflag instead of global in getProdSourceLabel(), getJob(), getNewJob(), diskCleanup() (pilot)
Using local variables _pshttpurl, _queuename instead of globals in handleQueuedata() (pilot)
Using local variable _jobrec instead of global in postProcessQueuedata() (pilot)
Using local variable _psport instead of global in runJobRecovery(), RecoverLostJobs() (pilot)
Corrected misuse of global prodJobDone in pollChildren() (pilot)
Removed unused psport, ra, thisWorkNode, latereg variables from moveLostOutputFiles() and its calls (pilot)
Created backupDispatcherResponse(), getDispatcherDictionary() (pilot)
Replaced deprecated string.atoi with int in handle() and removed string import (pilot)
Removed unused sitename variable from checkLocalDiskSpace() and its call (pilot)
Renamed isAnalJob() to isAnalysisJob() (pUtil)
Replaced all occurances of isAnalJob() with isAnalysisJob() (JobLog, JobRecovery, pilot, runJob, RunJobUtilities)
Removed unnecessary import of isAnalJob (PandaServerClient)
Removed unused atlasRelease variable from getTag() and its call (RunJobUtilities)
Out-commented unused dotver, pth variables in installPyJobTransforms() (RunJobUtilities)
Removed re-imported socket module from queryActivationSvc() (RunJobUtilities)
Removed unused fail variable from setupTrf() (RunJobUtilities)
Now issuing a warning if release was not found in tags file in setupTrf() (RunJobUtilities)
Removed unused rf variable from updateJobInfo(), updatePilotServer() and its call (RunJobUtilities, runJob)
Corrected jobsite -> jobSite in updateRunCommand4JEM() (RunJobUtilities)
Corrected rest -> rets in downloadTrf() (RunJobUtilities)
Removed unused cmd1, cacheDir, cacheVer from updateAnalyCmd2() and its call (RunJobUtilities)
Corrected bad file close in getLFNsFromSkippedXML() (RunJobUtilities)
Created addFRONTIERVars() (RunJobUtilities)
Removed unnecessary prodSourceLabel from isSameType() and its calls (pUtil, JobRecovery, pilot)
Removed import of isSameType (Cleaner)
Removed repeated sys import (Mover)
Attempting to import md5 from hashlib (Mover)
Removed unused region, tin_0, skipForAnalJobs variables from get_data() (Mover)
Removed unnecessary code related to envsetup in getInitialTracingReport() (Mover)
Corrected number of return fields during exception in getFileInfoFromMetadata() (Mover)
Replaced unused variable o with variable output for initialization in getTURLs() (Mover)
Removed unused variables replicas_dic, first in mover_get_data() (Mover)
Corrected timeout in put_data() call in chirp_put_data() (Mover)
Removed unnecessary SiteMover import in mover_put_data() (Mover)
Removed unused error variable in getPoolFileCatalog() (Mover)
Removed unused dsname, dsdict arguments in getPoolFileCatalogDQ2() and its call (Mover)
Removed unused _maxinputsize in verifyAvailableSpace() and its call (Mover)
Removed unnecessary transfer info messages in put_data() (lcgcpSiteMover)
Removed unused token variable in get_data() (lcgcp2SiteMover)
Removed unnecessary code related to _setup_str in put_data() (lcgcp2SiteMover)
Removed unused dst_se variable in put_data() (lcgcp2SiteMover)
Removed unused string, random modules from global import (runJob)
Removed unused global e variable (runJob)
Renamed variable file to _file in cleanup() (runJob)
Out-commented currently unused read_write_exec_req variable in executePayloadGLExec() (runJob)
Replaced pinitdir with pilot_initdir in stageOut() (runJob)
Replaced proxycheck with proxycheckFlag in stageOut() (runJob)
Removed re-import of traceback and newJobDef (runJob)
Replaced deprecated string.atoi calls with int in setJobDef() (Job)
Removed unused code using datetag, newdatetag, _date in updateMetadata() (pUtil)
Removed unused date argument in updateMetadata() and its call (pUtil, JobLog)
Corrected uninitialized node variable in updatePandaServer() for fake server updates (PandaServerClient)
Removed unused sendAthenXML variable in updatePandaServer() (PandaServerClient)
Out-commented currently unused jobAtlasRelease variable in updatePandaServer() (PandaServerClient)
Replaced deprecated string.atoi call with int in updatePandaServer() (PandaServerClient)
Removed all occurances of unnecessary new_metadata4 variable in updateMetadata() (pUtil)
Removed unused worknode variable from setTimeConsumed() and its call (pUtil, runJob)
Removed unused sendproxy argument from httpConnect() and its call (pUtil, pilot)
Removed use of sendproxy from getNewJob() since it is no longer needed (pilot)
Removed unused pshttpurl argument from getQueuedata() and its call (pUtil)
Removed unused sitename argument from getQueuedata() and its call (pUtil)
Removed unused pshttpurl argument from processQueuedata() and its call (pUtil, pilot)
Removed unused sitename argument from processQueuedata() and its call (pUtil, pilot)
Removed unused args0 argument from processQueuedata() and its call (pilot)
Out-commented unused dst_host variable in put_data() (xrdcpSiteMover, xrootdSiteMover)
Removed unused error variable in copy() (xrdcpSiteMover)
Created getStubTracingReport() (SiteMover)
get_data() is now using getStubTracingReport() (xrdcpSiteMover, LocalSiteMover)
Removed test string for python version in lsg-df command (LocalSiteMover)
Added support for schedconfig.lfcregister in handleQueuedata() (pilot)
Added filesInDQ2Dataset data member to SiteMover class (SiteMover)
Created getFileInDataset(), getFileInfoFromDQ2() (SiteMover)
getFileInfo() is now using getFileInfoFromDQ2() (Mover)
Changed order of file size and checksum comparisons in get_data() and put_data() (xrootdSiteMover, xrdcpSiteMover)
Improved import of md5 module (Mover)
Removed exception string from print-out after calling getDQ2SiteName in put_data() since string cast failed at Taiwan (lcgcpSiteMover)
Removed exception string from print-out in isDPMSite() (Mover)
Added try-statement to getDQ2SiteName() (SiteMover)
Added "home" to rmRedundants() list (seen in jobs at UK.UKI-NORTHGRID-MAN-HEP) (JobLog)
Added "*.data*" to rmRedundants() list (such output files were left in the job logs) (JobLog)
Removed redundant rmRedundants() declaration (pilot)
Moved dbh definition from get_data() to mover_get_data(), removed dbh from function call (Mover)
Created isDBReleaseAvailable(), createdSkeletonDBRelease(), getDBReleaseVersion(), handleDBRelease(), createFileDicsForLFC() (Mover)
Rewrote DBRelease handling code in mover_get_data() (Mover)
Added dbh, DBReleaseIsAvailable argument to getFileInfo() and its call in mover_get_data() (Mover)
Added dbh, DBReleaseIsAvailable argument to getPoolFileCatalog() and its call in getFileInfo() (Mover)
Modified stageInPyModules() to exit with an error code (pilot, pUtil)
Added max throttle time protection in throttleJob(), requested by Rodney Walker (pilot)
Using 1.33 instead of 6 as default throttle parameter for CERN cloud (pilot)
Created filterSE() [used by getProperSE()] (SiteMover)
Using new convention for user job paths in getFinalLCGPaths() (SiteMover)
Removed old to_native_lfn() and renamed new to_native_lfn2() to to_native_lfn() (SiteMover)
Added try-statements around PFCxml() calls in stageOut() and createFileMetadata() (runJob)
Added timed_command to xrdcpSiteMover
Rewrote copy function in xrdcpSiteMover to handle time-outs
Calls to copy function now uses stagein argument in get_data() (xrdcpSiteMover)
Negated useFileStager in shouldPFC4TURLsBeCreated() (Mover)
Added isPilotResubmissionRequired to Job class (Job)
Added PilotResubmissionErrorCodes and isPilotResubmissionErrorCode() (PilotErrors)
Moved updateRunCommand4JEM() from setupTrf() to JEMstub module (RunJobUtilities)
New JEMstub module from Tim
Added pfnList handling in setJobDef() (Job)
Increased max file size from 50 MB to 200 MB (ChirpSiteMover)
Calling removeFile() after failed addFileInfo() in put_data() (lcgcpSiteMover) [see job 1335901247]
Added missing D: in getopt() call in argParser() (pilot)
Added new global stageoutTries used in argParser(), dumpVars(), runMain(), transferLogFile(), postJobTask() (pilot)
Added new global stageoutTries used in argParser(), stageOut() (runJob)
Added new argument stageoutTries to mover_put_data() (Mover, pilot)
Added stageoutTries handling to mover_put_data() (Mover)
Added new data member stageoutTries to JobLog class, used by __init__(), transferLogFile(), transferAdditionalFile() (JobLog)
Added new argument stageoutTries to __init__() (JobLog)
Added missing useCoPilot argument in postJobTask() (pilot)
Downgraded string call to support python 2.3 in filter_text() (dCacheSiteMover)
No comments:
Post a Comment