* Memory monitor utility update
- Restarting the monitor in case of crash
- Memory info set to -1 in case of file reading problem or if no update
within 120 s
* Support for multiple clones of a job
- Idea: reduce latency for high-priority tasks (Rod Walker)
- Supporting modes 'runonce' (before payload is executed: should events be
processed?) and 'storeonce' (before stage-out: should output be
transferred?)
* Monitoring and reporting of payload work directory size
- Pilot measures the size of the work directory every ten minutes (or once
before creating the log for jobs shorter than 10 minutes)
- At the end of the job, the maximum size is reported with jobMetrics
(workDirSize)
- Total size of input + output files (whatever files are present during
measurement) is deducted. Total input file size is deducted since optional
direct i/o mode might otherwise complicate matters. Total output file size
is deducted since 'outputfilesize' is anyway added in the task definition
* Now setting job status to failed when stage-out is stuck
- Previously missed and led to failed jobs being stuck in transferring
state until later time-out (these jobs will always fail in the end, but
will take an unnecessarily long time
to do so)
- Requested by Rod Walker
* FAX updates
- In forced FAX mode (i.e. FAX as primary stage-in tool as opposed to
fallback mechanism), the pilot now skips catalog lookups of replicas
requested by Tadashi Maeno
- Update to correctly handle file paths for derived dataset (scope
extraction method was outdated), requested by Tadashi Maeno and Ilija
Vukotic
* Event Service updates
- Failing job in case of Token Extractor crash
- Correction of failure message (job status ³Failed² -> ³failed²)
- Updated payload stdout file name (athenamp_std*.txt -> athena_std*.txt)
to bypass a harmless logging problem (requested by Sasha Vanyashin)
* Preparations for benchmark tests
- Idea: pilot runs a quick (~seconds) benchmark test before downloading a
job
- Results will be reported with jobMetrics
- Currently running condor kflops script written by Rod Walker from afs
(will be moved to dedicated cvmfs area; other benchmark tools will be
tested)
- Implementation not finalized - benchmarking is not activated in this
pilot version
* glExec update from Edward Karavakis
- Patch for chmod problem seen at BNL
* Development of new Site Mover structure by Alexey Anisenkov
- Part of long-term pilot development; Modernization and improvements of
ancient Site Mover classes
- Will offer huge simplification of current usage of schedconfig fields
- Current status
-- Structure of base class created
-- xrdcp stage-in developed, stage-out pending
-- Test code part of this pilot release, can be enabled through AGIS
Added ddmEndPointIn, ddmEndPointOut, ddmEndPointLog to Job class, and handling in displayJob() and setJobDef() (Job)
Moved exception handling for utility subprocess from executePayload() to getUtilitySubprocess() (RunJob)
Making sure that the subprocess is still running in executePayload() (RunJob)
Setting memory info = -1 in case of file reading trouble or if memory summary file was not updated in less than 120 s in getUtilityInfo(). Req\
ested by Rodney Walker (PandaServerClient)
Added error code 1234, ERR_EXECUTEDCLONEJOB: "Already executed clone job" (PilotErrors)
Now using event range functions to determine whether a cloned job should be executed in executePayload() and __main__() (RunJob)
Moved extractEventRanges() from RunJobEvent to RunJob (RunJob, RunJobEvent)
Setting ATHENA_PROC_NUMBER to 1 on single code nodes in __main__() (RunJobEvent)
Removed test method updatePilotErrorReport() and renamed updatePilotErrorReportTMP() to updatePilotErrorReport() (FileHandling)
Now recognizing a cvmfs time out in getProperSiterootAndCmtconfig() (ATLASExperiment)
Using updatePilotErrorReport to set the error priority in getJobExecutionCommand() (ATLASExperiment)
Created optional getGUIDSourceFilename() (Experiment, ATLASExperiment)
Added experiment argument to getOutFilesGuids() (RunJobUtilities)
Now sending experiment name to getOutFilesGuids() in createFileMetadata() (RunJob)
Importing getExperiment, getGUID (RunJobUtilities)
Created shouldExecuteBenchmark() and executeBenchmark() (SiteInformation, ATLASSiteInformation)
Moved getSiteInformation() to collect WN info section in runMain() (pilot)
Created getBenchmarkDictionary() (Node)
Now getting the benchmark dictionary in runMain() (pilot)
Now setting job to failed in __killLoopingJob() for stuck stage-outs (Monitor)
Added optional ddmEndPointIn argument to getPoolFileCatalog() (Mover)
Created buildFAXPath() (Experiment)
Sending optional computingSite and sourceSite to getFileInfo() from mover_get_data() (Mover)
Added optional computingSite and sourceSite arguments to getFileInfo(), getPoolFileCatalog() (Mover)
Sending optional computingSite and sourceSite to getPoolFileCatalog() from getFileInfo() (Mover)
Created FAXTools module
Moved _getFAXRedirectors() and getFAXRedirectors() from FAXSiteMover to FAXTools
Moved updateRedirector() from FAXSiteMover to FAXTools
Sending ddmEndPointIn to getPoolFileCatalog() from getFileInfo() (Mover)
Added optional argument ddmEndPointIn to getPoolFileCatalog() (Mover)
Sending ddmEndPointIn to getFileInfo() from mover_get_data() (Mover)
Added optional argument ddmEndPointIn to getPoolFileCatalog() (RunJobEvent)
Sending ddmEndPointIn to Mover::getPoolFileCatalog() from RunJobEvent::getPoolFileCatalog() (RunJobEvent)
Added argument ddmEndPointIn to createPoolFileCatalog() (RunJobEvent)
Sending ddmEndPointIn to Mover::getPoolFileCatalog() from RunJobEvent::createPoolFileCatalog() (RunJobEvent)
Sending job.ddmEndPointIn to createPoolFileCatalog() from __main__() (RunJobEvent)
Sending ddmEndPointIn to getPoolFileCatalog() from createPoolFileCatalogFromMessage() (RunJobEvent)
Updated findGlobalFilePath() and getGlobalFilePaths() to accept already defined FAX paths (i.e. taken from PFC) (FAXSiteMover)
Renamed file_list to file_dictionary in createPoolFileCatalog() (pUtil)
Created storeWorkDirSize(), used by __checkWorkDir() (FileHandling, Monitor)
Created getMaxWorkDirSize(), used by addToJobMetrics() (FileHandling, Node)
Created getWorkDirSizeFilename() used by getMaxWorkDirSize() and storeWorkDirSize() (FileHandling)
Added restart of utility subprocess and aborting job if token extractor has crashed at the end of the event range loop in __main__() (RunJobEv\
ent)
Aborting immediately if Token Extractor has died instead of waiting for AthenaMP to finish, in __main__() (RunJobEvent)
Changed waiting time from 600 to 30, in __main__() (RunJobEvent)
Removed outdated usages of dsname in getGlobalFilePaths() and replaced with scope which replaces dsname as an argument (FAXSiteMover)
Sending scope instead of dsname to getGlobalFilePaths() from findGlobalFilePath() (FAXSiteMover, SiteMover)
Added scope argument to findGlobalFilePath() (FAXSiteMover, SiteMover)
Sending scope to findGlobalFilePath() from get_data() (FAXSiteMover)
Extracting scope in get_data() (FAXSiteMover, aria2cSiteMover)
Changed scope_dict to scope and sending scope_dict[lfn] instead of scope_dict to get_data(), in sitemover_get_data() (Mover)
Removed outdated scope extraction from get_data() (aria2cSiteMover)
Sending scope instead of dataset to getGlobalFilePaths() from convertSURLtoTURLUsingDataset() (Mover)
Renamed convertSURLtoTURLUsingDataset() to convertSURLtoTURLUsingScope() (Mover)
Sending scope instead of dataset to convertSURLtoTURLUsingScope() (also renamed, see previous comment) from convertSURLtoTURL() (Mover)
Added scope argument to convertSURLtoTURL() (Mover)
Sending scope to convertSURLtoTURL() from getTURLs() (Mover)
Sending scope_dict to PFC4TURLs() from mover_get_data() (Mover)
Added scope_dict argument to PFC4TURLs() (Mover)
Sending scope_dict to createPFC4TURLs() from PFC4TURLs() (Mover)
Added scope_dict argument to createPFC4TURLs() (Mover)
Sending lfns and scope_dict to getTURLs() from createPFC4TURLs() (Mover)
Added lfns and scope_dict arguments to getTURLs() (Mover)
Created getMatchingLFN() used by getTURLs() (Mover)
Sending scope_dict to PFC4TURLs() from getPoolFileCatalog() (RunJobEvent)
Corrected 'Failed' -> 'failed' message sent with updateEventRange() in listener() (RunJobEvent)
[Out-commented call for now] Moved discoverAdditionalOutputFiles() from RunJob to FileHandling, added import and removed a comment
Sending scope to findGlobalFilePath() from mover_get_data() (Mover)
Renamed __getMaxWorkDirSize() to __getMaxAllowedWorkDirSize(), used by __checkWorkDir() (Monitor)
Using getDirSize(), getWorkDirSizeFilename() and storeWorkDirSize() in createLogFile() (JobLog)
Now returning xml_source from getFileInfo(), received in mover_get_data() (Mover)
Skipping call to PFC4TURLs() in mover_get_data() in FAX mode (Mover)
Renamed athenamp_std*.txt to athena_std*.txt in __main__() (RunJobEvent)
Code merge with Alexey Anisenkov
- Added new movers sub directory with new site mover (xrdcp) and mover module
Update info from Edward Karavakis:
- Patch for chmod problem seen at BNL
Update info from Wen Guan:
- Added exceptions for ptest uflag in getProdSourceLabel() (pilot)
Non-ATLAS related changes:
- Now generating output file GUIDs in getOutFilesGuids() in case PoolFileCatalog.xml is not to be used (e.g. for AMS) (RunJobUtilities)
No comments:
Post a Comment