* New setup function for prod jobs (also for analy jobs in the next release, or so) using atlasLocalSetup
* General code cleanup (Mario Lassnig)
* ARGO updates (Danila Oleynik, Taylor Childers)
- New ARGO pilot modules, serialization of ARGO messages
* New site movers for testing (Alexey Anisenknv, Mario Lassnig); xrdcp, lcgcp, dcache, rucio
* EventStager for NERSC, multi-jobs (Wen Guan)
* Memory monitoring updates
- Note: Current pilot version does not kill when max memory has been exceeded, only monitors it every minute and updates the server during regular heartbeats
* Alternative stage-out updates (both for normal SE:s and OS:s)
- Pilot now reports alt stage-outs to normal SE:s with jobMetrics
- Alternative OS stage-outs reported with event range updates (i.e. for ES) and logs are reported with jobMetrics. Failover to BNL OS for all OS:s - BNL failover to CERN
- WORLD cloud update
* Added time-outs to log file creation + explicit removal of *.a files (pilot does symbolic link dereference at the log creation which can cause problems with e.g. MadGraph jobs)
* MAKEFLAGS update
- Now also set for generic jobs (i.e. jobs with no ATLAS release specified - caused problems for some users)
- Added for ND jobs (previously set by script on the ND WN)
* New job recovery for testing
- Not activated in current version – testing/development not ready (Daniel Driezshuk)
General changes:
- Replaced useFileStager and directIn with _dummy strings in getPrefices() and shouldPFC4TURLsBeCreated() since they are not used in these functions (Mover)
- Now getting the value of directIn from useDirectAccessLAN() instead of from getFileAccessInfo() in shouldPFC4TURLsBeCreated() (Mover)
- Replaced useFileStager and directIn with _dummy in stageIn() since they are not used (RunJob)
- Removed import of getFileAccessInfo() since it is not used (RunJobEvent)
- Created getDirectAccess(), used by shouldPFC4TURLsBeCreated() (Mover)
- Simplified getFileAccessInfo() (pUtil)
- Updated return values from getFileAccessInfo() in getPrefices() and shouldPFC4TURLsBeCreated() (Mover)
- Updated return values from getFileAccessInfo() in stageIn() (RunJob)
- Now using useDirectAccessWAN() in getPoolFileCatalog() to determine if a PFC with direct I/O over FAX should be produced (Mover)
- Now failing immediately if PFC could not be created in getPoolFileCatalog() (Mover)
- Simplified willDoFileLookups() (ATLASExperiment)
- Avoiding None value while getting number of events in processMetadata() (ATLASExperiment)
- Ignoring test mode in updateJobStateTest() (JobRecovery)
- Removing archive files if present prior to log file creation; requested by Emmanouil Vamvakopoulos et al (ATLASExperiment)
- Cleaned up reporting of transferred file state in mover_put_data_new() and mover_put_data() (Mover)
- Changed headPilotErrorDiag() to headPilotErrorDiag() to avoid annoying tailed error diagnostics, in get_data() and changed import (Mover)
- Created headPilotErrorDiag() (pUtil)
- Now using headPilotErrorDiag in makeJobReport() (pUtil)
- Replaced all usages of type as a variable and replaced it with ftype (FileState, FileStateClient, all site movers, Monitor, RunJob*, mover)
- Removed obsolete and unused lateRegistration() (pUtil)
- Removed duplicated, obsolete and unused getField(), getNewQueuedataFilename() (FileHandling)
- Now using dictionary get function instead of has_key() in stageIn() (RunJob)
- Removed 60s initial delay in killProcesses() since MemoryMonitor is now the first process that will be killed (processes)
- Using createLockFile() in __checkPayloadStdout(), __checkWorkDir(), __checkLocalSpace(), __failMaxTimeJob(), __killLoopingJob(), __updateJobs(),
__cleanUpEndedJobs() and in Monitor main exception handling to prevent MemoryMonitor to restart while the job is being killed (Monitor)
- Checking that JOBWILLBEKILLED lock file does not exist before restarting MemoryMonitor in executePayload() (RunJob)
To avoid creating dark data in case the server has already failed the job before the pilot has finished
The problem can happen if the server times out the job before tar+zip of huge log files have not finished within two hours
- Added 55 minute time-outs each around tar+zip of log file in createLogFile() (JobLog)
Updates to alternative stage-out mechanism (related to WORLD cloud and jobMetrics reporting)
- Updated getTier1Queue() to handle WORLD cloud for alternative stage-out (ATLASSiteInformation)
- Removed unused getTier1Queue2() (ATLASSiteInformation, AMSTaiwanSiteInformation)
- Now sending token to prepareAlternativeStageOut() from mover_put_data() (Mover)
- Added token argument to prepareAlternativeStageOut() (Mover)
- Sending token to getTier1Queue() from prepareAlternativeStageOut() (Mover)
- Reporting alt_transferred file state in mover_put_data_new() and mover_put_data() (Mover)
- Created getFilesOfState() (FileState, FileStateClient)
- Using getFilesOfState() in getJobMetrics() (PandaServerClient)
- Added --quiet to getLocalEMISetup to avoid interrupting JAVA warnings (ATLASSiteInformation)
- Sending site to getJobMetrics() from getNodeStructure() (PandaServerClient)
- Added site argument to getJobMetrics() (PandaServerClient)
General bug fixes:
- Corrected bad return (function returned pilotErrorDiag string, but should have returned an outputRet dictionary) in errorToReport(); caused timed-out jobs at BNL
to get the wrong error label (LocalSiteMover)
- Corrected usage of new method processCanBeKilled() (missing self; also in its call in __updateJobs()) (Monitor)
- Removed half-implemented getFilenamesAndGuid() (Mover)
- Changed 'eventservice' to 'logs' in transferLogFile() before adding info to OS transfer dictionary (JobLog)
- Added missing MAKEFLAGS for ND jobs, in setupNordugridTrf() (NordugridATLASExperiment)
- Improved handling of lcg-get-checksum output in case of irrelevant warnings in lcgGetChecksum()
- Added missed file state (no-transfer; for DBRelease files) in hasOnlyCopyToScratch(). When prodDBlockToken had 'local' values --directIn was wrongly sent to runAthena (FileState)
Multi-core update for generic jobs
- Now setting MAKEFLAGS for generic analysis jobs (i.e. analysis jobs with swRelease = 'Atlas-*') in getJobExecutionCommandOld() (ATLASExperiment)
Setup of ATLAS Release changes
- Created a new getJobExecutionCommand(), renamed old function to getJobExecutionCommandOld() (ATLASExperiment)
- Changed useAtlasSetup() to always return True for all releases (ATLASExperiment)
- Updated getSplitHomePackage() to handle AnalysisTransforms-AtlasP1HLT_* correctly (ATLASexperiment)
- Now using getSplitHomePackage() in getJobExecutionCommand() (ATLASExperiment)
Objectstore updates
Added fail-over to alternative OS
- Created copyFullQueuedata(), getFullQueuedataDictionary(), findOSEnabledQueuesInFullQueuedata(), findAllObjectStores(), getAlternativeOS(), getFu\
llQueuedataFilePath() (SiteInformation, ATLASSiteInformation)
- Now using getNewOSStoragePath() in mover_put_data() to get updated path and os_id (Mover)
OS identification
- Created getOSIDFromName() (SiteInformation)
- Returning the os_bucket_id in getObjectstorePath() (SiteInformation)
- Receiving and returning the os_bucket_id in getLogPath() (JobLog)
- Receiving the os_bucket_id in transferActualLogFile() (JobLog)
- Now receiving and returning os_bucket_id from getDDMStorage() (Mover)
- Receiving os_bucket_id in mover_put_data() and mover_put_data_new() (Mover)
- Receiving os_bucket_id in getFileInfo() (Mover)
- Sending os_bucket_id to mover_put_data() from transferActualLogFile() (JobLog)
- Added os_bucket_id argument to mover_put_data() and mover_put_data_new() (Mover)
- Sending os_bucket_id to getDDMStorage() from mover_put_data() and mover_put_data_new() (Mover)
- Sending os_bucket_id to getObjectstorePath() from ..
- Added os_bucket_id argument to getObjectstorePath() (SiteInformation)
- Sending os_bucket_id to getObjectstoresField() from getObjectstorePath() (SiteInformation)
- Added os_bucket_id argument to getObjectstoresField() (SiteInformation)
- Sending os_bucket_id to getObjectstoresList() from getObjectstoresField() (SiteInformation)
- Added os_bucket_id argument to getObjectstoresList() (SiteInformation)
- Removed useless code that looked for objectstores in queuedata in getObjectstoresList() (SiteInformation)
- Sending os_bucket_id to getNewQueuedata() from getObjectstoresList() (SiteInformation)
- Added os_bucket_id argument to getNewQueuedata() (SiteInformation)
- getNewQueuedata() can now find queuedata matching a given os_bucket_id (SiteInformation)
- Now returning os_bucket_id from mover_put_data() and mover_put_data_new() (Mover)
- Receiving os_bucket_id from mover_put_data() call in transferActualLogFile() and transferAdditionalFile() (JobLog)
- Receiving os_bucket_id from mover_put_data() call in TransferFiles() (DeferredStageout)
- Receiving os_bucket_id from mover_put_data() call in stageOut() (RunJob, RunJobEvent)
- Receiving os_bucket_id from mover_put_data() call in moveLostOutputFiles() (pilot)
- Returning os_bucket_id from stageOut() and receiving it in transferToObjectStore() (RunJobEvent)
- Returning os_bucket_id from transferToObjectStore(), received in asynchronousOutputStager() (RunJobEvent)
- Sending os_bucket_id from asynchronousOutputStager() to updateEventRange() (RunJobEvent)
- Added os_bucket_id argument to updateEventRange() (EventRange)
- Sending objstoreID=os_bucket_id with updateEventRange call in updateEventRange() (EventRange)
- Added os_id argument to getObjectstoreName() (SiteInformation)
- Added os_bucket_id argument to getObjectstoreBucketEndpoint() (SiteInformation)
- Created findObjectStore(), getOSInfoFromBucketID(), getBucketID() (SiteInformation)
- Created logBucketID variable (Job)
- Reporting which OS bucket the log was written to, if any, in getJobMetrics() (PandaServerClient)
- Added experiment argument to transferActualLogFile() and sending it from transferLogFile() (JobLog)
- Created hasOSBucketIDs() to be used for merge jobs to identify prodDBlockTokens containing OS bucket IDs (SiteInformation)
- Created convertBucketIDsToOSIDs() (SiteInformation)
- Added prodDBlockToken argument to getFileInfo() (Mover)
- Sending prodDBlockToken to getFileInfo() from _mover_get_data_new() and mover_get_data() (Mover)
- Now getting file info from different OS:s (if necessary) in getFileInfo() (Mover)
- Changed addToOSTransferDictionary() to store os_bucket_id instead of os_name (FileHandling)
- Updated calls to addToOSTransferDictionary() to send os_bucket_id instead of os_name (JobLog, RunJobEvent)
- Updated getOSNames() to work populate dictionary using os_bucket_id instead of os_name (FileHandling)
- Renamed getOSNames() to getOSTransferDictionary(), and updated the calls to it (FileHandling, PandaServerClient)
- Created getQueuenameFromOSBucketID() (SiteInformation)
- Removed preset queuename from getNewQueuedata() (SiteInformation)
- Added queuename argument to getObjectstorePath(), getObjectstoresField(), getObjectstoresList(), getNewQueuedata() (SiteInformation)
- Sending queuename argument to getNewQueuedata() from getObjectstoresList() (SiteInformation)
- Sending queuename argument to getObjectstoresList() from getObjectstoresField() (SiteInformation)
- Sending queuename argument to getObjectstoresField() from getObjectstorePath() (SiteInformation)
- Sending queuename argument to getObjectstorePath() from getLogPath() (JobLog)
- Added queuename argument to getDDMStorage() (Mover)
- Sending queuename argument from getDDMStorage() to getObjectstorePath() (Mover)
- Sending queuename argument from mover_put_data() and mover_put_data_new() to getDDMStorage() (Mover)
- Added queuename argument to getQueuedataFileName(), getField(), readpar() (SiteInformation)
- Sending queuename to getQueuedataFileName() from getObjectstoresList(), getNewQueuedata(), getField(), readpar() (SiteInformation)
- Sending queuename to getField() from getObjectstoresList(), getNewQueuedata() (SiteInformation)
- Added queuename parameter to readpar() and sending it forward to si.readpar() (pUtil)
- Sending queuename to getObjectstoresField() from getObjectstorePath() (SiteInformation)
- Added queuename argument to getObjectstoreName() (SiteInformation)
- Sending queuename to getObjectstoresField() from getObjectstoreName() (SiteInformation)
- Added os_bucket_id to fileInfoDic in getFileInfo() (Mover)
- Extracting os_bucket_id in extractInputFileInfo() (Mover)
- Receiving os_bucket_id in _mover_get_data_new(), mover_get_data() from extractInputFileInfo() (Mover)
- Now using getObjectstorePath() in mover_get_data() to trigger a proper queuedata JSON download/copy (Mover)
General changes
- Minor cleanup; removed usage of alt, prodSourceLabel, testLevel, analysisJob, extradirs, proxyCheck in put_data(), removed proxyCheck in get_data\
() (S3ObjectstoreSiteMover)
- Removed unnecessary cleanupOS() and shouldCleanupOS() (RunJob)
- Removed useless getNewQueuedataXXX() (SiteInformation)
- Now removing current output files from stage-out queue in case of exception in transferToObjectStore() to prevent endless loop in asynchronousOutputStager() (RunJobEvent)
Memory monitoring updates:
- Calling __check_memory_usage() in monitoring loop, in monitor_job() (Monitor)
- Created merge_dictionaries() (pUtil)
- Now importing merge_dictionaries (PandaServerClient)
- Moved getUtilityInfo() from PandaServerClient to ATLASExperiment
- Created empty getUtilityInfo() (Experiment)
- Updated call to getUtilityInfo() in getNodeStructure() (PandaServerClient)
- Now importing getJSONDictionary in ATLASExperiment
- Created getUtilityOutputFilename() used by getUtilityCommand() (ATLASExperiment, Experiment)
- Created new error code ERR_PAYLOADEXCEEDMAXMEM, 1235, "Payload exceeded maximum allowed memory" (PilotErrors)
- Now using getJSONDictionary() in getJobReport() to avoid problems with unicode seen in test jobs (FileHandling)
- Created parseUtilityOutput() (ATLASExperiment)
- Creating a lockfile in __check_memory_usage() to prevent the MemoryMonitor to restart after it has been killed (Monitor)
- Avoiding restart of memory monitor in __main__() if lockfile MEMORYEXCEEDED is found (RunJob, RunJobEvent)
- Sending workdir to getUtilityCommand() from getUtilitySubprocess() (RunJob)
- Extracting argument workdir in getUtilityCommand() (ATLASExperiment)
- Changing to workdir before utility command is executed in getUtilityCommand() (ATLASExperiment)
- Improved reading of memory_monitor_summary.json in getUtilityInfo() (ATLASExperiment)
- Corrected time() -> time.time() in getUtilityInfo() (ATLASExperiment)
- Created getMemoryValues() and getMaxUtilityValue() (ATLASExperiment)
- Created getUtilityInfoPath() used by getUtilityInfo() and getMemoryValues() (ATLASExperiment)
- __check_memory_usage() now using getMemoryValues() (Monitor)
- Added allowTxtFile option to getUtilityInfo() (ATLASExperiment)
- Added pilot_initdir, allowTxtFile arguments to getUtilityInfo() (Experiment)
- Added allowTxtFile argument to getUtilityInfo() (ATLASExperiment)
- Removed getTracingReportFilename, writeJSON from import list (xrdcpSiteMover)
- Now catching general Exception instead of KeyError in __check_memory_usage() to avoid problem with empty JSON (Monitor)
LFC cleanup:
- Removed getLFCDir(), lfc_mkdir(), lcg_rf() (SiteMover)
- Ignoring empty lFC paths from getLFCPath() call in getProperPaths()
- Protecting against empty lfcpaths in getUserLFCDir() (SiteMover)
- Removed obsolete function getLFCChecksumType() (SiteMover)
- Removed obsolete function addMD5sum() (castorSvcClassSiteMover, dCacheLFCSiteMover, rfcpLFCSiteMover)
Time-out changes:
- Protected against exitcode None being sent in run() (TimerCommand)
Logging changes: (mainly invisible changes in this version)
- Change default log file name from pilotlog.out to pilotlog.txt (pUtil)
- Added new logging module to distribution: Logger.py with code constributions from Wen Guan (logging) and Jose Caballero (formatting)
- Created new global essentialPilotlogFilename (pUtil)
- Setting new global essentialPilotlogFilename in setPilotlogFilename() (pUtil)
- Added arguments label and essential to tolog() (pUtil)
Code cleanup: (apart from Mario's cleanup)
- Removed outdated getPoolFileCatalogDQ2() (Mover)
- Removed usages of rsespace variable in monitor_job() and monitor_recovery_job() (Monitor)
- Removed rsespace and rsespmsg from setSiteInfo() (Site)
- Removed usages of rsespace and rsespmsg, and sending remainingSpace and messageLevel to server from getNodeStructure() (PandaServerClient)
- Code cleanup in various pilot modules
Update from Danila Oleynik, Taylor Childers:
- New modules: ArgoJob, RunJobArgo, BalsamJob, MessageInterface
- Updated RunJobFactory; replaced Mira with Argo
- Added functions to serialize ARGO messages: serialize(), deserialize(), convert_unicode_string() (pUtil)
Update from Wen Guan:
- Added EventStager module for HPC Nersc
- Added GetJob module for getting multiple jobs
- Update EventServerManager to let it run in a sub-process. Before it was running in the current process, in this case it cannot be switched to another job which is needed by multi-jobs Yoda.
- In EventServerManager add functions to handle AthenaMP error messages.
- Update to use bulk event ranges updates to panda server
- Fix cases where all ranks of Yoda were killed when one rank finished.
- Update HPCManager to handle multi-jobs.
- Update Yoda database function to fix the bottleneck which was slow to update db changes
- Update Yoda Interaction module to fix the messages delay
- Update Yoda and Droid to handle multi-jobs
- Add Signal_block module in Yoda to make Yoda soft termination
- Update monitor module to monitor multi-jobs, terminate multi-jobs, handle multi-jobs logs and recover HPC jobs
- Restored is_secure variable which got lost in the merging process (S3ObjectstoreSiteMover)
Update from Alexey Anisenkon:
- Job stage-in base workflow implemented:
- . direct access support
- . no FAX for stagein
- . no DBReleases (as special input case) support
- . FileSpec object integration for stagein
- . xrdcp mover: get functionality implemented
- . Job object updates
- . other mover related updates
Update from Daniil Drizhuk:
- New job recovery algorithm implemented (DeferredStageout module)
No comments:
Post a Comment