General changes:
- Looping job killer update to skip output from new version of memory monitor
- User request to update AtlasOffline nightlies releases (same setup as AtlasDerivation; currently release string is missing)
- Number of events, cpu time not recorded for preempted jobs (AES on HPC, now fixed)
- Now reporting cpu time with heartbeats
- New objectstore site mover (based on rucio site mover)
- Service update incl. logging and minor fixes to facilitate job debugging on especially NERSC
Additional site mover changes since this version was presented in the last ADC Weekly; highlights:
- Log transfers to OS now using the new site movers
- Updates to traces for the new site movers
- Job log processing has been integrated into newer site mover architecture both for normal stage-out to primary SE and special log file transfers
- Cleaned and updated logic for normal stage-out of logs to primary SE
- Implemented special log transfer to ObjectStore DDMEndpoints within new site mover architecture: -- new activity='pls' ("pilot log secondary/special") introduced to customize the workflow if needed -- objectstoreSiteMover is hardcoded as default site mover for special log transfers to OS
Workflow details:
In current implementation, normal stage-out of logs to primary SE is always executed. Special transfer of log files is applied as optional action and controlled by 'catchall' queue parameter (log_to_objectstore) or jobSpec options like 'putLogToOS'.
Contributions from A. Anisenkov, W. Guan, T. Childers and P. Nilsson.
Looping job killer update:
- Updated __loopingJobKiller() to ignore file name patterns 'memory_' and 'mem.'. Previously, the pilot only ignore 'memory_monitor' patterns (Monitor)
- Discussed in JIRA ticket https://its.cern.ch/jira/browse/ATLASPANDA-281
User request to allow AtlasOffline nightlies releases (same setup as AtlasDerivation)
- Update getCacheInfo() to handle AtlasOffline nightlies releases (ATLASExperiment)
Updates from Wen Guan:
Commit Summary
record number events to fix nEvents for preempted jobs
update schedconfig
merge with main-dev
File Changes
M ATLASSiteInformation.py (2)
M RunJobEvent.py (14)
Updates from Wen Guan (2):
Commit Summary
fix panda client to report cputime in heartbeat
fix wrong pilot error
fix nevnts and cputime for preempted jobs
File Changes
M Monitor.py (3)
M PandaServerClient.py (19)
M RunJobEvent.py (14)
M RunJobUtilities.py (2)
Updates from Wen Guan (3):
Commit Summary
define event service error code because of over subscribed events
define event service error code because of no available events
remove agis overrides in pilot
File Changes
M ATLASSiteInformation.py (72)
M PilotErrors.py (6)
Updates from Wen Guan (4):
Commit Summary
add new objectstore sitemover
fix pilot to handle errors in priority error report
report over subscribe events error in RunJobHPCEvent
define message server error and objectstore setup error
reorganize pilot errors
fix runjobevent to use the new site mover
remove space
File Changes
M PilotErrors.py (6)
M RunJobEvent.py (34)
M RunJobHpcEvent.py (10)
M UpdateHandler.py (8)
A movers/objectstore_sitemover.py (29)
M movers/sitemovers.py (1)
Updates from Taylor Childers:
- Added code to write a file named batchid..txt to disk after the batch job is submitted so one does not have to scour the logs to find the id.
- Standardizing the many Loggers we have under HPC. They now all use UTC time in the same format and include the process id to help isolate multithreading messages. Al
so added the process id to the main pilot Logger.
- Removed the sys.exit call which do not allow MPI to finalize properly. If MPI Finize is never called jobs can run on past execution end, wasting wallclock hours
- Added more logging messages and changed from using the depricated commands module to the subprocess module for executing slurm commands
- Changed Logging to standard logging module and brought the MPI import to the top so that MPI.COMM_WORLD.Abort could be called when errors occur. Abort ensures all
MPI ranks are killed. I added this because I was seeing failures where the worker nodes were not being released even though all processes had exited.
Files changed:
HPC/HPCJob.py
HPC/HPCManagerPlugins/slurm.py
HPC/Logger.py
HPC/pandayoda/yodacore/Logger.py
HPC/pandayoda/yodaexe/Droid.py
Logger.py
RunJobHpcEvent.py
pUtil.py
Updates from Alexey Anisenkov:
https://github.com/PanDAWMS/pilot/pull/59
sitemovers: review and update logic to send traces to Rucio
Major sitemovers updates:
Job log processing has been integrated into newer sitemover architecture both for normal stage-out to primary SE and special log file transfers. (get rid of old wrappers) [JobLog]
cleaned and updated logic for normal stage-out of logs to primary SE [JobLog.transferLogFile]
implemented special log transfer to ObjectStore DDMEndpoints within new sitemover architecture: -- new activity='pls' ("pilot log secondary/special") introduced to customize the w\
orkflow if need -- objectstoreSiteMover is hardcoded as default sitemover for special log transfers to OS -- added GetSURL stub function for objectstoreSiteMover
various small fixes
Workflow details:
In current implementation, normal stage-out of logs to primary SE is always executed.
Special transfer of log files is applied as optional action and controlled by 'catchall' queue parameter (log_to_objectstore) or jobSpec options like 'eventService' and 'putLogToO\
S' (actually old doSpecialLogFileTransfer() function is reused).
Job ignores (possible) failures of special log transfer to OS and will continue job processing in case of OS transfer issues.
Commit Summary
sitemovers: JobLog processing migrated to newer wrapper of sitemovers
JobLog: implemented special log transfer to OS DDMEndpoints within new sitemover architecture (new activity='pls' introduced)
sitemovers: hardcoded default sitemover (objectstoreSiteMover) for special log transfers to OS (activity='pls')
sitemover: cleanup and fix. GetSURL stub function for objectstoreSiteMover
File Changes
M Job.py (10)
M JobLog.py (156)
M Mover.py (15)
M SiteInformation.py (36)
M movers/mover.py (62)
M movers/objectstore_sitemover.py (15)
https://github.com/PanDAWMS/pilot/pull/60
quick fix: keep job.state untouched when log transfer successfully finished
File Changes
M JobLog.py (2)
https://github.com/PanDAWMS/pilot/pull/61
Commit Summary
sitemovers: review and updated logic to send traces
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
trace ClientState fixed
File Changes
M Mover.py (30)
M movers/mover.py (29)
No comments:
Post a Comment