Checking job status after job download
- After job has been downloaded, check with the PanDA server that the job is not already in a running state. This can happen due to a bug on the batch system side on Nordugrid resources
- Requested by Andrej Filipcic, David Cameron
- Pre-version running since Christmas on ND resources
Removed hardcoded PanDA URLs
- Requested by Peter Love
Memory Monitoring updates
- Now using release 21.0.12 (next pilot version will use version from payload release area)
- Extracting and reporting new output; totRCHAR, totWCHAR, totRBYTES, totWBYTES, rateRCHAR, rateWCHAR, rateRBYTES, rateWBYTES
- Requested by Johannes Elmsheuser
Avoiding killing BOINC client process at the end of the pilot
- Requested by David Cameron
Time-out around pstack command and fix for bad usage of os.killpg() (removed negation of pid)
- Requested by Rod Walker
Mover updates
- Now sending file size with trace report
- Requested by David Cameron
- Support for direct access in all new movers
- Upgrade to stage-in workflow
- Execute resolve input replicas by demand only for required movers (mv, storm and rucio are excluded)
Contributions from M. Lassnig, A. Anisenkov, W. Guan, D. Cameron, P. Nilsson.
General changes:
- Added call to pUtil.getJobStatus() in getNewJob(). After job has been downloaded, check with the PanDA server that the job is not already in a running state. This can happen due\
to a bug on the batch system side on Nordugrud resources. Requested by Andrej Filipcic, David Cameron
- Removed negation of id number sent to os.killpg() in killProcesses(). Requested by Rod Walker (processes)
- Added time-out to pstack command in dumpStackTrace() (processes)
Benchmarks:
- Added pdict argument to executeBenchmark() (SiteInformation, ATLASSiteInformation)
- Added cloud argument to getBenchmarkDictionary(), sent from runMain() (node, pilot)
- Created executeBenchmarks(), new getBenchmarkDictionary(). Added benchmarks private data member (Node)
- Created getBenchmarkDictionary(), updated executeBenchmarks(), added self.__benchmarks (SiteInformation, ATLASSiteInformation)
Hardcoded pandaserver urls:
- Removed hardcoded pandaserver url from various places (note: cannot currently remove hardcoded url for S3 secret key downloads since keys are only known to pandaserver and not a\
ny dev server)
- Added new argument url to downloadEventRanges(), updateEventRanges() (EventRanges)
- Setting PanDA server url in RunJob* argument list in getSubprocessArguments() (Experiment)
- Added -W server url argument in argumentParser() (RunJob)
- Added __pandaserver variable (RunJob)
- Using __pandaserver with downloadEventRanges() in executePayload(), __main__() (RunJob)
- Using __pandaserver with downloadEventRanges() in __main__() (RunJobEvent)
- Using __pandaserver with downloadEventRanges() in getJobEventRanges() (RunJobHpcEvent)
- Using __pandaserver with updateEventRanges() in stageOutZipFiles_new(), stageOutZipFiles() (RunJobEvent)
- Using __pandaserver with updateEventRanges() in updateEventRanges(), (RunJobHpcEvent)
- Removed import of httpConnect in RunJobEvent, RunJobHpcEvent
Memory Monitoring:
- Now using release 21.0.12 to setup the memory monitoring (ATLASExperiment)
- Added calculation and handling of variables related to new outputs RCHAR, WCHAR, RBYTES, WBYTES in getMemoryValues() (ATLASExperiment)
Mover fixes:
- Now sending filesize with trace report, from stageout(), stagein_real() (mover)
Updates from Wen Guan:
https://github.com/PanDAWMS/pilot/pull/104
- Removed deprecated modules EventStager.py and MVEventStager.py
- Updated RunJobHpcEvent.py to remove EventStager usage
Commit Summary
deprecate eventstager
remove eventStager
File Changes
D EventStager.py (559)
D MVEventStager.py (298)
M RunJobHpcEvent.py (95)
https://github.com/PanDAWMS/pilot/pull/105
Commit Summary
Fixed trace report which overwrote localsite
File Changes
M Mover.py (3)
Updates from Mario Lassnig:
https://github.com/PanDAWMS/pilot/pull/106
Commit Summary
ruciomover: tiny updates
storm: fix for timestamp corner case
Merge pull request #91 from mlassnig/storm-update
Merge pull request #90 from mlassnig/rucio-mover-updates
Merge commit 'e68a141debeafff49cf89d28047d0faf599e5c87' into main-dev
File Changes
M movers/rucio_sitemover.py (26)
https://github.com/PanDAWMS/pilot/pull/107
Commit Summary
Explicit import of OS
File Changes
M movers/rucio_sitemover.py (4)
Updates from Alexey Anisenkov:
https://github.com/PanDAWMS/pilot/pull/108
Commit Summary
movers: update direct access workflow (force to check root protocol in case of PQ supports direct access and jobspec allows it as well)
cosmetic fix
movers: rewrite resolve_replica to iterate over accepted schemes first
movers bugfix: prevent stage-in error (Argument list too long) while printing details about input files(1k+).
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
sitemovers: stage-in workflow upgrade: resolve input replicas by demand only for required movers (mv, storm, rucio are excluded)
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
bugfix: ATLASExperiment.getMemoryValues() fix local var declaration ('rchar' referenced before assignment issue)
* movers bugfix: prevent stage-in error (Argument list too long) while printing details about input files(1k+).
sitemovers fix
directaccess fixes
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
ATLASExperiment.py: revert back line endings style to win
ATLASExperiment.py: fix line endings style
sitemovers: exclude DISABLED ddms from inputddms
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev
ATLASExperiment.py fix endline style
Update ATLASExperiment.py
File Changes
M ATLASExperiment.py (7089)
M Job.py (30)
M movers/base.py (63)
M movers/mover.py (128)
M movers/mv_sitemover.py (3)
M movers/rucio_sitemover.py (2)
M movers/storm_sitemover.py (2)
https://github.com/PanDAWMS/pilot/pull/109
Commit Summary
sitemovers: protect sitemover.resolve_replica()
File Changes
M movers/mover.py (2)
Updates from David Cameron
https://github.com/PanDAWMS/pilot/pull/110
Commit Summary
do not kill boinc_client process on exit
File Changes
M processes.py (2)
https://github.com/PanDAWMS/pilot/pull/111
Commit Summary
rucioSiteMover updates: use default resolve_replica() implementation; fix upload cmd to consider --guid value for .root files
File Changes
M movers/rucio_sitemover.py (14)
No comments:
Post a Comment