66.4

Summary

- Updated RunJobEvent to support new movers

- Updated RunJob to support new movers with es merge jobs.

- Updated movers to support stageout es to objectstore.

- Updated movers to support stagein es from objectstore.

- Report external stageout time (time after athenamp terminated) in jobmetrics.

- Fix to report pilot substatus 'pilot_failed' and fail state to panda.

- Set 'pilot_failed' substatus in RunJobEvent when Pilot fails a job.

- Cosmetic change in logic to apply site mover check_availablespace before real transfers (minor fix allowed to skip space check for a file being used for direct reading)

- Cleanup of misleading error messages in (old) local site mover

Code contributions from Wen Guan, Alexey Anisenkov and Paul Nilsson.

(*Alessandro Di Girolamo requested the release today ahead of schedule to speed up ES validation).

Version info

General updates:

- Removed gfal references from LocalSiteMover

Updates from Alexey Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/74

Commit Summary

mover: check space size workflow cosmetic

Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev

File Changes

M movers/mover.py (6)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/75

Update RunJobEvent.py to support new movers

Update RunJob.py to support new movers.

update movers to support stageout es to objectstore.

update movers to support stagein es from objectstore.

report external stageout time(time after athenamp terminated) in jobmetrics.

fix to report pilot substatus and fail state.

You can view, comment on, or merge this pull request online at:

Commit Summary

es to support new movers

es to support new movers

recover old asynchronous stageout

update runjobEvent to support new mover

set pilot final failed state

report external stagout time

fix event ranges updates

implemented stageout os

in new movers adding stagin os and stageout os

fix refer of FileSpec

fix os keys

fix issues when staging out es

fix external stageout time

update movers to support objectstore stagein

add pfn to refer the full file path

rebase with upstream main-dev

File Changes

M EventRanges.py (43)

M Job.py (13)

M Mover.py (54)

M PandaServerClient.py (2)

M RunJobEvent.py (408)

M RunJobUtilities.py (3)

M UpdateHandler.py (8)

M movers/base.py (2)

M movers/mover.py (112)

M movers/objectstore_sitemover.py (20)

M movers/rucio_sitemover.py (19)

  https://github.com/PanDAWMS/pilot/pull/76

Commit Summary

set substatus when failJob

File Changes

M RunJobEvent.py (1)