67.0

Summary

Verifying remaining proxy life time (during payload running)

- Currently every 5 minutes, req. minimum 10 minutes remaining [Requested by R. Walker]

- NB. Switched off in case proxy verification is not used

Bug fix for nightlies in ‘new’ setup function

- Release was missing in asetup option

- NB. Nightlies on the grid have not been used for some time so this was not discovered until an independent test was done

Event service:

- Setting ‘pilot_killed’ substatus error instead of ‘pilot_failed’ for kill signals

- Fixed event status when an AthenaMP error is received (previsouly a ‘normal’ error would be set to ‘fatal’)

- Downloading events before starting AthenaMP and fail immediately if no events are available to avoid wasting time starting up AthenaMP

Updates to old movers

- OS mover correction

- Now closing S3 connection (secret key query) to avoid too many open connections

- GFAL2 mover fix (Explicitly adding double slashes to ‘file:/’ to avoid issues on one (CC7) site)

Updates to new movers

- Several updates to new mv mover (Incl. creation of output.list, mv instead of copy, code cleanup)

- Storm mover using webdav etags (requires davix to be locally setup – taken care of by P. Love)

- Extended xrdcp stage-out error handling (Now considering ‘Unable to checksum’ error (as well as ‘query cksum is not supported’) - For sites that do not support CRC calculation on the fly)

- Added gsiftp to the list of supported protocol schemes in lcgcp mover

BONUS FEATURE:

In addition to the changes above (that were presented in the last ADC Weekly), we have since the meeting written a new mover for gfal-copy which is integrated in the new mover architecture – and also has the ‘file://’ fix as the old gfal2 mover.

Code contributions from Wen Guan, Alexey Anisenkov, Mario Lassnig, David Cameron and Paul Nilsson.

Version info

General updates:

- Created __check_remaining_proxy() which is called every five minutes from monitor_job() and monitor_recovery_job() te ensure there is at least ten minutes left of the proxy (Mon\

itor)

Bug fix for nightlies:

- Setting cacheVer to release for cases "," in job.homePackage or "rel_" in job.homePackage in getJobExecutionCommand() (ATLASExperiment)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/82

Commit Summary

To support s3+rucio

Fixed bug

File Changes

M RunJobHpcEvent.py (2)

M objectstoreSiteMover.py (2)

  https://github.com/PanDAWMS/pilot/pull/84

1. To use 'pilot_killed' instead of 'pilot_failed' when a signal is caught

2. Fix event status when an AthenaMP error message is received. Normal athenaMP errors should not be 'fatal'. And currently we will not report fatal until we validate AthenaMP err\

or reports

Commit Summary

update substatus when pilot is killed

fix event status

File Changes

M RunJobEvent.py (6)

M pUtil.py (2)

  https://github.com/PanDAWMS/pilot/pull/85

1. Close s3 connection in objectstore sitemover, to avoid too many open connections

2. Download events before starting athenaMP (fail immediately if no events available so not to waste time with starting up AthenaMP)

Commit Summary

close os connections

download events before startAthenaMP

File Changes

M RunJobEvent.py (28)

M S3ObjectstoreSiteMover.py (9)

  https://github.com/PanDAWMS/pilot/pull/86

1) In gfal sitemover, now using 'file:///path' instead of 'file:path'.

2) Set corecount before setup, because in preemption jobs, jobs are not preempted before reporting corecount.

3) Send heartbeat to panda when job state is changed.

Commit Summary

set corecount before setup

send heartbeat to panda when job state changes

in gfal, to use file:///

File Changes

M GFAL2SiteMover.py (6)

M Job.py (1)

M Monitor.py (9)

M RunJobEvent.py (14)

M UpdateHandler.py (1)

Updates from David Cameron:

  https://github.com/PanDAWMS/pilot/pull/83

Commit Summary

Tidier handling of init_dir, add method to write output.list

Move instead of copy for stage out

Rename to be consistent with other movers

Added mv mover to mover list

File Changes

R movers/mv_sitemover.py (38)

M movers/sitemovers.py (1)

Updates from Mario Lassnig:

  https://github.com/PanDAWMS/pilot/pull/87

Commit Summary

storm: file protocol sitemover using webdav etags

File Changes

M movers/sitemovers.py (1)

A movers/storm_sitemover.py (125)

Updates from Alexey Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/88

xrdcp stage-out fix for sites do not supported CRC calc on fly (consider "Unable to checksum" error)

lcgcp sitemover: added gsiftp into the list of supported protocol schemes

Commit Summary

lcgcp sitemover: added gsiftp into the list of supported protocol schemes

Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev

xrdcp mover: consider "Unable to checksum" error to activate workaround workflow for sites does not supported remote crc calculation on fly

File Changes

M movers/base.py (2)

M movers/lcgcp_sitemover.py (2)

  https://github.com/PanDAWMS/pilot/pull/89

gfalcopySiteMover sitemover implemented

You can view, comment on, or merge this pull request online at:

Commit Summary

gfalcopySiteMover sitemover implemented

Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev

File Changes

M movers/base.py (19)

A movers/gfalcopy_sitemover.py (210)

M movers/sitemovers.py (1)