66.1

Summary

- Previously pilot only avoided setting memory limit for CGROUPS sites, now avoiding it for all sites (Requested by A. Forti, R. Walker)
- Report only staged out events to PanDA (for monitoring reasons)
- Currently pilot reports processed events and sometimes events are processed but not staged out
- Fix for core count (ES) on single core queues 
- Disabling scheduling same jobs to the same rank when failure (seen on ARC)
- New schedconfig.astorages attribute with associated list of local storages per activity used by the pilot to look up default DDMEndpoint destination. If requested destination is not in astorages list, pilot will consider it to be a nucleus destination and will instead use first entry in astorages as default local destination 

Additional info from developers

A. Anisenkov: https://github.com/PanDAWMS/pilot/pull/66

'-M ' option will activate new site mover workflow, it actually sets queuedata.use_newmover=True

W. Guan: https://github.com/PanDAWMS/pilot/pull/65

1) report only number of staged out events to panda.
Currently pilot reports processed events. sometimes events are processed but not staged out. After discussing with monitor and others, it's good to only report staged out events.
2) fix s3 objectstore to disable http proxy.
s3 boto library doesn't support http proxy.
3) report fatal event status to panda when athenamp crashes.

A. Anisenkov: https://github.com/PanDAWMS/pilot/pull/64

one line typo fix

A. Anisenkov: https://github.com/PanDAWMS/pilot/pull/63

Sitemover logic upgrade:

• introduced new schedconf.astorages attribute represented associated list of local storages per activity to look up default DDMEndpoint destination (Pilot side destination choice). Pilot either takes destination suggested by Panda (if it's in the list of astorages) or makes own decision about what destination should be used. bugfixes:
• fixed typo in time function usage (incorrect import statement) -- affected traces reporting in few cases.

Panda Associated storages workflow overview:
* if requested by Panda destination (jobspec.DDMEndpointOut) is not defined in astorages list for given activity then pilot will consider it as a nucleus destination and will use first entry from astorages as default local destination.
* At the moment all transfers to nuclei are ignored. (to be covered later by alternative stageout workflow).

W. Guan: https://github.com/PanDAWMS/pilot/pull/62

1) Fix core count of ES on single core queues
2) disable to schedule same jobs to the same rank again and again when it fails, found at ARC
3) use gmtime in logger

Version info

General changes:

- Out-commented setting memory in getsetWNMem() and __getsetWNMem() (pilot)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/62

Commit Summary

avoid schedule same jobs to same ranks

Yoda to use gmtime

test new mover

fix job corecount of es on single core situation

fix merge conflicts

File Changes

M ATLASSiteInformation.py (1)

M HPC/EventServer/Logger.py (2)

M HPC/Logger.py (4)

M HPC/pandayoda/yodacore/Yoda.py (16)

M RunJobEvent.py (1)

  https://github.com/PanDAWMS/pilot/pull/65

Commit Summary

report only stagedout events to panda

fix s3objectstore sitemover with disabling http proxy

report es fatal error when athenamp crashed on some events

File Changes

M RunJobEvent.py (33)

M S3ObjectstoreSiteMover.py (80)

Updates from Alexey Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/63

Commit Summary

Mover bugfix: fixed typo in time() usage (wrong import statement)

sitemover updates: use new schedconf.astorages attribute (associated list of local storages) to look up default DDMEndpoint destination (Pilot side des\

tination choice).

Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev

fixed typo

small fixes, increase verbosity

File Changes

M Job.py (1)

M Mover.py (4)

M RunJob.py (2)

M SiteInformation.py (19)

M movers/mover.py (67)

  https://github.com/PanDAWMS/pilot/pull/64

Commit Summary

typo fix

Merge branch 'main-dev' of https://github.com/PanDAWMS/pilot into main-dev

File Changes

M movers/base.py (2)

  https://github.com/PanDAWMS/pilot/pull/66

Commit Summary

pilot: -M option implemented:

File Changes

M movers/base.py (2)

  https://github.com/PanDAWMS/pilot/pull/66

Commit Summary

pilot: -M option implemented:

File Changes

M pilot.py (16)