72.0

Support for direct access over LAN/WAN in production jobs

- For the moment 'root' is the only allowed schema in the replica lookup, but can easily be extended with davs, gsiftp, etc.

- Replica selection for direct access over WAN is achieved through geoip sorting in rucio.list_replicas().

- Removed requirement of .root to be present in files to be accessed directly

Event service updates

- Clean-up of outdated AGIS activity name: es_read

- Support for ES fail-over; Two storages for ES can be added in AGIS; if the first one fails, the second one can be used

- Updated the default ZIP time gap; If pledgedcpu is -1 (opportunistic resource), default ZIP time gap is 10 minutes (otherwise 2 h)

- Based on es_failover, ES can support storages blacklisting; Either es_events or es_failover can be blacklisted 

- Pilot maxtime and softkill implemented for ES

- Fail pilot if several failed stage-out attempts

- Fix for relative in-file positional event ranges for prefetcher; Now relying on inFilePosEvtNum flag. Subsequent fixes for inFilePosEvtNum for ES and prefetcher

- Using read_wan to replace r activity for ddm (for object stores keys)

- No assumption of in-file positional event ranges for prefetcher

- Can now pass multiple input files to prefetcher command line

- Changed inFilePosEvtNum to job parameter (not an event range parameter)

- Reporting 'AllSuccess' when all events finish, to distinguish with lost heartbeat cases

- Bug fix for failed esmerge jobs

- Added option to define maxwait for tail events

Associated storages

- Now using associated storages when looking up replicas (from AGIS astorages)

- As discussed in thread "Multiple read DATADISK per PQ"

Overwriting AGIS data

- Pilot can now parse and overwrite AGIS data passed via jobPars settings

- Added copytools and acopytools entries to overwriteAGISdata, update affected stagein/out functions

- The format of overwriteAGISdata command line option in JobParameters is a string representation of python structures, e.g.

“””

jobPars = '

--overwriteAGISdata "{\'allowfax\':False,\'copytools\':{\'rucio\':{\'setup\':\'fakesetup.sh\'}, \'acopytools\':{\'pr\':[\'rucio\']}}"

'

to simply overwrite queuedata both for stage-in and stage-out (e.g. by HC jobs) and force use of Rucio sitemover, the following settings can be passed:

--overwriteAGISdata "{'copytools':{'rucio':{'setup':''}, 'acopytools':{}}"

“””

File size checks:

- Exclude available space checks during stage-in for mv and storm sitemovers

- Stage-in logic updates: first process files which can be potentially used for direct access to exclude them from available space checks

Hot fix

- Fix for bug which updated final state two times

Contributions from W. Guan, A. Anisenkov, N. Magini, P. Nilsson