ATHENA_PROC_NUMBER
- If env variable not set, set it to job.coreCount – If set, do not overwrite. Note: not set if coreCount = 1.
- Requested by R. Walker
Reset cpuConsumptionTime to -1 if > 10**9 to avoid unrealistic consumption times
- Caused by frozen VM (affected BOINC jobs)
- DB cleaned up by Gancho (a handful of jobs the last year)
- Requested by Wenjing Wu
Now first looking for a local copy of the agis_ddmendpoints json file before copying from cvmfs or downloading from AGIS
- To alleviate a situation a Chinese HPC where cvmfs is not updated frequently and with no outbound connection
- JSON preplaced by aCT
- Requested by Wenjing Wu
WebDAV changes
- WebDAV protocol added to GFAL2 and rucio movers. “Rucio now supports the DAV protocol. It uses the protocol to slightly modify it's behavior when talking to a strict DAV endpoint compared to a more relaxed HTTP endpoint. Integrating the DynaFed into ATLAS DDM requires Rucio to make this distinction” – M. Lassnig
- https protocol replaced with davs in storm and rucio movers (should fix a problem seen yesterday at UKI-LT2-QMUL reported first by Mark Sosebee), requested by Petr Vokac
Core information added to benchmark dictionary
- threads_per_core, cores_per_socket, sockets and if running in a VM
- Requested by Ilija Vukotic
Event Service updates
- Direct access for event service
- AGIS: direct_access_lan=True, direct_access_wan=True
- Task: allowInputLAN=only (allowInputWAN not used? To be discussed)
- Generates job with transferType=direct
- Testing successful at FZK, BNL and UTA test sites but needs some further development+testing
- Remote replica currently random, geo-sorting will be done on Rucio server side ~end of July, additional pilot changes needed
- ROOT access of WAN will be discussed when people are back from vacations. For now, WAN access is assumed to be working if enabled as above
- Zipping of output files is now default
- Support for multiple buckets
- tar/zip disabled when PanDA Proxy is used
- Fix for objectstoremover to handle rucio standard paths
Event Service for BOINC
- BOINC clients do not have GRID certificate and are instead using PanDA Proxy
- A PanDA proxy access key in job specification allows BOINC pilot to:
- get/update event ranges through PanDA Proxy on the PanDA server
- get pre-signed links from PanDA Proxy for uploading files directly to object store
- New module EventRangesPandaProxy to get/update event ranges through PanDA proxy
- New mover for uploading output files to OS with PanDA proxy
Contributions from W. Guan, A. Bogdanchikov, F. Berghaus, M. Lassnig, P. Vokac, P. Nilsson
Version info
70.0
General changes
- Removed function getNumberOfCores() as it is not used (Node)
- Modernized file open in collectWNInfo() (Node)
- Reset cpuConsumptionTime to -1 if > 10**9 in getNodeStructure(). Requested by Wenjing Wu (PandaServerClient)
- Now looking for local agis_ddmendpoints.json before trying download from AGIS/CVMFS, in loadDDMConfData(). Requested by Wenjing Wu (SiteInformation)
- Added ckpt* to removeRedundantFiles() (ATLASExperiment)
- Using with open in loadURLData(), grammatical change, removed warning text (SiteInformation)
- Added WebDAV protocol ('davs') to GFAL2 mover, requested Frank Berghaus (gfalcopy_sitemover)
Core count and core info
- Simplified setNumberOfCores() as discussed in JIRA ticket https://its.cern.ch/jira/browse/ADCDPA-49
- Renamed setNumberOfCores() to getNumberOfCores() (Node)
- Created getNumberOfCoresFromEnvironment() (Node)
- Now setting job.coreCount in getNewJob() (pilot)
- Created collectCoreInfo() (Node)
- Created isAVirtualMachine() (Node)
- Added core info (threads_per_core, cores_per_socket, sockets and if running in a VM) to benchmark dictionary. Requested by Ilija Vukotic (JobLog)
ATHENA_PROC_NUMBER
- Created addAthenaProcNumber() (ATLASExperiment)
- Now adding ATHENA_PROC_NUMBER explicitly to payload execution command, as long as it has a value > 1 (ATLASExperiment)
- The pilot is not overwriting ATHENA_PROC_NUMBER but is using a tmp variable instead (ATHENA_PROC_NUMBER_JOB)
Direct access
- Removed name conflicts (there is both is_directaccess and is_directaccess() in stagein_real()) (mover)
- Rewrote is_directaccess(), renamed it to get_directaccess(), returning both allow_directaccess and type (mover)
- Sending directaccesstype to resolve_replicas(), added argument (mover)
- Created get_pfns() and get_turls() (mover)
- Created getReplicaDictionaryFromXML() (FileHandling)
- Created addFullPathsAsInput() (RunJobUtilities)
- Updated getDirectAccess() to be stricter (FileHandling)
Updates from Alexander Bogdanchikov
https://github.com/PanDAWMS/pilot/pull/137
Commit Summary
new ES panda proxy features for BOINC
File Changes
M EventRanges.py (24)
A EventRangesPandaProxy.py (145)
M Job.py (8)
M PILOTVERSION (2)
M RunJobEvent.py (46)
A S3ObjectstorePresignedURLSiteMover.py (510)
M movers/mv_sitemover.py (31)
A movers/pandaproxy_sitemover.py (252)
M movers/sitemovers.py (1)
https://github.com/PanDAWMS/pilot/pull/143
Commit Summary
more details in the log when download event ranges failure: print exception
File Changes
M EventRanges.py (4)
Updates from Wen Guan:
https://github.com/PanDAWMS/pilot/pull/138
Commit Summary
update to set zip as default
File Changes
M RunJobEvent.py (4)
https://github.com/PanDAWMS/pilot/pull/144
1. using storageId but fallback to bucket id if storageid is not available
2. only when storageId is set, objectstoresitemover will build replica.
3. implementation to support multiple buckets.
4. fix getSURL in objectstoresitemover to use pathConvention.
5. replace readpar to use new function. because readpar is reading old schedconf which can be different from the schedconfi the new mover is using.
Commit Summary
use storageid but fallback to bucket id if storageid is not available
fix to only buil replica url when storageId is set
multiple buckets implementation
fix pathConvention in stagein
change default num of buckets to 1
fix getSURL with pathConvention
replace readpar to use new function based on new mover agis
File Changes
M Job.py (18)
M RunJobEvent.py (87)
M SiteInformation.py (18)
M movers/base.py (2)
M movers/mover.py (20)
M movers/objectstore_sitemover.py (42)
https://github.com/PanDAWMS/pilot/pull/145
Commit Summary
disable tar/zip when pandaproxy is used
fix mover at checking errors part by checking file status, not checking failed_transfers
fix objectstoremover to handle rucio standard paths
File Changes
M Mover.py (2)
M RunJobEvent.py (12)
M UpdateHandler.py (2)
M movers/objectstore_sitemover.py (15)
https://github.com/PanDAWMS/pilot/pull/146
Commit Summary
pilot failed should be false when no events
File Changes
M RunJobEvent.py (2)
https://github.com/PanDAWMS/pilot/pull/147
Commit Summary
update
File Changes
M movers/objectstore_sitemover.py (6)
Updates from Mario Lassnig/Frank Berghaus
Rucio now supports the DAV protocol. It uses the protocol to slightly modify it's behavior when talking to a strict DAV endpoint compared to a more relaxed HTTP
endpoint. Integrating the DynaFed into ATLAS DDM requires Rucio to make this distinction. Davs protocol added to the pilot.
https://github.com/PanDAWMS/pilot/pull/139
Commit Summary
add DAVs to protocols supported by rucio
File Changes
M movers/rucio_sitemover.py (2)
Updates from Petr Vokac:
Rucio no longer provides replica location for HTTPS protocol, because the name
was changed to DAVS. We had at least one site where our jobs failed due to
no access to input files using StoRM site mover (GGUS:129509) and there are
more sites that also fail while use StoRM site mover but luckily they fallback
to to lcgcp site mover.
https://github.com/PanDAWMS/pilot/pull/148
Commit Summary
Updated protocol name "https" -> "davs" for listing rucio replicas
File Changes
M movers/mover.py (2)
M movers/storm_sitemover.py (2)
No comments:
Post a Comment