65.0

Summary

1. Event Service related changes:

- Merge job optimizations - the previous pilot had problems handling large ES merge jobs. Mainly thanks to new AGIS OS queuedata format which has been highly optimized (A. Anisenkov, A. Di Girolamo). The new JSON file is primarily taken from CVMFS (A. de Salvo) and otherwise from the AGIS server. The schedconfig.objectstores field is currently only used for getting the default OS ddmpoint. All other OS lookups are done with the new JSON file.
- The CPU time for ES jobs is now taken from the jobReport (summed up). This is more accurate than measuring the CPU time as it is done in non-ES jobs (around the payload execution) since pilot does several other things aside from running AthenaMP at the same time.
- Support for having multiple files returned from AthenaMP in yampl messages. This is a new workflow for derivation ES jobs, which will have more than one output per event range; pending a new AtlasDerivation release (V. Tsulaia).
- Support for direct access with AthenaMP (to be tested with ES jobs); the tests are pending a new release.

2. Logs to Object Stores:

- Support for new 'putLogToOS' server instruction telling the pilot to stage-out the log file to an objectstore. [New since this pilot was presented last week:] This pilot version also sends the ddm endpoint for the OS to the panda server and uses the new file path (scope:filename) as agreed with the Rucio team. Tests are pending.

3. Memory monitoring:

- Dangling subprocess pointer fixed.
- Limited number of memory monitor restarts (default set to max 5 restarts). Requested by Rod Walker.

4. Job metrics:

- Now sending jobReport parameters dbTime and dbData with jobMetrics. Requested by Rod Walker.

5. Log file creation:

- Now explicitly removing the troublesome 'madevents' sub directory prior to log file creation.    Requested by Rod Walker.

6. Multi-jobs:

- Sequential multi-jobs are now from the same task only (can be switched off with pilot option). Requested by Emmanouil Vamvakopoulos.

7. Local disk size    check:

- Server update for ND true pilots after out of local disk space. Previously pilot quit before getting job definition (i.e. too early to report to server). Requested by Andrej Filipcic.

8. User jobs:

- Fix for user jobs when user requested copy-to-scratch on a direct access site. Previously pilot ignored it (by mistake).

9. Frontier:

- FRONTIER_ID now has the format TASKID_PANDAID.

10. Alternative stage-out:

- Alt. stage-out can now be controlled from the panda server (on/off/force).

11. MAKEFLAGS:

- Pilot is now setting new asetup option --makeflags=“$MAKEFLAGS". Requested by Asoka de Silva.

12. HPC updates from Wen Guan:

- Turned off memory monitoring, not implemented in Yoda (especially useless on edge node).
- Corrections for ATHENA_PROC_NUMBER.
- Yoda ARC stage-out and POE plug-in (for SuperMUC) added.
- Various other minor fixes and changes.

Version info

General changes
- Added taskId argument to addEnvVars2Cmd() (ATLASExperiment)
- Now sending job.taskID to addEnvVars2Cmd() from getJobExecutionCommand() and getJobExecutionCommandOld() (ATLASExperiment, NordugridATLASExperiment)
- Added taskId to FRONTIER_ID (taskId_jobId) in addEnvVars2Cmd(), getEnvVars2Cmd() requested by Rodney Walker (ATLASExperiment)
- Simplified call to mover_put_data() from stageOut() (RunJob, RunJobEvent)
- Simplified call to mover_put_data() from transferActualLogFile(), transferAdditionalFile() (JobLog)
- Simplified call to mover_put_data() from TransferFiles() (DeferredStageout)
- Simplified call to mover_put_data() from moveLostOutputFiles() (pilot)
- Created removeNoOutputFiles() (FileHandling)
- Now removing files from output file list if they are listed in allowNoOutput and do not exist, in __main__(). Requested by Tadashi Maeno (RunJob)
- Renamed output_files_json to extracted_output_files in __main__() (RunJob)
- Now explicitly removing 'madevent' in removeRedundantFiles(), requested by Rodney Walker (ATLASExperiment)
- Removed unnecessary functions, getProdCmd3() and getProdCmd3Old() and updated usages from getJobExecutionCommandOld() (ATLASExperiment)
- Simplified the implementation of getAnalysisRunCommand() in Experiment and moved the actual implementation to ATLASExperiment (Experiment, ATLASExper\
iment)
- Removed LFC_HOME variable and single usage in class declaration and to_native_lfn() (SiteMover)
- Added --makeflags=$MAKEFLAGS option to asetup in getJobExecutionCommand() and getProperASetup(), requested by Asoka De Silva (ATLASExperiment)
- Added exception for log file in __checkPayloadStdout() (Monitor)
- Created extractOutputFiles() used by __main__ in RunJob instead of extractOutputFilesJSON() (FileHandling, RunJob)
- Removed some debug messages that caused too much stdout leading to large log sizes, e.g. in getSURLDictionary() (SiteMover)
- Removed some debug messages that caused too much stdout leading to large log sizes, e.g. in getDatasetDict() (pUtil)
- Using os.getcwd() as default for wntmpdir instead of /tmp, in set_environment(). Requested by Andrej Filipcic (environment)
- Out-commented call to deprecated getNewQueuedata() in handleQueuedata() (pUtil)

Send DB info in jobMetrics, requested by Rodney Walker
- Created getDBInfo() (FileHandling)
- Added dbTime, dbData job object strings (Job)
- Added call to getDBInfo() from extractJobInformation() (ErrorDiagnosis)
- Now adding dbTime and dbData to jobMetrics in getJobMetrics() (PandaServerClient)
- Added dbTime and dbData to updateJobInfo() (RunJobUtilities)
- Added dbTime and dbData to handle() (UpdateHandler)

Limit number of MemoryMonitor restarts, requested by Rodney Walker
- Added utility_subprocess_launches variables + handling to __main__() (RunJob, RunJobEvent)

Memory monitoring
- Fix for dangling utility_subprocess (RunJob, RunJobEvent)

Same user multi-job request by Emmanouil Vamvakopoulos
- Added allowSameUser boolean and taskID (environment)
- Enabled pilot option -B (pilot)
- Sending prodUserID with job request in case allowSameUser is set, in getDispatcherDictionary() (pilot)
- Setting prodUserID in getNewJob() (pilot)

Bug fixes
- Removed tmp return from handleDBRelease() which caused skipping of DBRelease file to fail (Mover)
- Removed duplicated getFileInfoFromRucio() (SiteMover)
- Corrected _output to output in _getRemoteFileSizeLCGLS() (SiteMover)
- Removed -ALL suffix from agis_schedconf.json which seems to prevent the caching of the file to work properly, in getNewQueuedata() (SiteInformation)
- Sending job.prodDBlockToken to RunJobUtilities.updateRunCommandList() from __main__() (RunJob)
- Added prodDBlockToken argument to updateRunCommandList() (RunJobUtilities)
- Now removing --directIn payload option if 'local' is present in prodDBlockToken list, updateRunCommandList() (RunJobUtilities)
- Updated call to updateRunCommandList() in RunJob modules (RunJob, RunJobEvent, RunJobAnselm, RunJobEdison, RunJobHopper, RunJobTitan)
- Changed jobDic argument to job object in storeWorkDirSize() and removed unnecessary loop over job objects which caused problems with big Yoda jobs (F\
ileHandling)
- Sending self.__env['jobDic'][k][1] (job object) instead self.__env['jobDic'] (job dictionary) to storeWorkDirSize() from __checkWorkDir() (Monitor)
- Sending job instead jobDic to storeWorkDirSize() from createLogFile() (JobLog)
- Renamed config to configSiteMover to prevent problems seen on ND with late releases due to presence of a config module from TDAQ in the PYTHONPATH
- Updated config to configSiteMover import (S3ObjectstoreHttpSiteMover, S3ObjectstoreSiteMover, S3SiteMover, SiteMover, mvSiteMover, objectstoreSiteMov\
er, xrootdObjectstoreSiteMover, xrootdSiteMover)
- Removed unused config import (pUtil)
- Now recognizing copytool=fax in mover_get_data() when counting FAX statistics (Mover)

Server update for ND true pilots (pre-placed job def but with server updates) after out of local disk space, requested by Andrej Filipcic
- (i.e. updateServerFlag == True and jobRequestFlag == False)
- Avoiding checking local disk space in runMain(), before the getJob() (pilot)

Alt stage-out
- Added altStageOut data member plus handling (Job)
- Setting altStageOut in mover_put_data() (Mover)
- Changed argument flag to **pdict in allowAlternativeStageOut(), forceAlternativeStageOut() (SiteInformation)
- Using altStageOut in allowAlternativeStageOut() and forceAlternativeStageOut() (ATLASSiteInformation)
- Sending job.altStageOut to forceAlternativeStageOut() and allowAlternativeStageOut() from mover_put_data() and mover_put_data_new() (Mover)
- Turning off any forced alt stage-out for objectstore transfers in forceAlternativeStageOut() (ATLASSiteInformation)
- Created new functions for stage-in/out optimization: getObjectstoreBucketID(), getObjectstorePath(), getObjectstoreDDMEndpoint(),
  getAlternativeObjectstoreDDMEndpoint(), getObjectstoreEndpointID(), getObjectstoreKeyInfo() (SiteInformation)
- Updated getLogPath() to use getObjectstoreDDMEndpoint(), getObjectstorePath(), getObjectstoreBucketID() (JobLog)
- Updated getFileInfo() to use getObjectstoreDDMEndpoint(), getObjectstoreBucketID() and shortened logic (no special case for log files) (Mover)
- Updated getNewOSStoragePath() to use getObjectstoreDDMEndpoint(), getAlternativeObjectstoreDDMEndpoint(), getObjectstorePath()
  and getObjectstoreBucketID() (Mover)
- Removed unused variables from getDDMStorage() (os_bucket_id, queuename, jobId, ub) (Mover)
- Updated getDDMStorage() to use getObjectstoreDDMEndpoint(), getObjectstorePath(), getObjectstoreBucketID() (Mover)
- Removed unnecessary OS functions in transferToObjectStore() (RunJobEvent)
- Created getObjectstoreDDMEndpointFromBucketID() (SiteInformation)
- Now using getObjectstoreDDMEndpointFromBucketID() in transferLogFile() (JobLog)
- Now using getObjectstoreDDMEndpointFromBucketID() in transferToObjectStore() (RunJobEvent)
- Now using os_ddmendpoint instead of os_bucket_endpoint in addToOSTransferDictionary() (FileHandling)
- Removed unwanted methods getObjectstoreBucketEndpoint(), getObjectstoreName() (SiteInformation)
- Changed os_name to os_ddmendpoint (S3ObjectstoreHttpSiteMover)
- Updated setup() to use getObjectstoreDDMEndpointFromBucketID(), getObjectstoreEndpointID(), getObjectstoreKeyInfo() (S3ObjectstoreSiteMover)
- Added argument label to setup() and its calls (S3ObjectstoreSiteMover)
- Removed deprecated methods getAlternativeOS(), convertBucketIDsToOSIDs(), getOSInfoFromBucketID(), getBucketID(),
  findObjectStore(), getOSIDFromName(), getQueuenameFromOSID(), findAllObjectStores(),
  findOSEnabledQueuesInFullQueuedata(), getObjectstoresList() (SiteInformation)
- Added getObjectstoreFilename() (ATLASSiteInformation)

Event service
- Setting PILOT_EVENTRANGECHANNEL env variable instead of manipulating jobPars, in __main__() (RunJobEvent)
- Wen Guan: Fix in init function which prevented the S3 site mover from being used (S3ObjectstoreSiteMover)
- Sending os_bucket_id to getQueuedataFileName() from getObjectstoresList(), getNewQueuedata(), getField() (SiteInformation)
- Added os_bucket_id argument to getQueuedataFileName() (SiteInformation)
- Using os_bucket_id in queuedata filename in getQueuedataFileName() (SiteInformation)
- Will abandon getNewQueuedata() if file already exists (filename should now be unique to the bucket id) (SiteInformation)
- Added os_bucket_id argument to getNewQueuedata() (SiteInformation)
- Sending os_bucket_id to getField() from getObjectstoresList() (SiteInformation)
- Added os_bucket_id argument to getField() (SiteInformation)
- Extracting os_bucket_id in get_data(), put_data() (S3ObjectstoreSiteMover, S3SiteMover)
- Sending os_bucket_id to stageOut() from put_data() (S3ObjectstoreSiteMover, S3SiteMover)
- Sending os_bucket_id to stageIn() from get_data() (S3ObjectstoreSiteMover, S3SiteMover)
- Added os_bucket_id argument to setup(), stageIn(), stageOut() (S3ObjectstoreSiteMover)
- Sending os_bucket_id to setup() from stageIn(), stageOut() (S3ObjectstoreSiteMover)
- Sending os_bucket_id to getObjectstoresField() from setup() (S3ObjectstoreSiteMover)
- Sending os_bucket_id to sitemover_get_data() from mover_get_data(), _mover_get_data_new() (Mover)
- Added os_bucket_id argument to sitemover_get_data() (Mover)
- Sending os_bucket_id to get_data() from sitemover_get_data() (Mover)
- Sending os_bucket_id to sitemover_put_data() from mover_put_data(), mover_put_data_new() (Mover)
- Added os_bucket_id argument to sitemover_put_data() (Mover)
- Sending os_bucket_id to put_data() from sitemover_put_data() (Mover)
-- Changes below affects the changes above --
- Created getObjectstoreInfoFile(), getObjectstorePath(), getObjectstoreFilename(), getObjectstoreBucketID() (SiteInformation)
- Renamed old getObjectstorePath() to getObjectstorePathOld() - to be removed shortly (SiteInformation)
- Created getObjectstoresInfo() (SiteInformation)
- Created getObjectstoresField() (SiteInformation)
- Created getObjectstorePath() (SiteInformation)
- Now using getObjectstoresField() instead of getObjectstoreName() and getObjectstoreBucketEndpoint() in transferLogFile() (JobLog)
- Now using getObjectstoresField() and new getObjectstorePath() instead of old getObjectstorePath() in transferLogFile() (JobLog)
- Updated call to getObjectstoresField() in getFileInfo() (Mover)
- Updated two calls to getObjectstorePath(), and using getObjectstoresField() in getFileInfo() (Mover)
- Using new getObjectstoresField() instead of getObjectstoreName() in getNewOSStoragePath() (Mover)
- Using new getObjectstoresField() and getObjectstorePath() instead of old getObjectstorePath() in getDDMStorage() (Mover)
- Updated calls to getObjectstoresField() (S3ObjectstoreHttpSiteMover, S3ObjectstoreSiteMover, SiteInformation)
- Avoiding None values in prodDBlockToken list, in hasOSBucketIDs() (SiteInformation)
- Created getObjectstoreDDMEndpoint() (SiteInformation)
- Created getCPUTimes(), used in RunJobEvent::__main__() (FileHandling, RunJobEvent)
- Updated interpretMessage() to handle multiple files (RunJobEvent)
- Updated listener() to handle list of files from interpretMessage() (RunJobEvent)

Logs to OS:
- Added putLogToOS to Job class (Job)
- Changed argument from eventService to **argdict in doSpecialLogtransfer() (Experiment, ATLASExperiment)
- Now sending log to OS if job.putLogToOS is set, in doSpecialLogFileTransfer() (JobLog)
- Added logToOS boolean argument to PFCxml(), used to add an endpoint tag to the xml (pUtil)
- Sending logToOS to PFCxml() from createFileMetadata() (RunJobEvent, RunJob)

Direct access updates (to enable this functionality for ES)
- Using transferType fax in shouldPFC4TURLsBeCreated()
- Sending job.transferType to getFileAccessInfo() from stageIn() (RunJob)
- Sending transferType to getFileAccessInfo() from shouldPFC4TURLsBeCreated() (Mover)
- Sending transferType to createPFC4TURLs() from PFC4TURLs() (Mover)
- Added transferType argument to createPFC4TURLs() (Mover)
- Sending transferType to getTURLs() from createPFC4TURLs() (Mover)
- Added transferType argument to getTURLs() (Mover)
- Sending transferType to getPrefices() from getTURLs() (Mover)
- Added transferType argument to getPrefices() (Mover)
- Sending transferType to getFileAccessInfo() from getPrefices() (Mover)
- Sending job.transferType to pUtil.getFileAccessInfo() (RunJobHpcEvent)
- Added transferType argument to getFileAccessInfo() (pUtil)
- Using transferType and direct_access_wan in getFileAccessInfo() (pUtil)
- Added transferType argument to getCopyFileAccessInfo() (SiteInformation)
- Sending transferType to getCopyFileAccessInfo() from getDirectInAccessMode() (SiteInformation)
- Removed useFileStager from getCopyFileAccessInfo() and getDirectInAccessMode() (SiteInformation)
- Using transferType and direct_access_wan in getCopyFileAccessInfo() (SiteInformation)
- Moved getDirectAccess(), useDirectAccessLAN(), useDirectAccessWAN() from Mover to FileHandling
- Now using getDirectAccess() in getDirectInAccessMode() (SiteInformation)
- Removed directIn from getCopyFileAccessInfo() (SiteInformation)
- Now using self.__siteInfo.getCopyFileAccessInfo() instead of pUtil.getFileAccessInfo() (RunJobHpcEvent)
- Now using si.getCopyFileAccessInfo() instead of pUtil.getFileAccessInfo() (RunJob)
- Added experiment argument to getTURLs(), createPFC4TURLs(), PFC4TURLs() (Mover)
- Sending experiment to getTURLs() from createPFC4TURLs() (Mover)
- Sending experiment to createPFC4TURLs() from PFC4TURLs() (Mover)
- Sending experiment to PFC4TURLs() from _mover_get_data_new() and mover_get_data() (Mover)
- Added experiment argument to getPrefices() (Mover)
- Sending experiment to getPrefices() from getTURLs() (Mover)
- Now using si.getCopyFileAccessInfo() instead of pUtil.getFileAccessInfo() in getPrefices(), shouldPFC4TURLsBeCreated() (Mover)
- Added experiment argument to shouldPFC4TURLsBeCreated() (Mover)
- Sending experiment to shouldPFC4TURLsBeCreated() from PFC4TURLs() (Mover)
- Removed getFileAccessInfo() which has now become a duplicate of SiteInformation.getCopyFileAccessInfo() (pUtil)
- Renamed getCopyFileAccessInfo() to getFileAccessInfo() (SiteInformation, Mover, RunJob, RunJobHpcEvent)
- Removed sending stageIn boolean to getFileAccessInfo() from getDirectInAccessMode() (SiteInformation)
- Added transferType argument to getDirectInAccessMode() (SiteInformation)
- Sending transferType to getDirectInAccessMode() from getStageInMode() (FAXSiteMover, GFAL2SiteMover, LocalSiteMover, xrdcpSiteMover, xrootdObjectstor\
eSiteMover)
- Sending transferType to getStageInMode() from get_data() (FAXSiteMover, GFAL2SiteMover, LocalSiteMover, xrdcpSiteMover, xrootdObjectstoreSiteMover)
- Added transferType argument to getStageInMode() (FAXSiteMover, GFAL2SiteMover, LocalSiteMover, xrdcpSiteMover, xrootdObjectstoreSiteMover)
- Extracting transferType in get_data() (FAXSiteMover, GFAL2SiteMover, LocalSiteMover, xrdcpSiteMover, xrootdObjectstoreSiteMover)
- Sending transferType to get_data() from sitemover_get_data() (Mover)
- Sending transferType to sitemover_get_data() from _mover_get_data_new(), mover_get_data() (Mover)
- Added transferType argument to sitemover_get_data() (Mover)
- For transferType direct and fax, now calling si.updateDirectAccess() from __main__() (RunJob, RunJobEvent)
- Created updateDirectAccess() (SiteInformation)

Updates from Wen Guan:
- Now using curl to get keypair (ATLASSiteInformation, pUtil)
- Add hostname check with http proxy (S3ObjectstoreSiteMover)
- Fixed hostname is None issue (S3ObjectstoreSiteMover)
- Solve the region problem with AWS OS (S3ObjectstoreSiteMover, objectstoreSiteMover)

CMS Experiment
- Removed allowAlternativeStageOut() (CMSExperiment)

AMS Experiment
- Changed argument flag to **pdict in allowAlternativeStageOut(), forceAlternativeStageOut() (AMSTaiwanSiteInformation)