PICARD 60a

Summary

* Pilot is now receiving both container and dataset name from the server,

forwarded to rucio tracing server Previously pilot only received dataset name. Requested by Vincent Garonne

* Updates to the complex copysetup and copyprefix fields. Previously lists defined in copyprefix could not be used for resolving SURL to TURL conversions in direct access mode at CERN, now supported Major revision of several AGIS/schedconfig fields is being discussed

(to be implemented)

* Support for Event Index in Event Service code. Replacing previous method of dynamically creating TAG files. Currently using hardcoded EI server located in ES

* Always setting MAKEFLAGS and ROOTCORE_NCPUS for user jobs. Previously only set when coreCount was set. Requested by Andrej Filipcic

* Usage of ATLAS_CGROUPS_BASE now supports older cgroups installations. Previously /cgroup/memory was presumed to be located in /var. Requested by Alexey Sedov

* Changes for fax and overflow jobs. Now skipping fax fallback for overflow jobs. Overflow job definition from server now contains source site which is used to resolve the correct fax redirector using a GAE translator application written by Ilija Vukotic. Using xrdcp option ­np, no progress info. Requested by Ilija Vukotic

* Added switching between three access modes for davix/webdav input access. Code from Johannes Elmsheuser

* pwg*, pwhg*, *PROC* explicitly deleted before log is created. Requested by Zachary Marshall

* Time-out function fix (Wen Guan). Fixed a problem with the time-out function that sometimes did not time out commands properly (time-out not honored)

* Code merges

­ Latest codes from Danila Oleynik (HPC) and Wen Guan (ES on HPC)

­ SAGA updated to recent version (DO)

­ Added Anselm HPC support (DO)

­ Parameters for some tuning added (DO)

­ Added HPC Event Service component which includes (WG)

­ HPCManager.py to manage HPC job submitting and polling

­ HPCJob.py to run in HPC cluster to start Yoda(MPI rank 0) and Droid(MPI rank 1..n)

­ Yoda to manage MPI job status and Event status from Tadashi

­ Droid to start EventSeriveManager to run ES jobs

­ EventServiceManager.py to manage Yampl messaging thread,

TokenExtractProcess and AthenaMP process

­ Added RunJobHpcEvent.py, which gets the pilot job, setup HPC

requirement and start HPCManager (WG)

Version info

Removed the old outdated sendTrace() and renamed sendTraceToRucio() to sendTrace() (SiteMover)

Added new data member prodDBlocks to Job class, which will contain the correct container or dataset name for traces  (Job)

Created getProperDatasetNames() (Mover)

Sending rucio_dataset_dictionary to mover_get_data() (Mover)

Now sending the proper container/dataset name to the tracing server (Mover)

Added fileList argument to getPrefices and from its call in (Mover)

Created matchCopyprefixReplica(), used by getPrefices() (Mover)

Added prefix_dictionary argument to convertSURLtoTURL() and from its call in getTURLs() (Mover)

Now overwriting old/newPrefix if prefix_dictionary has an entry for key value 'surl' in convertSURLtoTURL() (Mover)

Created getSurlTokenDictionary() used from mover_get_data() (Mover)

Sending tokens_dictionary to PFC4TURLs() from mover_get_data() (Mover)

Added tokens_dictionary argument to PFC4TURLs() (Mover)

Sending tokens_dictionary to createPFC4TURLs() from PFC4TURLs() (Mover)

Added tokens_dictionary argument to createPFC4TURLs() (Mover)

Sending tokens_dictionary to getTURLs() from createPFC4TURLs() (Mover)

Added tokens_dictionary argument to getTURLs() (Mover)

Sending the space token corresponding to the LFN to convertSURLtoTURL() from getTURLs(), requested by Johannes Elmsheuser (Mover)

Added token argument to convertSURLtoTURL() (Mover)

Sending token to convertSURLtoTURLUsingHTTP() from convertSURLtoTURL() (Mover)

Added token argument to convertSURLtoTURLUsingHTTP() requested by Johannes Elmsheuser (Mover)

Support for event index, removed usage and handling of tag files (RunJobEvent)

Always setting MAKEFLAGS for analysis jobs in getJobExecutionCommand() (ATLASExperiment)

Always setting ROOTCORE_NCPUS in addEnvVars2Cmd() (ATLASExperiment)

Removed unnecessary import of Popen from RunJobEvent

Created getPlainCopyPrefices() used by shouldPFC4TURLsBeCreated() (Mover)

Added exception for dummy in shouldPFC4TURLsBeCreated() (Mover)

Code merge: Code from Wen Guan:

Added Hpc identifier in getSubprocessName() (Experiment)

Added new data members to Job class: eventServiceMerge, mode, hpcStatus, refreshNow (Job)

Added new setter methods for the data members above; setMode(), getMode(), setHpcStatus(), getHpcStatus() (Job)

Setting new variables mentioned above (Job)

Added HPC directory to distribution

Added HPC directory to file list in getFileList() (pUtil)

Corrected for HPC environment in curl command in _Curl::get(),post(),put() (pUtil)

Outcommented the pwd part in createESFileDictionary() (pUtil)

Replacing .pool.root. with .txt. in writeToInputFile() (pUtil)

Updated the inputHitsFile handling in writeToInputFile(), updateJobPars() (pUtil)

Added exception for catchall = HPC_HPC in shouldPFC4TURLsBeCreated() (Mover)

Getting the coreCount primarily from job.coreCount in getJobMetrics() (PandaServerClient)

Forwarding HPC parameters mode and HPCStatus to jobMetrics in getJobMetrics() (PandaServerClient)

Setting jobSubStatus to job.hpcStatus in getNodeStructure() (PandaServerClient)

Making the makeHTTPUpdate() call in the job.workdir instead of __pilot_initdir in updatePandaServer()

Adding --tls to curl command in post(),put(),get() (pUtil)

Added new error codes; 1224 (ERR_ESRECOVERABLE), 1225 (ERR_ESMERGERECOVERABLE) (PilotErrors)

Using ERR_ESRECOVERABLE in failJob() (RunJob)

Now importing RunJobHpcEvent (RunJobFactory)

New files added to distribution (ThreadPool.py, RunJobHpcEvent.py)

Sending mode,hpcStatus,refreshNow,coreCount to pilot from updateJobInfo() (RunJobUtilities)

Receiving mode,hpcStatus,refreshNow,coreCount in handle() (UpdateHandler)

Added exception for S3 objectstore for merge jobs/log files, in mover_get_data() (Mover)

Code merge: Code from Danila Oleynik:

Updated ATLASExperiment

Returning None if token not set in verifyGroupSpaceToken() (SiteMover)

Updated processes

New versions of many of the HPC classes

Code merge: Code from Luke Wayne (AMS):

New version of AMSTaiwanExperiment

New version of AMSTaiwanSiteInformation