UHURA 36a

Summary

* Infrastructural changes, major changes (refactoring) needed for adding pilot support for glExec. Work will be ongoing for several coming pilot releases.

* Verification of URL base is now done for trfs in user jobs. The following URLs are valid trf download locations: http://www.usatlas.bnl.gov/, https://www.usatlas.bnl.gov/,
http://atlpan.web.cern.ch/atlpan, https://atlpan.web.cern.ch/atlpan, http://homepages.physik.uni-muenchen.de/~Johannes.Ebke

* Matching of SE destination path rewritten. Previously, space token name in lower case was required in the path. Matching is now based on position of space token and the corresponding seprodpath in the schedconfig table. NOTE: the length of schedconfig.tokens must be the same as the length of schedconfig.seprodpath.

* Failed SITEROOT pattern matchings were not handled correctly, now fixed (caused some jobs at FZK to report a more difficult to read error message).

* LFC replica sorting has been updated. Previously, tape resident replicas were pushed to the end of the matching list . The remaining replicas were assumed to be on disk. Now, tape resident replicas are pushed to the end of the matching list AND disk resident replicas (with explicit “proddisk”/"mcdisk" present in storage path) are pulled to the beginning of the matching list.

* Added randomization in LFC get replica algorithm for found (= site matches) DBRelease replicas to prevent same file from always being downloaded from the same location. Requested by CNAF.

* Dataset name was not extracted correctly which caused output files to end up in the dataset corresponding to the first output file. Now corrected.

* lcg-cp site movers are now verifying file size after transfer (put function).

Version info
Sending prodUserID (user DN) to site mover as is needed by the DQ2 tracing [yet to be fully implemented] (pilot, runJob, RunJobUtilities, Mover)
Switched off proxy check in get_data (already verified) (runJob)
Now sending spsetup to get_data (needed by dq2SiteMover) (Mover)
Created verifyTrfBaseURL() (RunJobUtilities)
downloadTrf() now verifies that the trf contains a valid URL (RunJobUtilities)
Now checking "0" and not only 0 in file size check in get_data() (dCacheSiteMover)
Corrected wrong order of remote and local file size in pilotErrorDiag in get_data (dCacheSiteMover)
Corrected wrong usage of dsname in dq2-get command (dsname instead of dsn) (dq2SiteMover)
getMatchingDestinationPath() rewritten using space token path based on list position (SiteMover)
Warning messages modified in getDestination() to match new getMatchingDestinationPath() (SiteMover)
Commented-out athena metadata removal (pilot)
tail() protected with exception handling against index out of range errors (pUtil)
Created getProcessCommands() (pilot)
The pilot is now printing info about which processes are still running during cleanup (pilot)
Increased sleep from 5s to 10s in killProcesses() to allow e.g. runJob to exit cleanly (pilot)
Cleaned up useless comments and old code (20kB) from SiteMover, Mover, pUtil and pilot
Removed unused function killCopyProcess() (Mover)
Moved get_data from RunJobUtilities to Mover (runJob, RunJobUtilities, Mover)
Failed SITEROOT pattern matchings were not handled correctly, should be fixed now (caused some jobs at FZK
to report a more difficult to read error) (RunJobUtilities)
Added randomization in get replica algorithm for found DBRelease replicas (Mover)
Created randomizeReplicas() (pUtil)
Created removeDuplicates(), dumpOrderedItems() (pUtil)
Using dumpOrderedItems() in several functions (Mover, SiteMover, runJob, pilot)
Created getRemoteFileSize() using lcg-ls to extract the file size (lcgcp2SiteMover, lcgcpSiteMover, SiteMover)
Now testing remote file size against local value for EGEE sites in put_data() (lcgcpSiteMover)
Now primarily testing checksum and only testing file size if checksum test fails in put_data() (lcgcp2SiteMover)
pullMatchedAreas() moved out of replica loop (Mover)
Corrected 'Payload stdout file too big' message that had the wrong order or the file size (kB not B) (pilot)
Now pushing ahead mcdisk resident replicas to the beginning of the sorted list (Mover)
Renamed all outsDICT to outsDict (runJob, RunJobUtilities)
Created getDatasetDict() which combines the output file and dataset lists into a dictionary (pUtil)
runJob and pilot are now setting the dataset that correspond to the output file rather than using the
first job.destinationDblock entry (runJob, pilot)
convertMetadata4NG() is now using datasetDict (RunJobUtilities)