SULU 58e

Summary

* Cleaned up unnecessary md5/hashlib deprecation warnings in stderr.

Problems seen on python 2.6 sites where the hashlib new() operator was

used explicitly. The new pilot is using hashlib.md5() instead

* Now sending FAX usage info to PandaLogger. Report includes number of

transfers with/without FAX, number of bytes transferred, and time spent

during transfer. Code from Ilija Vukotic. Note: previously the pilot sent

some FAX info with jobMetrics, this has now been removed

* Improved error handling for failures in VmPeak script (executed by the

pilot), to prevent unnecessary warning messages being printed when PerfMon

is not correctly setup (as was seen in some recent evgen jobs)

* Added additional protection to make sure the multi-job loop is aborted

after a job has been killed. Requested by John Hover. Note: tests have not

reproduced any problems with the multi-job loop (using the previous pilot

version)

* Corrected exclusion handling of files allowed in job log. Previously

some files that were wanted in the log were removed before the log was

created. Requested by Wolfgang Ehrenfeld

* Added protection against unexpected print-outs from voms-proxy-info.

Problems seen on a couple of sites where _JAVA_OPTIONS environment

variable was set which triggers an error in the command (command printed

additional information to the expected timing info which caused the pilot

to misread the output ­ pilot is now ignoring nonsense output but is

expecting the last output line to contain timing info). Requested by

Rodney Walker and Jhen Wei Huang

Version info

Removed legacy code related to md5 and hashlib (Mover)
Renamed m to has_userid, hashf to hash_pilotid in getInitialTracingReport() (Mover)
Now importing relevant md5 module/function in getInitialTracingReport() to avoid deprecation warning (Mover)
Applied same trick as above for the getPathFromScope() which previously required hashlib, now works with older python as well (Mover)
Created getFAXDictionary() used by mover_get_data() (Mover)
Now returning FAX dictionary instead of complex FAX tuple from mover_get_data(), get_data() (Mover)
Now receiving FAX dictionary from mover_get_data() in get_data() (Mover)
Now receiving FAX dictionary from get_data() in stageIn(), added extraction of info from dictionary (runJob)
Added bytesWithoutFAX and bytesWithFAX to Job class (Job)
Updated Panda Logger message for FAX changes (above) in updatePandaServer() (PandaServerClient)
Removed legacy code (updatePandaLogger) (PandaServerClient)
Added file size counters for FAX transfers in mover_get_data(), reported to Panda Logger in FAX mode (Mover)
Added file size counters for FAX transfers to jobMetrics in getJobMetrics() (PandaServerClient)
Added protection against None return from PerfMonComps.PMonSD.parse() in the VmPeak script (VmPeak)
Added bytesWith[out]FAX to updateJobInfo() (RunJobUtilities)
Added bytesWith[out]FAX to updateHandler::handle() (pilot)
Now touching a KILLED file in signal handler, to abort multi-job loop after kill signal, requested by John Hover (pilot)
Checking for KILLED file in multi-job loop and aborting if necessary, in runMain() (pilot)
Corrected exclusions in removeRedundantFiles(), requested by Wolfgang Ehrenfeld (ATLASExperiment)
Added protections against additional print-outs from voms-proxy-info, in verifyProxy(). Requested by Rodney Walker (SiteMover)
Registered signal.SIGUSR1 un runMain() (pilot)
Improved log message for local access in getFileAccess() (Mover)
Removed FAX info from jobMetrics, in getJobMetrics() (PandaServerClient)