SULU 47f

Summary

* lcg-get-checksum is now also used on WEIZMANN-LCG2 and ANALY_WEIZMANN

* New pilot option D added, to be used by CERNVM. Default
CERNVM mode is now to not use the Co-Pilot mode, which means that the
pilot is responsible for transferring output to final storage and send the
final status update to server

* Post-job processing has been updated for Nordugrid to properly handle
local job log transfer of failed jobs (previous version introduced this
problem with an undefined variable leading to an exception). Metadata
handling has also been updated for Nordugrid (the case when the metadata
file does not exist)

* Pilot now keeps track of maximum number of running processes (user job
behaviour to be analysed to see if we need to put an upper limit to the
maximum number of allowed processes started by a user it was recently
discovered that a user started a PROOF lite session through a Panda job
that spawned many sub processes). Current pilot performs the poll every
five minutes and reports the found number in the pilot log

* Updated asetup handling when cmtconfig is in the software path.
Previously the pilot would choose setup.sh ahead of asetup.sh which lead
to problems with 64 bit validation tests. Similar change in find python
algorithm

* Find python algorithm now defaults to 'python' rather than system
python, when the algorithm fails to locate the python in the release.
Requested by Rodney Walker

* The pilot is now verifying that $VO_ATLAS_SW_DIR, $VO_ATLAS_SW_DIR/local
and $VO_ATLAS_SW_DIR/software exists before asking for a job (when
VO_ATLAS_SW_DIR is defined). Requested by Rodney Walker and Ian Collier

* Updated exception handling in the case that the queuedata file could not
be read. Problem seen at MWT2 which led to slightly misleading log error
messages. Actual problem with reading queuedata file currently not
understood

* Corrected bad path in job definition backup leading to file not being
transferred to the right place. Requested by Peter Love

Version info

Added new global variable useCoPilot (pilot)
Added new pilot option -D (pilot)
Added useCoPilot to JobLog() in postJobTask() (pilot)
Added useCoPilot to updatePandaServer() call in updatePandaServer() (pilot)
Added argument useCoPilot to updatePandaServer() (PandaServerClient)
Only setting final holding state for CERNVM if Co-Pilot is used (PandaServerClient)
Added __useCoPilot to __init__() (JobLog)
Added useCoPilot argument to __init__() and to initialization (JobLog)
Sending useCoPilot in updatePandaServer() (JobLog)
Added error handling to checkProcesses() (http://panda.cern.ch/server/pandamon/query?job=1240727862) (pilot)
Added error handling around zlib.adler32 in adler32() (http://panda.cern.ch/server/pandamon/query?job=1240677905) (SiteMover)
checkProcesses() is now returning the number of processes
Now monitoring the maximum number of monitored processes in main job loop (pilot)
Added sitename to findPythonInRelease() arguments (pUtil)
Now using sitename and isTier3() in findPythonInRelease() to strengthen if-statement (pUtil)
Added sitename to setPython() arguments (RunJobUtilities)
Calling findPythonInRelease() with sitename in setPython() (RunJobUtilities)
Sending sitename to setPython() in setupTrf() (three times) (RunJobUtilities)
Updated path for cvmfs in getProperASetup() (pUtil)
Changed order of useAtlasSetup() in findPythonInRelease() - can affect CERNVM jobs (pUtil)
Cleaned up some string conversions in cleanup() (runJob)
Added missing argument logMsg to transferLogExtracts() (JobLog)
Now sending logMsg to transferLogExtracts() in postJobTask() (JobLog)
Updated updateOutputFilesXMLWithSURLs4NG() for the case that the metadata file does not exist (PandaServerClient)
Replaced sys.last_traceback with traceback.print_tb() in safe_call() (pUtil)
Default to 'python' in findPythonInRelease() (pUtil)
Added WEIZMANN-LCG2 and ANALY_WEIZMANN to inclusion list in put_data() (lcgcpSiteMover)
Added VO_ATLAS_SW_DIR dir tests to checkSpecialEnvVars() (pUtil)
checkSpecialEnvVars() is now returning an exit code (pUtil)
checkSpecialEnvVars() return value handled in runMain() (pilot)
Checking processes every five minutes in runMain() (pilot)
Added cmtconfig as argument to useAtlasSetup() (pUtil)
Sending cmtconfig to useAtlasSetup() in findPythonInRelease() and getSiteroot() (pUtil)
Sending cmtconfig to useAtlasSetup() in getAnalyCmd2(), getCmtsiteCmd(), getSpecialSetupCommand(), setupTrf(), installPyJobTransforms() (RunJobUtilities)
Only changing queuedata file name if file actually exists (pUtil)
Corrected bad path in backupJobDef() (pilot)