UHURA 42a

Summary

New features include:

* CERNVM modifications allowing the pilot to communicate with the dispatcher, run jobs inside the virtual machine. Output file handling done in a similar way to NG using mv site mover. Requested by CERNVM team.

* Refactoring in various pilot codes to facilitate glExec integration (next couple of pilot versions) and multi-job handling.

* Error handling added to curl class responsible for server communications.

* Pilot is now searching for orphan processes after found looping job and eliminates them. Requested by John Brunelle.

* Changed 'source setup.sh' to 'source ./setup.sh' in payload setup. Requested by Rod Walker.
* Fix for space token handling related to the issue of datasets being split between MCDISK and MCTAPE. Space tokens did not arrive to the pilot in the expected order. Requested by Stephane Jezequel et al.

* Replaced slc5,gcc43 tags with cmtconfig value in setup used for SITEROOT extraction/dynamic trf installation. Seen to have caused problems at TRIUMF with SL(C)5 jobs (fixed earlier by installing patch release) Requested by Rod Walker.

Version info
findPythonInRelease() updated for CERNVM (pUtil)
Renamed verifyHost to _verifyHost in _Curl() (pUtil)
Created verifyHost() method in _Curl() (pUtil)
toServer() is setting insecure curl mode for voatlas48 server (pUtil)
Some cleanups in RunJobUtilities
Now sending copysetup[in] as an argument to updateCopysetup() (runJob, pUtil)
Created setup(), stageIn(), executePayload(), interpretPayloadOutput(), moveTrfMetadata(), createFileMetadata(),
getDatasets(), stageOut() (runJob)
Refactored __main__(); tested setup(): OK, stageIn(): ?, executePayload(): ?, interpretPayloadOutput(): ? (runJob)
Minor cleanups in dq2SiteMover
tolog() now using timeStampUTC() explicitly (pUtil)
IOError exception handling in _Curl class (pUtil)
Verifying that curl.config exists before trying to send it with the curl command (pUtil)
sys.path.append removed from pilot and runJob
Replaced $USER with whoami in ps commands (pilot)
Created killOrphans() (pilot)
Pilot is now searching for orphan processes after found looping job (pilot)
Replaced many transExitCode's with job.result[1] (runJob)
Looking for --connect-timeout instead of -t timeout in isNewLCGVersion() (SiteMover)
getRemoteFileSize() is now verifying timeout options of lcg-ls command to avoid deprecation warning (SiteMover)
Added envsetup to timeout option verifications (lcgcpSiteMover, lcgcp2SiteMover)
Created getEnvsetup() (SiteMover)
Get and put functions now use getEnvsetup() (lcgcpSiteMover, lcgcp2SiteMover, dq2SiteMover, LocalSiteMover, rfcpLFCSiteMover,
HUSiteMover, dCacheSiteMover, castorSvcClassSiteMover, CastorSiteMover, BNLdCacheSiteMover)
Removed unused status variable from verifyProxyValidity() (pUtil)
Renamed setupstr to envsetup in verifyProxy() (SiteMover)
Explicitly setting envsetup in verifyProxy() calls (CastorSiteMover, COGSiteMover, GSIftpSiteMover, lcgcp2SiteMover,
lcgcpSiteMover, SRMSiteMover, UberftpSiteMover)
Cleaned up unnecessary code in verifyProxyValidity() (pUtil)
Changed 'source setup.sh' to 'source ./setup.sh' in setup() and installPyJobTransforms() (RunJobUtilities)
Added the special command setup to the run command for xrdcp in setupTrf() (RunJobUtilities)
Using string.endswith() in getCmd3WithReleasePy(), setupTrf() (RunJobUtilities)
Moved space token for log file to end of list in setJobDef() (Job)
Avoiding to set ATLAS_CONDDB if schedconfig.gatekeeper is 'to.be.set' in setCONDDBVar() (RunJobUtilities)
Verifying that setup string ends with a ; in get_data() and put_data() (xrdcpSiteMover)
Setting various env variables for xrdcp incl an alias in job command in setupTrf() (RunJobUtilities)
Created getInOutDirs(), diskCleanup(), verifyProxyValidity(), createPilotWorkDir(), getWNMem() (pilot)
Begun refactoring of pilot main (pilot)
Removed unnecessary trial import of lfc module in runMain() (pilot)
Replaced slc5,gcc43 tags with cmtconfig value in getSiteroot() (pUtil)
Added handling of current job number to executePayload() and interpretPayloadOutput() (runJob)
Added cmtconfig to installPyJobTransforms() (RunJobUtilities)
Removed undeclared vo_atlas_sw_dir in findPythonInRelease() (pUtil)
Text formatting correction in makeJobReport() for multi-trf jobs (pilot)
Corrected missing argument in updateCopysetup() call (RunJobUtilities)
Verifying copysetups before calling updateCopysetup() (runJob, RunJobUtilities)