SULU 50a

Summary

* The JEM module has been updated (Tim dos Santos). JEM module now gets
notified of job-termination in addition to job-start. It also uses JEM
from cvmfs if available, and will reduce traffic and load on the JEM
services and removes a slight delay in job execution

* Curl based site mover, tested at ANALY_TAIWAN_TEST. The initial version
supports stage-in using curl (stage-out is in development, to be tested
during next week). Developed by Jhen-Wei Huang and Shu-Ting Ling (TW)

* Now aborting stage-in/out retry if the file could not be removed from
storage/catalog after a failed stage-in/out

* Updated xrdcp site mover. Xrdcp can now be used for stage-out at CERN.
LFC registration is done by pilot for CERN in this case [or by the PanDA
server..]. For this to work, schedconfig.copyprefix should be set to
"srm://srm-eosatlas.cern.ch/^root://eosatlas.cern.ch/" and
schedconfig.copytool to "xrdcp"

* Now only sourcing setup outside cmtsite area if release < 16.1.0.
Previous pilot version sourced both that setup file as well as the asetup
file for all releases

* Using pandaserver.cern.ch for job updates instead of hardcoded
randomized URL list (voatlas57-59, which will be retired). Requested by
Serguei Baranov et al

* Now setting environment variables PANDA_SITE_NAME and COPY_TOOL before
running job. Requested by Ilija Vukotic and Wahid Bhimji

* Re-enabled the possibility of sending personal pilots (using pilot
option '­u self'). The pilot will only pick up the job corresponding to
the DN of the owner of the pilot proxy. Code from Rodney Walker

* Updated multi-job error handling. In case of stage-in/out errors, the
pilot will take a random nap of 2,3 or 4 minutes before asking for a new
job. The pilot will now abort after a maximum of three failed jobs. Next
major pilot version will perform an SE test before asking for any jobs.
Requested by Christopher Walker (QMUL) et al

Version info

Added GOSiteMover (SiteMoverFarm) [out-commented until subprocess alternative is used, fails with python 2.3]
Changed copycommand to globusonline (GOSiteMover)
Renamed setUpdateFrequency() to setUpdateFrequencies() (pilot)
Changed global variables to local variables: update_freq_proc, update_freq_space (pilot)
setUpdateFrequencies() now returning tuple, used in runMain() (pilot)
Renamed misleading updateEndedJobs() to updateJobs(), used in runMain() (pilot)
Added handling of "debug" mode in updateJobs() (pilot)
Lowering update_freq_server to 5 minutes in case "debug" mode is recived from server, in updateJobs() (pilot)
Created getStagingRetry() used by handleQueuedata() (pilot)
Number of stage-in/out retries can now be set using schedconfig values stage[in|out]retry [need another pilot update to work] (pilot)
Allowed LFC registeration by pilot for CERN if xrdcp is used for stage-out in mover_put_data() (Mover)
Changed default exit code from 0 to -1 in removeFile() (SiteMover)
Added "failed to remove file"-string to pilotErrorDiag after removeFile() failure in put_data() (lcgcpSiteMover, lcgcp2SiteMover)
Now aborting retry if the file could not be removed from storage/catalog in mover_put_data() (Mover)
Corrected return string (rs -> o) in copy() (xrdcpSiteMover)Rewrote getChecksum() to handle either md5 or adler32 checksums (xrdcpSiteMover)
Changed put_data() to use adler32 checksums (xrdcpSiteMover)
Added call to getSpecialSetupCommand() in setupTrf() generic job section (RunJobUtilities)
Updated getSpecialSetupCommand() to not require siteroot if not necessary (RunJobUtilities)
Now setting hardcoded path for xrdcp for production jobs as well in getSpecialSetupCommand() (RunJobUtilities)
Changed default checksum command from md5sum to adler32 (xrdcpSiteMover)
Created getCmtconfig() (pUtil)
setupTrf() is now using getCmtconfig() (RunJobUtilities)
stageOut() is now using getCmtconfig() (runJob)
Sending cmtconfig to mover_put_data() in stageOut() (runJob)
Added cmtconfig argument to mover_put_data() (Mover)
Sending cmtconfig to put_data() in mover_put_data() (Mover)
put_data() is now using cmtconfig to figure out path to xrdcp command at CERN (xrdcpSiteMover)
Now only using setup outside cmtsite (cmd0) if release < 16.1.0 in setupTrf() (RunJobUtilities)
Not using getRandomURL() any longer in makeHTTPUpdate(), using pandaserver.cern.ch (pUtil)
Not using getRandomURL() any longer in checkLocalDDMSpace() and getNewJob(), using pandaserver.cern.ch (pilot)
Removed unused function getRandomURL() (pUtil)
Using getCmtconfig() in moveLostOutputFiles() (pilot)
Sending cmtconfig to mover_put_data() in moveLostOutputFiles() (pilot)
Using getCmtconfig() in transferLogFile(), transferAdditionalFile() (JobLog)
Sending cmtconfig to mover_put_data() in transferLogFile(), transferAdditionalFile() (pilot)
Renamed setCONDDBVar() to setEnvVars4() (RunJobUtilities)
Now setting PANDA_SITE_NAME and COPY_TOOL in setEnvVars() (RunJobUtilities)
Created verifyPaths() used by put_data() (SiteMover, xrdcpSiteMover)
put_data() now using cmtconfig to setup xrdcp command (xrdcpSiteMover)
Added 0 to beginning of adler checksum if xrdcp returns a length 7 checksum in getChecksum() (xrdcpSiteMover)
Removed special_setup_cmd for CERN in getSpecialSetupCommand() (RunJobUtilities)
Added cmtconfig handling to get_data() (xrdcpSiteMover)
Sending cmtconfig to mover_get_data() in get_data() (Mover)
Added cmtconfig argument to mover_get_data() (Mover)
Sending cmtconfig to get_data() in mover_get_data() used by xrdcp site mover (Mover)
Removed useless sitename argument from copytool(), used in mover_get_data(), mover_put_data(), check_localSE_space() (Mover)
Created checkLocalSE() (Mover)
Added new error code 1193, ERR_SEPROBLEM (PilotError)
New JEMstub module from Tim
Added end of job callback to JEM in executePayload() (runJob)
Added new site mover for curl (curlSiteMover, SiteMoverFarm)
getDispatcherDictionary() now returns pilotErrorDiag (pilot)
Added code for extracting user DN in case of user=self in getDispatcherDictionary(). Code from Rodney Walker (pilot)
Added JEM directory to rmRedundants() list (JobLog)