The new features and fixes include:
* Multi-jobs. Several jobs can now be executed sequentially by the same pilot until it runs out of time. The total run time a multi-job pilot is allowed to run is defined by schedconfig.timefloor (minutes) [currently unset for all sites, so feature is not enabled anywhere as of today]. The primary purpose is to reduce the pilot rate when a lot of short jobs are in the system, and can be used for both production and analysis jobs. Initial testing will use a suggested timefloor of 15-20 minutes. Requested by Michael Ernst.
* Tier 3 modifications. Minor changes to skip e.g. LFC file registrations on tier 3 sites. cp and mv site movers can be used to transfer input/output files. Currently pilot is writing output to
* Further improvements in (especially) get and put pilot error diagnostic messages. Requested by I Ueda.
* Corrected problem with athena setup when option -f was used. Requested by Rod Walker.
* Added pilot support for 5-digit releases. Requested by Tadashi Maeno et al.
* Removed hardcoded slc5 software path from setup path since it is no longer needed. Requested by Alessandro Di Salvo.
* Replaced hardcoded panda servers with pandaserver.cern.ch for queuedata download. Requested by Graeme Stewart.
* Installation problem now recognized during CMTCONFIG verification (NotAvailable error). Requested by Rod Walker.
* Job definition printout now contains special command setup for xrdcp (when available). Note: printout done twice, at the beginning of the job and when all setups are done. Special command setup will only be set in the second printout. Requested by Johannes Elmsheuser et al.
* Corrected undefined variable in local site mover. Requested by Pedro Salgado.
* Minor change in queuedata read function needed for glExec integration to allow queuedata file to be found after identity switching. glExec/pilot integration done in parallel by Jose Caballero.
Version info
Added handling of bad HTTP response in isFileStaged() (dCacheSiteMover)
Removed unnecessary Xrd* env variables for ANALY_CERN (RunJobUtilities)
Created addSPSetupToCmd() (RunJobUtilities)
Corrected introduction of special_command_setup to main run command for -f option (RunJobUtilities)
Removed unsecure command call in evaluate(), rewrote function (pUtil)
Avoiding adding cacheDir and cacheVer for releases containing >= 4 periods, WZBenchmarks,15.5.4.1.1 (RunJobUtilities)
Moved PilotTCPServer outside of multi-job loop (pilot)
Created processQueuedata(), readStringFromFile(), getTimeFloor() (pUtil)
Refactored queuedata handling in runMain() (pilot)
Modified arguments for postProcessQueuedata() (pilot, pUtil)
Renamed createPilotWorkDir() to createSiteWorkDir()
Moved atexit.register() and createSiteWorkDir() outside of multi-job loop (pilot)
Created isTier3() (SiteMover)
Removed unused variables (dst_serv, dst_host) in put_data() (SiteMover)
Added Tier 3 support to put_data() (SiteMover, mvSiteMover)
Shortened copy command error string in get_data()/put_data() (lcgcpSiteMover, lcgcp2SiteMover)
Added size option to tailPilotErrorDiag() (pUtil)
Put and get errors should now be properly tailed using tailPilotErrorDiag() (pilot, runJob, Mover)
Removed CERNVM exception in getLocalSpaceLimit() (pUtil)
Moved getLocalSpaceLimit() from pUtil to Mover, modified arguments (pUtil, Mover)
Now verifying that enough space exists to stage input files and run the job (Mover)
Created verifyInputFileSize(), verifyAvailableSpace(), getFileAccess(), updateReportWithSiteName(), getFileInfoFromMetadata()
used in mover_get_data() (Mover)
Removed unnecessary _token_file from mover_get_data() (Mover)
Moved signal handler definition to before job recovery (pilot)
Corrected wrong return code in addFileInfo() for lfc_statg errors, previously leading to ERR_PUTLFCIMPORT error instead
of ERR_LFCADDCSUMFAILED (SiteMover)
Corrected multiple final server updates (runJob)
Added exception handling in readpar() (pUtil)
readpar() now tries to open queuedata in current dir if it can not read it from PilotHomeDir (needed for glExec) (pUtil)
No space token verification for tier 3s (Mover)
Removed useless variables fsetup, ftx (Mover)
Removed multi task code for user job (pilot)
Removed slc5 software path from swbase, no longer neeed (requested by A. De Salvo) (pUtil)
Removed use of multitask variable (renicing) (runJob)
Added evaluateQueuedata() to postProcessQueuedata() (pUtil)
Removed queuedata evaluation from getQueuedata() (pUtil)
Removed all occurances of multitask flag (including runJob -c option) in pilot and runJob
Only setting special setup command for xrdcp sites when copytool[in] is "xrdcp" (RunJobUtilities)
Created throttleJob(), checkPayloadStdout(), checkLocalSpace(), checkUserWorkDir(), checkOutputFileSizes(), updateEndedJobs(),
cleanUpEndedJobs(), backupJobDef() (pilot)
Created new globals; update_freq_space, update_freq_server (pilot)
Removed too long pilot error diags in put_data() (lcgcpSiteMover, lcgcp2SiteMover, stormSiteMover)
Added info about total file size in verifyInputFileSize() (Mover)
Moved setting of pilot log name to beginning of multi-job loop (pilot)
Removed chttpurl from pUtil
Now looping over pandaserver for queuedata download (pUtil)
Now takes 'pilot: ' into account in Put error pilotErrorDiag messages (pilot, runJob)
Added getWorkDir() to class, removed unused wkdir argument (Site)
Added spsetup to displayJob() (Job)
Installation problem now handled in verifyCMTCONFIG() (pUtil)
Added path to curl.config and relevant function calls (pUtil, pilot)
Removed special_command_setup from setup(), stageIn(), stageOut(), updatePilotServer(), get_data() (runJob)
Removed spsetup from updatePilotServer() (RunJobUtilities)
Removed spsetup from get_data() (Mover)
Increased timed-out limit from 300 s to 600 s in findPythonInRelease() to allow for AFS slowness (pUtil)
Corrected error->self.__error in put_data() (LocalSiteMover)
Added call to overall cleanup function in multi-job loop (pilot)
Allowing packages to have four dots in installPyJobTransforms() (RunJobUtilities)
Changed FAILED to WARNING in installPyJobTransforms() (RunJobUtilities)
No comments:
Post a Comment