Summary
* Support for nightlies on the grid. Successfully tested at CERN
* Support for new DEBUG mode. In case the pilot receives a mode='debug' command from the server it will start sending the tail of the payload stdout during server updates. It will also add the number of lines of the stdout (currently only when mode='debug'). Note: this functionality needs further development on the server, monitor and possibly pilot side
* Simplified and improved pilot monitor log messages; now printing current job state (starting, setup, stage-in, running, stage-out, finishing); e.g.
11 May 13:57:24| pilot.py | --- Main pilot monitoring loop (job id 1495306772, state:stageout, iteration 7)
(iteration N is just a counter of this type of log message)
replacing less informative log messages like
11 May 13:57:14| pilot.py | --- Main pilot monitoring loop (job id 1495318004, iteration 6)
11 May 13:58:14| pilot.py | ---pollChildren: {'prod': [3701,, 31628]}
* Skipping BDII in lcg-get-checksum calls, requested by Rodney Walker
* Added time-out protections for the 'which trf' command as well as for the dynamic release install command
* Added the "fast" option to the asetup.sh command
* Added various setup file verifications in site movers and some other functions
* Corrected wrong dCache -> BNLdCache strings in tracing report name
* Added exception handling in _Curl class (could previously lead to a thrown exception in rare cases of network problems)
* Added a grace time of 5 minutes for hanging stage-out commands in case file removal is not complete, in looping job killer
* Fixed a problem with zero size athena_stdout files in tail() (previously lead to unnecessary warning messages)
* Added further core file recognition (now recognizing core.* and core files, improving error reporting)
* Corrected wrong variable used in file removal command in lcgcp2 site mover (previously it would fail to remove file remnant after a failed file transfer)
* Pilot is now reporting job metrics:
Number of events read; Number of events written; Vmem maximum; Vmem average; RSS average;| JEM on/off
using the following format:
jobMetrics = "nEvents=<int> nEventsW= <int> vmPeakMax= vmPeakMean= RSSMean= JEM=<0|1>"
Note 1: The pilot currently zero suppresses jobMetrics, i.e. if any of the above is set to zero, 'keyword'=0 will not be included in the string
Note 2: Job metrics are uploaded on each server update, i.e. not all keywords might have values on all updates (nEvents is not known until after the payload has run e.g.)
Note 3: Will be documented on pilot wiki - not yet done
* JEM update from Tim dos Santos; the JEM code now sends metadata back to the pilot. Part of this metadata is included in jobMetrics (JEM on/off), more can be added if necessary (JEM failure info e.g.)
* The pilot is now using longer time-outs (1000 s) on cvmfs (otherwise 500 s). Requested by Alessandro di Salvo et al
* Pilot version now stored in file PILOTVERSION, which will be useful for cvmfs versioning scripts. Pilot source will soon be available on cvmfs, after which the wrappers don't need to download it from the server. The version file will be needed until pilot svn area will be updated to properly use branches etc
* Removed outdated xrdcp aliases as well as updated default xrdcp path in xrdcp site mover put function (caused problems with new xrdcp version at RAL). Note: not fully tested; possibly enough with a proper schedconfig.copyprefix entry
* Support for new DEBUG mode. In case the pilot receives a mode='debug' command from the server it will start sending the tail of the payload stdout during server updates. It will also add the number of lines of the stdout (currently only when mode='debug'). Note: this functionality needs further development on the server, monitor and possibly pilot side
* Simplified and improved pilot monitor log messages; now printing current job state (starting, setup, stage-in, running, stage-out, finishing); e.g.
11 May 13:57:24| pilot.py | --- Main pilot monitoring loop (job id 1495306772, state:stageout, iteration 7)
(iteration N is just a counter of this type of log message)
replacing less informative log messages like
11 May 13:57:14| pilot.py | --- Main pilot monitoring loop (job id 1495318004, iteration 6)
11 May 13:58:14| pilot.py | ---pollChildren: {'prod': [3701,
* Skipping BDII in lcg-get-checksum calls, requested by Rodney Walker
* Added time-out protections for the 'which trf' command as well as for the dynamic release install command
* Added the "fast" option to the asetup.sh command
* Added various setup file verifications in site movers and some other functions
* Corrected wrong dCache -> BNLdCache strings in tracing report name
* Added exception handling in _Curl class (could previously lead to a thrown exception in rare cases of network problems)
* Added a grace time of 5 minutes for hanging stage-out commands in case file removal is not complete, in looping job killer
* Fixed a problem with zero size athena_stdout files in tail() (previously lead to unnecessary warning messages)
* Added further core file recognition (now recognizing core.* and core files, improving error reporting)
* Corrected wrong variable used in file removal command in lcgcp2 site mover (previously it would fail to remove file remnant after a failed file transfer)
* Pilot is now reporting job metrics:
Number of events read; Number of events written; Vmem maximum; Vmem average; RSS average;| JEM on/off
using the following format:
jobMetrics = "nEvents=<int>
Note 1: The pilot currently zero suppresses jobMetrics, i.e. if any of the above is set to zero, 'keyword'=0 will not be included in the string
Note 2: Job metrics are uploaded on each server update, i.e. not all keywords might have values on all updates (nEvents is not known until after the payload has run e.g.)
Note 3: Will be documented on pilot wiki - not yet done
* JEM update from Tim dos Santos; the JEM code now sends metadata back to the pilot. Part of this metadata is included in jobMetrics (JEM on/off), more can be added if necessary (JEM failure info e.g.)
* The pilot is now using longer time-outs (1000 s) on cvmfs (otherwise 500 s). Requested by Alessandro di Salvo et al
* Pilot version now stored in file PILOTVERSION, which will be useful for cvmfs versioning scripts. Pilot source will soon be available on cvmfs, after which the wrappers don't need to download it from the server. The version file will be needed until pilot svn area will be updated to properly use branches etc
* Removed outdated xrdcp aliases as well as updated default xrdcp path in xrdcp site mover put function (caused problems with new xrdcp version at RAL). Note: not fully tested; possibly enough with a proper schedconfig.copyprefix entry
Version info
getProdCmd3() is now using timedCommand() and is thus catching timed out "which trf" cases (RunJobUtilities)
installPyJobTransforms() is now using timedCommand() (RunJobUtilities)
Added "fast" option to asetup in getProperASetup() (pUtil)
Created verifySetupCommand(), extractFilePaths() (pUtil)
get_data() is now using verifySetupCommand() (rfcpLFCSiteMover, xrdcpSiteMover, castorSvcClassSiteMover, curlSiteMover)
put_data() is now using verifySetupCommand() (dCacheLFCSiteMover)
get_data() and put_data() are now using verifySetupCommand() (lcgcpSiteMover, lcgcp2SiteMover, LocalSiteMover, xrootdSiteMover, CastorSiteMover,
dCacheSiteMover, BNLdCacheSiteMover)
get_data() and put_data() are now using getStubTracingReport (lcgcpSiteMover, lcgcp2SiteMover, xrootdSiteMover, SiteMover, CastorSiteMover,
dCacheSiteMover, BNLdCacheSiteMover, castorSvcClassSiteMover, curlSiteMover)
Corrected dCache -> BNLdCache in tracing report name in get_data() (BNLdCacheSiteMover)
Added exception handling in _Curl class (pUtil)
Added a grace time of 5 minutes for hanging stage-out commands in case file removal is not complete, in loopingJobKiller() (pilot)
Fixed a problem with zero size athena_stdout files in tail() (pUtil)
Rewrote removeCoreDumps(), added further core file recognition (now recognizing core.* and core) (JobLog)
Added support for comma-separated homePackages in getCmtsiteCmd() (RunJobUtilities)
getSiteroot() is now using verifySetupCommand() (pUtil)
Now calling getSiteroot() with homePackage from setupTrf() (RunJobUtilities)
Now calling getSiteroot() with homePackage from getHighestVersionDir() (pUtil)
Now calling getHighestVersionDir() with homePackage from isForceConfigCompatible() (pUtil)
Now calling isForceConfigCompatible() with homePackage from findPythonInRelease() (pUtil)
Now calling findPythonInRelease() with homePackage from setPython() (RunJobUtilities)
Now calling isForceConfigCompatible() with homePackage from getCmtsiteCmd() (RunJobUtilities)
Now calling getProperASetup() with (home)package from getCmtsiteCmd(), getSpecialSetupCommand(), installPyJobTransforms() (RunJobUtilities)
Now calling getProperASetup() with homePackage from findPythonInRelease() (2 times), getSiteroot() (pUtil)
Now calling getSpecialSetupCommand() with job.homePackage from setupTrf() (3 times) (RunJobUtilities)
Now calling setPython() with job.homePackage from setupTrf() (3 times) (RunJobUtilities)
Now calling useAtlasSetup() with homePackage from findPythonInRelease() (2 times), getSiteroot() (pUtil)
Now calling useAtlasSetup() with homePackage/job.homePackage/package from getAnalyCmd2(), getCmtsiteCmd(), getSpecialSetupCommand(),
setupTrf(), installPyJobTransforms() (RunJobUtilities)
Added homePackage argument to getHighestVersionDir(), findPythonInRelease(), getProperASetup(), useAtlasSetup() (pUtil)
Added homePackage argument to getSpecialSetupCommand(), setPython() (RunJobUtilities)
Created extractRelN() (pUtil)
Using extractRelN() in getProperASetup() (pUtil)
Updated useAtlasSetup() for rel_N (nightlies) (pUtil)
Removed checkLocalDDMSpace(), out-commented call since before (pilot)
Added new member mode to Job class (Job)
Removed buildLogExtracts(), moved to JobLog since before (pilot)
Created getStdoutDictionary() (pilot)
Removed pollChildren and updateHandler log messages (pilot)
Calling getStdoutDictionary in updateJobs() (pilot)
Now sending stdout_tail to updatePandaServer() in updateJob() (pilot)
Now sending stdout_tail to PandaServerClient.updatePandaServer() in updatePandaServer() (pilot)
Added new argument stdout_tail to updatePandaServer() (pilot, PandaServerClient)
Now sending stdout tail with job updates in updatePandaServer() (PandaServerClient)
Removed alias usage in getSpecialSetupCommand() (RunJobUtilities)
Skipping BDII in lcgGetChecksum(), requested by Rodney Walker (SiteMover)
Added argument nolfc to removeFile() (SiteMover)
Now using nolfc option in removeFile() call in put_data() (lcgcp2SiteMover)
Corrected dst_gpfn -> full_surl in removeFile() call in put_data() (lcgcp2SiteMover)
processAthenaSummary() now grabs and returns number of written events as well as the read events (RunJobUtilities)
Added nEventsW to Job class (Job)
getNumberOfEvents() now returns nEventsWritten. Removed some useless nEvents_str (RunJobUtilities)
Now setting job.nEventsW in interpretPayloadOutput() (runJob)
Now receiving nEventsW in updateHandler::handle() (pilot)
Now sending nEventsW in updateJobInfo() (RunJobUtilities)
Now adding nEventsW to node structure in getNodeStructure() (PandaServerClient)
Corrected confusing log message in createPoolFileCatalog() (pUtil)
Created updateServer() intermediary function (JobRecovery)
removeFile() can now remove an LFC directory (SiteMover)
Created useNoLFC() used in put_data() (lcgcpSiteMover)
Now using removeFile() in file corruption cases in put_data() (lcgcpSiteMover)
Removed "no time to send log" from sig2exc() (pilot)
Adding a potentially missing slash sign in pfrom and hostname in put_data() (xrdcpSiteMover)
Added currentState to Job class (Job)
Setting job.currentState in setState() (Job)
Setting job.currentState in pollChildren(), handle(), killLoopingJob(), checkPayloadStdout(), checkLocalSpace(), checkUserWorkDir(),
checkOutputFileSizes(), updateJobs(), sig2exc(), runMain() (6 places), moveLostOutputFiles() [skipped recovery function] (pilot)
Now displaying current job state in "Main pilot monitoring loop" log message in runMain() (pilot)
Now attempting remove file removal even if file size is reported as zero in removeFile() (file can still be registered in LFC) (SiteMover)
Added MC11JobOptions to rmRedundants() (JobLog)
Created lfcrm(), used by removeFile() [for testing only] (SiteMover)
Created SiteInformation class to consolidate schedconfig methods (SiteInformation)
Now sending nEventsW with jobMetrics node info in getNodeStructure() (PandaServerClient)
Created findVmPeaks(), getStdoutFilename() (RunJobUtilities)
Calling findVmPeaks() from interpretPayloadOutput() (runJob)
Added correction for nightlies in getInstallDir(), getProdCmd2() (RunJobUtilities)
Stopped calling collectWNInfo() twice before getJob() (pilot)
Added job.vmPeakMax, job.vmPeakMean, job.RSSMean to Job class (Job)
Added job.vmPeakMax, job.vmPeakMean, job.RSSMean, job.JEM to updateJobInfo() (RunJobUtilities)
Receiving job.vmPeakMax, job.vmPeakMean, job.RSSMean, job.JEM in handle() (pilot)
Created VmPeak script (VmPeak)
Added missing argument runCommandList to interpretPayloadOutput() (runJob)
Now sending runCommandList to interpretPayloadOutput() in main() (runJob)
Added JEM to Job class (Job)
Sending metaOut to updateRunCommand4JEM() in setupTrf() (RunJobUtilities)
setupTrf() is now returning the JEM (RunJobUtilities)
The JEM is now received in setup() (runJob)
Created getProperTimeout() (pUtil)
getProperTimeout() used in findPythonInRelease(), verifyCMTCONFIG(), getSiteroot() (pUtil)
getProperTimeout() used in getProdCmd3(), findVmPeaks() (RunJobUtilities)
Created getPilotVersion() (pUtil)
Pilot version is now read from file (pilot)
Modified default copytool path in put_data() (xrdcpSiteMover)
Modified return check of getProdCmd2() from cmd2 to pilotErrorDiag in setupTrf() (RunJobUtilities)
Removed unnecessary string conversion using str() with pilotTiming in getNodeStructure() (PandaServerClient)
No comments:
Post a Comment