SULU 54a

Summary 

* Introducing the FAX site mover. In case the pilot fails a repeated
stage-in (default: two attempts), it will try to use FAX to stage-in the
file from a remote storage. This mechanism will only be used if the new
schedconfig fields 'allowfax' is set to True and 'faxredirector' is
defined. It has been tested on one site (UTA) only, but will be further
tested asap in the US (MWT2) and UK (RAL, tbc).

* Introducing the new Experiment classes; ATLAS specifics are now being
moved to new singleton class ATLASExperiment which inherits from the
Experiment class. Implemented methods include retrieval of number of
events, if LFC should be used, special tests, metadata processing, special
file handling (AthenaSummary), etc. CMS (Daniele Spiga) are using this new
structure to implement their version of required methods (not all of the
previously mentioned) in new class CMSExperiment (not yet added to pilot
distribution). Model class OtherExperiment can be used as a starting
point. In relation to the Experiment classes, the pilot can now be
launched with option "-F " (default experiment is ATLAS) used
by the pilot to select the current experiment. The next major pilot
version will implement the getJobExecutionCommand() which will require a
large rewrite.

* Introducing the new Diagnosis and ErrorDiagnosis classes; Diagnosis can
be used for special job analysis features (none are currently implemented
- e.g. JEM handling should be moved here), while ErrorDiagnosis is used
for investigating failed jobs (payload stdout interpretation, extraction
of job parameters such as number of events (using the relevant Experiment
method), and error code identification).

* Partially fixed problem with JSON unicode when using python 2.6 and
lfc_addreplicas(). Note: cannot be fully solved by the pilot due to a
severe bug in the LFC API. A GGUS ticket has been created but no action
yet. Until this issue has been resolved, we cannot use python 2.6 in
combination with LFC registrations. Full details here:
https://ggus.eu/tech/ticket_show.php?ticket=84716

* Added an improved version of timed_command that solved the issue with
transfer+registration timing of 2 minutes on some sites (the commands
returned correctly in less than 2 minutes but the timed_command did not
notice until 2 minutes had passed). Kindly fixed by Ilija Vukotic

* Minor change to NSF4 mounting related to SFN, code from Jhen-Wei Huang

* Improvements to nightlies on the grid, including a correction for appdir
handling on analysis sites (previous pilot version had a problem with
nightlies jobs in multi-job mode).

* Refactoring of pilot module pUtil; site information handling (e.g.
Download, handling and manipulation of queuedata) is now handled by new
pilot module SiteInformation.

* Removed sending of available_releases since it is not needed (not used
by the server)
 

* Replaced internal evaluate() method with os.path.expandvars(). Requested
by Peter Love

* dumpStackTrace() is now avoiding using pstack for zombie processes.
Requested by Asoka de Silva

* Removed trailing slash from copyprefix destination  since it causes
problems when SFN's appear in the PFN's (can also be fixed by removing the
slash in schedconfig as was done for one site)

* Fixed problem with source commands in job_setup script. Requested by
Rodney Walker

* Added scratch did to list of redundant directories that are otherwise
included in the log. Seen e.g. in job 1576405345

* The pilot now knows about maxtime and will take down a running job if it
gets close to the maximum batch job time limit. Added new error code 1213,
ERR_REACHEDMAXTIME

* Changed error message "Athena ran out of memory" to "Payload ran out of
memory"

* Changed error code 1210 diagnosis from "No athena output" to "No payload
output"

* Added signal handling for SIGBUS error (previously this would lead to
lost heartbeat). Added new error code 1206 ("Job killed by signal: SIGBUS")

* Updated error codes 1201-1204 "Job killed from the batch system" -> "Job
killed by signal XXX"

More work is needed for these issues:

* Corrections to TURL based PFC creation, resolving some issues with HC
filestager tests (including harmless appearance of double --lfcHost in job
command). Previous pilot version had a broken file stager handling, most
of which is now restored, but further changes related to file stager and
direct access modes are expected in the next pilot version (including
usage of schedconfig.allowdirectaccess which is currently not set
correctly for most sites ­ details to be discussed).

* The pilot is basically ready to start sending: (preliminary variables)

hs06=<float> shutdownTime=<int> cpuFactor=<float> cpuLimit=<float>
diskLimit=<float> jobStart=<int> memLimit=<int> runLimit=<float>

with jobMetrics. However, the final specifications by the WLCG are not yet
defined. When the specification has become stable (determined by Ulrich
Schwickerath) the pilot will start sending the info after a minor update.
For more info, see
https://twiki.cern.ch/twiki/bin/view/LCG/WMTEGEnvironmentVariables

Version info 

Minor change to mountNSF4AndGetPFN() related to SFN, code from Jhen-Wei Huang (SiteMover)
Created FAXSiteMover using xrdcpSiteMover
Updated mover_get_data() to use FAX (Mover)
Created sitemover_get_data() (Mover)
Added FAXSiteMover to SiteMoverFarm
Added homePackage, processingType to getSwbase() call in setupTrf() (RunJobUtilities)
Added "" (for homePackage and processingType) in getSwbase() call in verifySwbase() (pUtil)
Updated complex appdir handling in getSwbase() (pUtil)
Resetting thisSite.appdir if schedconfig.appdir is set in setUnsetVars() (pUtil)
Automatically setting processingType=nightlies also for production jobs in extractAppdir() (pUtil)
Converted pfn to lfn in getFileInfoDictionaryFromDispatcher() (Mover)
Created stripDQ2FromLFN() used in getFileInfo() (Mover)
Corrected double --lfcHost in job command in getAnalyCmd3() (RunJobUtilities)
Correction for usePFCTurl in getFileTransferInfo() (RunJobUtilities)
Added buildJob argument to getFileTransferInfo() used in getAnalyCmd3() (RunJobUtilities)
Removed dependence on getQueuedataFileName() in copytool() (Mover)
Added SiteInformation import to atlasProdPilot
Using SiteInformation instead of utils in VerifyQueuedata(), getWorkdir(), getrunpars()
Removed extractAppdir() from pUtil
Added extractAppdir() to SiteInformation
runmain() is now using SiteInformation (pilot)
getSwbase() is now using SiteInformation (pilot)
Removed old getpar() and readpar() from pUtil
Created new readpar() using SiteInformation class (pUtil)
Added transferType to shouldPFC4TURLsBeCreated() and its call in mover_get_data() (Mover)
Added argument transferType to mover_get_data() and its call in get_data() (Mover)
Using transferType in shouldPFC4TURLsBeCreated() (Mover)
Removed processQueuedata() and postProcessQueuedata() from pUtil (pUtil)
Added updateCopysetup() to SiteInformation
Using SiteInformation in updateCopysetups() (RunJobUtilities)
Using si.replaceQueuedataField() in getNewJob() (pilot)
Removed updateCopysetup(), evaluateQueuedata(), replaceQueuedataField(), getQueuedata(), replaceJSON(), setUnsetVars() from pUtil
Created ExperimentFactory, Experiment, ATLASExperiment, OtherExperiment
Added new global variable experiment (pilot, runJob)
Added new pilot option -F in usage() and argParser() (pilot, runJob)
Added experiment to jobargs list in runMain() (pilot)
Added NordugridATLASExperiment class
Added experiment argument to mover_get_data(), get_data() (Mover)
Instantiating Experiment in runMain() (pilot)
Instantiating Experiment in main function (runJob)
Instantiating Experiment in mover_get_data() (Mover)
get_data() now using experiment (Mover)

Sending experiment to get_data() in stageIn() (runJob)
Sending thisExperiment to getFileInfo(), getPoolFileCatalog() (Mover)
Added thisExperiment argument to getFileInfo(), getPoolFileCatalog() (Mover)
Updating experiment global in handleQueuedata() for Nordugrid, for now (pilot)
Changed "Athena ran out of memory" to "Payload ran out of memory" in interpretPayloadOutput() (runJob)
Renamed interpretPayloadOutput to interpretPayloadStdout (RunJobUtilities)
Moved interpretPayloadStdout to ErrorDiagnosis
Main function now using new ErrorDiagnosis.interpretPayload() (runJob)
Changed error code 1210 diag from "No athena output" to "No payload output" (PilotErrors)
Renamed error code ERR_NOATHENAOUTPUT to ERR_NOPAYLOADOUTPUT (PilotErrors)
updatePandaServer() now only uses nodename instead of full workerNode structure (PandaServerClient)
All calls to updatePandaServer() now using workNode.nodename (pilot, JobLog, JobRecovery)
Main function now using Experiment::specialChecks() (pilot)
Created getAppdirs() (used by ATLASExperiment) (SiteInformation)
Removed unused getSiteSpecs() (pUtil)
Removed available_releases from getJob(), getNewJob(), getDispatcherDictionary() (pilot)
Removed unused getAvailableReleases() and its call in runMain() (pilot)
Added signal handling for SIGBUS (pilot, runJob)
Updated error codes 1201-1204 "Job killed from the batch system" -> "Job killed by signal" (PilotErrors)
Added new error code 1206 ("Job killed by signal: SIGBUS") (PilotErrors)
Removed evaluate() (pUtil)
Replaced all calls to evaluate() with os.path.expandvars() (pUtil, DBReleaseHandler, SiteInformation)
Created isZombie() (pilot)
dumpStackTrace() is now avoiding using pstack for zombie processes (pilot)
Added modified timed_command (by Ilija Vukotic)
Created getMaxtime() used by runMain() (pilot)
Added new error code 1213, ERR_REACHEDMAXTIME (PilotErrors)
Removed trailing slash from pto in getCopyprefixFromTo() since it causes problems when SFN is in gpfn, see eg xrdcpSiteMover (pUtil)
Created extractPathFromPFN() (SiteMover)
specialChecks() now using setPilotPythonVersion() (ATLASExperiment)
Created testImportLFCModule() (ATLASExperiment)
specialChecks() now using testImportLFCModule() (ATLASExperiment)
addToJobSetupScript() is now writing out which command it has added to job_setup.py (pUtil)
Fixed problem with source command in job_setup script (xrdcpSiteMover, lcgcpSiteMover, stormSiteMover)
Added missing source command in createJobSetupScript() (pUtil)
Fixed problem with JSON unicode when using python 2.6 in getpar() (SiteInformation)
Moved getNumberOfEvents from RunJobUtilities to ATLASExperiment
Moved processAthenaSummary() and processMetadata() from RunJobUtilities to ATLASExperiment
Moved isOutOfMemory() from RunJobUtilities to ATLASExperiment
Removed useless failed boolean from isOutOfMemory() (ATLASExperiment)

Added new option fake in lfc_mkdir() for testing (SiteMover)
Simplified boolean expression in shouldPFC4TURLsBeCreated() (Mover)
Corrected stripPath() usage in getTURLFileInfoDic() (Mover)
Corrected stripPath() handling of :// instead of only // (SiteMover)
Now only updating copysetup if copysetupin is empty in updateCopysetups() (RunJobUtilities)
Changed return tuple for getAnalyCmd3() (RunJobUtilities)
getAnalyCmd3() can now abort setup (RunJobUtilities)
Enabled FAX in mover_get_data() (Mover)
Added scratch dir to rmRedundants() seen in job 1576405345 (JobLog)