SULU 53d

Summary

* Additional support for nightlies on the grid. The previous pilot version
allowed production jobs to use nightlies. This version extends this
functionality to user analysis jobs as well. Tadashi Maeno has updated
prun/pathena as well. Successfully tested at CERN and Wuppertal.
Schedconfig.appdir (e.g.
"/cvmfs/atlas.cern.ch/repo/sw|nightlies^/cvmfs/atlas-nightlies.cern.ch/repo
/sw/nightlies") and job brokering (installedSW table) need to be updated
to complete the machinery. When this has been done, HammerCloud tests will
be initiated.

* Modified/updated support for new DEBUG mode. In case the pilot receives
a debug='true' command from the server OR if the server returns a
'debugon' after a jobUpdate the pilot will start sending the tail of the
payload stdout during server updates (every 5 minutes). The server can
also instruct the pilot to switch off an ongoing debug session using the
command 'debugoff' (after which the pilot will reset the server update
interval to 30 minutes).

* The main site mover (with the default 'cp' copy tool) can now mount and
use NFS4 if necessary. Requested and implemented by Jhen-Wei Huang; to be
used in TW. Related to this, new error codes have been created: 1196: "Get
error: Failed to mount NSF4", 1197: "Get error: PNFS system error". The
new error codes have been added to the list of retry error codes.

* Now using UTC time in pilot wrapper. Requested by Jose Caballero.

* Now setting RUCIO_ACCOUNT to 'pilot' in pilot wrapper (also added to
pyfactory by Peter Love) as well as RUCIO_APPID to job processing type.
Requested by Vincent Garonne.

* Added "http://common-analysis-framework.cern.ch" to list of valid base
URLs used for analysis run script download. Requested by Fernando Barreiro.

* JEM is now available for sites in DE cloud in the standard production
pilot (to avoid having two pilot sources and since some sites receive
pilots from several pyfactories). Requested by Frank Volkmer, approved by
Guenter Duckeck.

Version info

Created mountNSF4AndGetPFN() used by get_data() (SiteMover)

Added new pilot error codes 1196, 1197 (PilotErrors)

Added new error codes to PilotResubmissionErrorCodes (PilotErrors)

Added protection against too large stdout tails (2048 B) in updatePandaServer() (PandaServerClient)

Replaced ctime() calls with time.strftime("%a %b %d %H:%M:%S %Y", time.gmtime(time.time())) in atlasProdPilot

Added additional error recognition in getSiteroot() (pUtil)

Added homePackage to extractAppdir() and its call in runMain() (pUtil, pilot)

Overriding processingType for nightlies analysis jobs in extractAppdir() (pUtil)

Renamed mode to debug in Job

Renamed mode to debug in getStdoutDictionary() (pilot)

Renamed mode to debug in updatePandaServer() (PandaServerClient)

Added support for turning on/off debug mode using action in updateJobs() (pilot)

Updated action handling for tobekilled in updateJobs() and RecoverLostJobs() (pilot)

Updated action handling for tobekilled in updateServer(), recoverWithServer() (JobRecovery)

Updating server frequency in getNewJob() for debug mode (pilot)

Now sending stdout_tail from cleanUpEndedJobs() as well to postJobTask() (pilot)

Receiving stdout_tail in postJobTask() and forwarding it to JobLog() (pilot)

Added stdout_tail handling to __init__(), postJobTask(), updatePandaServer() (JobLog)

Added protection against complex appdir form in getSwbase() (pUtil)

Now setting RUCIO_ACCOUNT to 'pilot' in pilot wrapper (atlasProdPilot)

Now setting RUCIO_APPID in setupTrf() (RunJobUtilities)

Renamed addFRONTIERVars() to addEnvVars2Cmd() (RunJobUtilities)

Now sending job.processingType to addEnvVars2Cmd() (RunJobUtilities)

Added a check for RUCIO_ACCOUNT in checkSpecialEnvVars() (pUtil)

Corrected undefined prodSourceLabel in return statement in getDispatcherDictionary() (pilot)

Added "http://common-analysis-framework.cern.ch" to list of valid base URLs in getValidBaseURLs() (RunJobUtilities)

Allowing JEM on DE cloud in setupTrf() (RunJobUtilities)