SULU 55a

Summary

* New SiteInformation classes containing ATLAS/"other experiment"
specifics, mainly related to queuedata handling (CMS can now have their
own methods for download and handling of their corresponding queuedata)

* Updated pre-pilot wrapper (atlasProdPilot) to handle pickle or JSON
format of queuedata dynamically (enabling US sites to use python 2.6)

* Refactoring of stage-out methods in preparation for alternative
stage-out algorithm (the algorithm is being tested but not fully
implemented yet)

* Refactoring of stage-in methods in preparation for using a new site
mover for bulk download, based on aria2c, (the site mover is being
developed by Rodney Walker)

* Now verifying that a lost job to be recovered ran on the same site as
the current pilot to avoid any issues on two sites with shared WNs

* Importing ErrorDiagnosis early in the pilot to avoid problems seen
(only) at BU with a missing pilot modules

* Changed to absolute permissions when creating work directories to avoid
recovery directories to end up being world readable, requested by QMUL

* Added --cacert option for curl operations (SL6 support)

* Pilot is now verifying the size of user job's work dir (job.workdir)
instead of the user workDir (job.workdir/workDir), using the new
schedconfig.maxwdir (it this is not set, it will instead use
schedconfig.maxinputsize + 2 GB as a maximum limit). Requested by Ikuo
Ueda et al

* Pilot is now listing (not yet reporting) the latest draft of the job and
machine features on the WN (see any job log), as outlined in a talk by
Ulrich Schwickerath
(http://indico.cern.ch/conferenceDisplay.py?confId=155073). Draft is close
to being finalised. One of the next pilot versions will start reporting
these to the server

* Added new version of timed_command (the pilots' internal time-out
module) from Ilija Vukotic. Additional exception protections have been
added and some code was removed to prevent spending a lot of time closing
file descriptors on some sites

* Now creating SE directories for castor sites during xrdcp stage-out
(requested by Alastair Dewhurst, in testing)
 
Version info

Created SiteInformationFactory
Created ATLASSiteInformation
Created OtherSiteInformation
Created getSiteInformation() (pUtil)
Created isTier1() (*SiteInformation)
Moved getTier1List() from SiteInformation to ATLASSiteInformation
getNewJob(), handleQueuedata(), runMain() are now using getSiteInformation() (pilot)
Removed isTier3() from pUtil
Now sending experiment string in findPythonInRelease() call in setPython() (RunJobUtilities)
Now sending experiment string in setupTrf()() call in setup() (runJob)
Now sending experiment string in setPython() call in setupTrf(), three times (RunJobUtilities)
Added experiment argument to findPythonInRelease(), setPython(), setupTrf() (RunJobUtilities)findPythonInRelease() is now using getSiteInformation getSiteInformation() (pUtil)
findPythonInRelease() is now using getSiteInformation getSiteInformation() (pUtil)
Now using getSiteInformation() in mover_put_data() (Mover)
Added experiment argument to mover_put_data() (Mover)
Sending experiment to mover_put_data() in stageOut() (runJob)
Sending experiment to mover_put_data() in transferLogFile(), transferAdditionalFile() (JobLog)
Added experiment argument to transferLogFile(), transferAdditionalFile(), transferAdditionalCERNVMFiles(), postJobTask() (JobLog)
Sending experiment to transferAdditionalFile() in transferAdditionalCERNVMFiles() (JobLog)
Sending experiment to transferAdditionalCERNVMFiles(), transferLogFile() in postJobTask() (JobLog)
Sending experiment to postJobTask() in postJobTask() (pilot)
Updated ReadQueueParams() to use SiteInformation.getQueuedataFileName() (enabling US sites for python 2.6 and JSON) (atlasProdPilot)
getSpecialSetupCommand() now using getSiteInformation() and si.istier3() (RunJobUtilities)
Added experiment argument to getSpecialSetupCommand() (RunJobUtilities)
Sending experiment to getSpecialSetupCommand() in setupTrf(), three times (RunJobUtilities)
Sending experiment to transferLogFile() in transferLogFile() (pilot)
Extracting experiment in get_data() (SiteMover)
Using si.istier3() in get_data() (SiteMover)
Sending experiment to get_data() in sitemover_get_data() (Mover)
Added experiment argument to sitemover_get_data() (Mover)
Sending experiment to mover_get_data() in mover_get_data() (Mover)
Extracting experiment in put_data() (SiteMover)
Using si.istier3() in put_data(), two times (SiteMover)
Sending experiment to put_data() in mover_put_data() (Mover)
Sending experiment to chirp_put_data() in mover_put_data() (Mover)
Added experiment argument to chirp_put_data() (Mover)
Sending experiment to put_data() in chirp_put_data() (Mover)
Extracting experiment in get_data() (mvSiteMover)
Using si.istier3() in get_data() (mvSiteMover)
Created doAlternativeStageOut(), called from mover_put_data() (Mover)
Added argument alt to getQueuedataFileName(), getQueuedata(), processQueuedata(), readpar() (SiteInformation)
Now supporting alternative filename for queuedata in getQueuedataFileName() (SiteInformation)
Now sending alt option to getQueuedataFileName() in getQueuedata(), two times (SiteInformation)
Now sending alt option to getQueuedata() in processQueuedata() (SiteInformation)
Now sending alt option to self.getQueuedataFileName() in readpar() (SiteInformation)
Created getFileList() used by mover_put_data() (Mover)
Renamed analJob to analysisJob in get_data() call in stageIn() (runJob)
Renamed analJob to analysisJob in mover_get_data() call in get_data() (Mover)
Renamed analJob argument to analysisJob in get_data(), mover_get_data(), chirp_put_data(), put_data(), etc (Mover)
Created getSpaceTokenList() used by mover_put_data() (Mover)
Created getFilenamesAndGuid() used by mover_put_data() (Mover)
Created getSpaceTokenForFile() used by mover_put_data() (Mover)
Created isOneByOneFileTransfer() used by mover_put_data() in Mover (SiteMover)
Renamed analJob to analysisJob in call to mover_put_data() in transferLogFile(), transferAdditionalFile() (JobLog)
Added support for aria2c in SiteMoverFarm
Replaced extraction of metadata with lfn, guid from put_data() arguments (lcgcpSiteMover, CastorSiteMover, dCacheLFCSiteMover, rfcpLFCSiteMover)
Renamed metadata to ddm_storage in unused put_data() (castorSvcClassSiteMover, OtherSiteMover)
Removed metadata in ddm_storage variable in mover_put_data() (Mover)