UHURA 35a

Summary
* The log extracts for problematic production jobs now shows the athena job report(s). A multi-trf job will show the full job report of the last trf and the header of the previous (successful) trfs' job report. If the job report is not available, the last ten lines of the athena stdout is shown.

* NFS/SQLite locking problems are now detected in multi-trf jobs, previously this only worked for single trfs

* Now checking and verifying file size and checksum for output files at US sites using dCache.

* Pilot cleanup is now picking up the process id of the main process and attempts to kill all of its subprocesses in order to avoid lingering subprocesses in case the pilot is killed by the batch system.

* Download of user trf is now using proxy certificate (previously the wget command had proxy switched off and did not verify the certificate). To be done: verification of base url's.

* Increased the limit of the user workDir from 2 GB to 3 GB for jobs using large DBRelease files.

Version info
Added monitor error code in grep function (pUtil)
Created new functions filterJobReport(), getJobReport() and tail() (pUtil)
Created buildLogExtracts() (pilot)
Removed useless logfile argument in postJobTask() and in buildLogExtracts() (pilot)
Replaced all _n with number_of_jobs (runJob)
Replaced some _i with current_job_number (runJob)
Now detecting NFS/SQLite locking problems in multi-trf jobs, previously this only worked for single trfs (runJob)
Renamed get_file_size() to getdCacheFileSize() and moved it to SiteMover (dCacheSiteMover)
Created getdCacheChecksum() (SiteMover)
Rewrote file size and checksum algorithms in getdCacheChecksum() and getdCacheFileSize() (SiteMover)
Now checking remote output file checksum in dCache against local value (BNLdCacheSiteMover, dCacheSiteMover)
Now checking remote output file size and checksum against local values for dCache sites (lcgcp2SiteMover)
Changed HOLDING to FAILED for ERR_WRONGAD/MD5 errors (runJob)
Renamed writeExitCode to more general writeToFile (pUtil, pilot)
Renamed readExitCode to more general readCodeFromFile (pUtil, pilot)
Storing process id of main process in file PROCESSID (pilot)
cleanup() is now picking up the process id of the main process and attempts to kill all of its
subprocesses in order to avoid lingering subprocesses in case the pilot is killed by the batch system (pilot)
Updated pilot error diag for mismatch in file size in put_data() (BNLdCacheSiteMover)
Changed sleeping time in killProcesses() from 60s to 5s (pilot)
Created a special setup command in setupTrf() to be used by the dq2SiteMover (RunJobUtilities)
Added spsetup argument in put_data() call (Mover)
Special command setup added to envsetup in get_data()/put_data() (dq2SiteMover)
wget is not using options --no-check-certificate and proxy=off anymore (RunJobUtilities)
Introduced localsizelimit_stdout to keep track of local size of payload stdout (pilot)
localspacelimit now only refers to minimum size of work area during running (pilot)
Increased localsizelimit from 2 GB to 3 GB (pilot)
Introduced localspacelimit_user to set a limit of the maximum size of the user work area (pilot)