UHURA 22a

Summary:

Exception handling has been added to protect the bulk of the runJob and pilot code. In case there are exceptions that are not caught by the previously existing error handlers (e.g. around the get and put functions), they should now be caught as well. runJob reports any exception back to the pilot via the pilot server, and the pilot exits gracefully for both cases when a job has been downloaded (log will be created) and when it has not (fast cleanup). The log tarball is created and the server is updated after an exception has been caught.

Caught exceptions will be reported with new error codes 1111 ("Exception caught by runJob") and 1112 ("Exception caught by pilot").

Test exception scenarios:

1. runJob get function: 12717742 (simulated error in Mover.py), 12724859 (simulated error in dCacheSiteMover.py)
2. runJob put function: 12718135 (simulated error in Mover.py)
3. runJob setupTrf: 12717741 (exception in RunJobUtilities::setupTrf(), caught in runJob)
4. runJob executing job commad: 12741244 (testing that hidden proxy is properly restored from disk)
5. pilot prior to job download: OK, no log, server not updated
6. pilot after job download (before server update, before forking of subprocess): OK, no log, server not updated
7. pilot after job download (after server update, before forking of subprocess): 12725354
8. pilot after job download (after forking of subprocess): 12725387
9. pilot could not find job: OK, no log (raises an exception to trigger cleanup)
10. Job killed by batch system: 12718138 (no log, raises an exception to trigger cleanup)
11. Job killed by Panda server: 12724857 (raises an exception to trigger cleanup)
12. Normal finished jobs: 12725353, 12741243, 12724858

Note: the old pilot already handles scenario 1 and 2 correctly. No new actions are taken. runJob and pilot have been updated to handle scenarios 3-11. Scenario 12 is to test that everything works as normal for finished jobs as well..

UHURA 22a version info:

Added exception handling in main functions in runJob, pilot
Added new error codes (1111, 1112) for runJob, pilot exception handling (PilotErrors)
Added missing command string in log message (CastorSiteMover)
Modified usage of schedconfig.appdir to use the value directly (RunJobUtilities)