jobsub jobs and queries will be affected by jobsub02.fnal.gov draining starting Friday, Oct. 25

WHAT ARE WE DOING?
Computing will start draining jobsub02.fnal.gov in preparation for the jobsub server upgrade to version 1.3.

WHEN WILL THIS OCCUR?
On Friday, Oct. 25, 2019, jobsub02 will no longer accept any new job submissions; however, jobs that were running before Oct. 25 will be allowed to finish before the server is upgraded.

WHAT IS THE IMPACT TO YOU?
Jobs in the queue on jobsub02 on Monday, Oct. 28, 2019, will be purged, and users will need to resubmit those jobs.

Due to the draining, jobsub_q queries will not be polling jobsub02 for jobs, meaning that users will not be able to query the jobs in the queue there. We recommend using the following FIFEMON pages instead to check job status during the draining:
• User Batch Details: https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1
• Experiment Batch Details: https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1

If you need to hold, release, or remove any jobs, you can do so using the jobsub_hold, jobsub_release, and jobsub_rm commands. However, any jobs on the jobsub02 server will have to be held, released, or removed individually, as they will be left out of blanket requests.

For example, <jobsub_hold -G myexperiment –user=me> will hold all jobs on jobsub01 and jobsub03 belonging to user “me” for experiment “myexperiment,” but jobs on jobsub02 WILL BE UNAFFECTED.

To hold jobs 123456789.0@jobsub02.fnal.gov and 987654321.0@jobsub02.fnal.gov (belonging to user “me” for experiment “myexperiment”), you will have to run the following commands:
• jobsub_hold -G myexperiment –user=me –jobid=123456789.0@jobsub02.fnal.gov
• jobsub_hold -G myexperiment –user=me –jobid=987654321.0@jobsub02.fnal.gov

This is a temporary condition. Once the drain and upgrade of jobsub02 is complete, those jobs will be accessible via more general jobsub_q, jobsub_hold, jobsub_release, and jobsub_rm queries like before.

If you have any questions, please open a Service Desk ticket for Distributed Computing Support Team.