Using Ganga with HPS MC Jobs

DEPRECATED

The hps-mc toolkit has a built-in interface to the Ganga job toolkit, which allows sophisticated management of batch jobs in multiple systems (LSF, PBS, Condor, etc).

Adding Jobs to Ganga

hps-mc includes a workflow script for automatically adding jobs to Ganga.

python hps-mc/python/hpsmc/workflow.py -j 1 -n 10 -w ap -g \
    -d workdir -u -p 6 ap_job.py job.json

Script arguments from the above example:

1. start at job number 1
2. create 10 jobs (1-10)
3. create workflow called 'ap'
4. register the jobs in Ganga
5. set the work dir for the unpacked JSON files (a lot of JSON files will be dumped here!)
6. automatically unpack JSON files to work dir so that we can submit to Ganga
7. use 6 character padding for the job numbers in various filenames
8. use an A-prime gen script (uses hps-mc framework)
9. use job parameters (format for hps-mc jobs)

The -g switch activates automatic creation of Ganga jobs, and -u will automatically unpack all the job JSON files into the work directory so that they are ready for immediate batch submission to LSF.

All the batch job management is now done with Ganga, which will manage all of the LSF jobs.

Using Ganga

To use the Ganga, interface you need to have it installed in the Python installation you are using.

You can setup a pre-installed python at SLAC using this command:

. /nfs/slac/g/hps/hps_soft/conda/setup.sh

Now, in a shell type ‘ganga’ which opens the special command shell.

Basic Job Submission

Type 'jobs' in this shell to see all the jobs in your Ganga repo and their statuses.

All of your jobs that were just created should have a status of ‘new’ and can be submitted using a simple command like this:

ganga> jobs.submit()

Sophisticated job manipulation can now be performed using interactive Python commands or the Ganga shell commands.

Once the jobs are launched, they will be in the LSF system and you may still issue LSF commands like 'bsub' directly from your shell, and Ganga will automatically see the changes.

Job Management

All Ganga jobs are kept in a database which is by default located at ~/gangadir. This keeps track of all yours jobs and their statuses.

Here are some common tasks you may execute in the Ganga shell.

Show All Jobs

jobs

Show All Jobs by Status

jobs.select(status='new')

Status can be 'new', 'running', 'completed', 'submitted' or 'failed'.

Submit All Jobs by Status

jobs.select(status='new').submit()

Submit a Subset of Jobs by ID

You can submit a range of jobs using the built-in slice feature of Python lists.

Suppose that you have ten jobs in total, then this command will submit the first 5.

jobs[0:5].submit()

This will submit the next 5 jobs.

jobs[5:10].submit()

Alternately, if there is a continuous range of jobs to submit, then a command like this can be used.

jobs.select(minid=10, maxid=20).submit()

etc.

Looking at Job Output

Show stdout of a single job.

jobs(10).peek('stdout')

Resubmitting Failed Jobs

jobs.select(status='failed').resubmit()

Kill All Jobs

jobs.kill()

Remove All Jobs

jobs.remove()

Most of these examples were adapted from the Job Manipulation section of the official documentation.

Help!

Questions about this interface or simple usage of Ganga can be posted to the #software channel in the HPS Slack or emailed to the hps software mailing list (the bug tracker of this project is an okay place to post technical questions/comments as well).