Script to do release comparison

[nothingface0]

Introduction

This PR adds a new script (cmssw_release_comparison.py) which combines the functionality of cmssw_deploy.py, an execution of the DQM clients and compareDQMOutput.py into a single script. The script is meant to be used in Point 5 machines.

ZhengChen/Praveen's scripts were used as a reference to make this script. The main difference is that this is a single script, written in python, takes all arguments from the commandline and does not require modifying the compareDQMOutput.py and compareHistograms.py scripts from CMSSW.

Tested with python3.8. No extra dependencies are introduced.

Functionality

The script works like so:

1. It creates two CMSSW releases (base and comp) with PRs optionally applied on them, using cmssw_deploy.py,
2. It changes their Global Tags and builds them, then
3. Runs the DQM clients, for each of the two releases, storing their output files. If any client returns a non-zero exit code, the script raises an exception.
4. Then, compareDQMOutput.py is executed to compare the output files of the two releases, upload the comparison results to the target DQMGUI and generate an HTML report of the changes found, and links to the DQMGUI to show them.

The main purpose is to make it as user-friendly as possible, making minimal changes to compareDQMOutput.py, while still retaining high configurability.

The script allows:

• Selecting a different CMSSW version for the base and for the comp release.
• Applying different PRs on the base or comp CMSSW.
• Specifying Global Tags to be used for each release.
• Optionally re-using existing CMSSW releases.
• Specifying which DQMGUI to upload the comparison results to (hence requires [DQM] Update compareDQMOutput.py cms-sw/cmssw#46854 to be applied to the base CMSSW release, if it's < 15_0_0).

Logs generated:

• Logs are stored for each client executed, under the logs dir of the CMSSW release they belong to.
• Logs of the comparison script are stored in the output_dir subdirectory.

Requirements

• A machine with cvmfs mounted, with access to CMSSW.
• Streamer files for a run to be used as input for the DQM clients.
• fff_dqmtools installed.

Example usage

python3 cmssw_release_comparison.py CMSSW_14_1_5_patch1 CMSSW_14_1_5_patch1 \
--pull-requests-base +46854 --pull-requests-comp +46622 \
--gt-express-base 141X_dataRun3_Express_v3 --gt-hlt-base 141X_dataRun3_HLT_v1 \
--gt-express-comp 141X_dataRun3_Express_v3 --gt-hlt-comp 141X_dataRun3_HLT_v1 \
--input-dir /eos/home-d/dpapagia/data/run384377 \
--run-key pp_run \
--output-dir /tmp/comparison_results \
--dqmgui-url http://localhost:8060/dqm/dev

In this example, the resulting comparison files are uploaded to a local DQMGUI (http://localhost:8060/dqm/dev), and the resulting HTML report will be found in /tmp/comparison_results/<datetime>_<base release>_vs_<comp release>/dqm-histo-comparison-summary.html.

Notes and caveats

Warning

Some clients' results are non-deterministic, and multiple runs of the same clients on the same CMSSW releases produce slightly different results.

Most of them seem to be harmless, e.g. differences between strings in the EventInfo directory.

Other examples are under the HcalCalib/LEDTask directory (TimingMean, TimingRMS) in the root file created by the Hcal client. Those look harmless too, as they probably have to do with timing the performance of the client which might vary per execution, depedning on the load of the machine.

Note

Several DQM clients seem to have special requirements to run and are not currently taken into account:

• beam_dqm_sourceclient-live_cfg.py: Requires a db.key
• beamhlt_dqm_sourceclient-live_cfg.py: Same as above.
• beamspotdip_dqm_sourceclient-live_cfg.py: Must be run on P5 to access a DIP server
• visualization-live_cfg.py: It doesn't seem to produce any ROOT file we can use for comparison.
• visualization-live-secondInstance_cfg.py: Same as above

Note

cms-sw/cmssw#46854 needs to be applied to the base release, if a CMSSW version < 15_0_0 is used

Dummy workflows

compareDQMOutput.py was made to compare .root files generated from different "workflows". For this reason, it expects each .root file to be under a directory which is named after the workflow (in the format <int>.<int>_<description>).

To make the DQM clients' output compatible to compareDQMOutput.py, a "random" workflow number is given to each generated .root file, by placing each .root file in a dedicated directory, named after the dummy workflow id. This has the downside that it's not immediately apparent which "workflow" corresponds to which client. Some effort is put into replacing the workflow ids in the generated HTML, post-execution.

For reference, the resulting HTML will look something like this:

Tested comparisons

All the comparisons were run from within the dqm-dev-vm VM.

1. CMSSW_14_1_5_patch1 + 46854(required) vs CMSSW_14_1_5_patch1 + 46854: I.e., the exact same release against itself. No changes were expected, but still one can find at least 70 differences are found. The number of differences varies from execution to execution, ranging from 70 to 100. Almost all come from the EventInfo directory, so at least this many differences are always expected to be present. The command was:

python3 ~/workspace/fff_dqmtools/utils/cmssw_release_comparison.py CMSSW_14_1_5_patch1 CMSSW_14_1_5_patch1 -v -w /tmp --pull-requests-comp +46854 --pull-requests-base +46854 --gt-express-base 141X_dataRun3_Express_v3 --gt-hlt-base 141X_dataRun3_HLT_v1 --gt-express-comp 141X_dataRun3_Express_v3 --gt-hlt-comp 141X_dataRun3_HLT_v1 -c  ~/workspace/fff_dqmtools/utils/cmssw_deploy.py -i /eos/home-d/dpapagia/data/run384377 --reuse-releases -k pp_run --output-dir /tmp/comparison_results --dqmgui-url http://localhost:8060/dqm/dev

5. CMSSW_14_2_0 + 46854(required) vs CMSSW_14_2_0 + 41932. No GTs applied, as there are none for 14_2. Command:

    python3 ~/workspace/fff_dqmtools/utils/cmssw_release_comparison.py CMSSW_14_2_0 CMSSW_14_2_0 -v -w /tmp --pull-requests-comp +41932 --pull-requests-base +46854 -c  ~/workspace/fff_dqmtools/utils/cmssw_deploy.py -i /eos/home-d/dpapagia/data/run384377 --reuse-releases -k pp_run --output-dir /tmp/comparison_results --dqmgui-url http://localhost:8060/dqm/dev