Running the framework

Running locally

XAMPPmonoH.runAthena.runAthena()

Run Athena with options defined via the command line arguments. This runs the XAMPPmonoH analysis code locally and processes DxAODs to generate XAMPP ntuples.

This script will launch a local run of the XAMPPmonoH code. Usually this is only done for testing during development or to check if the settings are correct before launching a production on the grid.

The three most useful options here are

  • --testJob: runs a test job (only PRW info for sample is loaded) without systematics
  • --testJobWithSyst: runs a test job (only PRW info for sample is loaded) with systematics
  • --nevents: specifies number of events to be processed

This script starts the analysis code. For more help type “python XAMPPbase/python/runAthena.py -h”

usage: runAthena [-h] [--evtMax EVTMAX] [--skipEvents SKIPEVENTS]
                 [--filesInput FILESINPUT] [--outFile OUTFILE]
                 [--analysis ANALYSIS] [--noSyst] [--parseFilesForPRW]
                 [--STConfig STCONFIG] [--testJob]
                 [--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
                 [--jobOptions JOBOPTIONS] [--valgrind {,memcheck,callgrind}]

Named Arguments

--evtMax

Max number of events to process

Default: 10000

--skipEvents

Number of events to skip

Default: -1

--filesInput, --inFile
 

Number of events to skip

Default: “”

--outFile, -o

name of the output file

Default: “AnalysisOutput.root”

--analysis, -a

select the analysis you want to run on

Default: “MyCutFlow”

--noSyst

run without systematic uncertainties

Default: False

--parseFilesForPRW
 

flag to tune the PRW based on the MC files we encounter - only for local jobs

Default: False

--STConfig

name of custom SUSYTools config file located in data folder

Default: “SUSYTools/SUSYTools_Default.conf”

--testJob

If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file

Default: False

--dsidBugFix

Possible choices: , BFilter, CFilterBVeto, CVetoBVeto

Specify a string for the dsid bug fix in case someone messes it up again with the dsid

--jobOptions

The athena jobOptions file to be executed

Default: “XAMPPmonoH/runMonoH.py”

--valgrind

Possible choices: , memcheck, callgrind

Search for memory leaks/call structure using valgrind

Default: “”

Running on the Grid

XAMPPmonoH.SubmitToGrid.SubmitToGrid()

Submit jobs to grid using XAMPPmonoH framework.

This script will launch a grid production. Usually this is done to produce a new set of ntuples for all samples listed on the sample lists in https://gitlab.cern.ch/atlas-mpp-xampp/XAMPPmonoH/tree/master/XAMPPmonoH/data/SampleLists

Before launching the script make sure that you have set up panda and rucio and have obtained a valid grid proxy:

setupATLAS
lsetup rucio
lsetup panda
voms-proxy-init -voms atlas

Please have a look in the section about ntuple production: Checklist for Launching a Ntuple Production

This script submits the analysis code to the grid. For more help type “python XAMPPmonoH/python/SubmitToGrid.py -h”

usage: SubmitToGrid [-h] [--noSyst] --list LIST [--jobOptions JOBOPTIONS]
                    --production PRODUCTION [--filesPerJob FILESPERJOB]
                    [--productionRole]

Named Arguments

--noSyst

No systematics will be submitted

Default: False

--list List of datasets given
--jobOptions

The athena jobOptions file to be executed

Default: “XAMPPmonoH/share/runMonoH.py”

--production Name of the production, must be less than 15 characters
--filesPerJob

Files per job to be submitted (warning: small value might drive job into exhausted state)

Default: -1

--productionRole
 

Use exotics production role for submitting jobs

Default: False

Running on a batch system

XAMPPbase.SubmitToBatch.main()

This script will launch a production on a local batch system. This only makes sense if you want to process TRUTH DxAODs or you have the DxAODs required for production on some local group disk to which your batch jobs have access.

In order to be able to process the files it is expected that you have a folder:

XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK

or similar avaliable, where MPPMU_LOCALGROUPDISK is the name of your RSE disk. This folder should contain a text file for each sample that list the paths to all files that are associated to this sample. You will have to specify the project (in this case always XAMPPmonoH using the argument --BaseProject) to ensure that the right directory is scanned for the group disk lists.

Example:

XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0.txt

contains all DxAOD ROOT files associated to the sample 600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0.

To run the batch jobs you will in addition have to specify the batch system using the argument --engine.

Example command to process all sample lists whose DSID starts with 600:

python XAMPPbase/XAMPPbase/python/SubmitToBatch.py \
--engine SLURM --jobName monoSbb_grid --BaseProject XAMPPmonoH \
--jobOptions XAMPPmonoH/runTruth.py  \
--inputDS `ls XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/700* | xargs -n 1 basename | sed -e 's/\.txt*$//'`

This script provides the basic arguments to setup the cluster engine

usage: SubmitToBatch [-h] --jobName JOBNAME
                     [--HoldBuildJob HOLDBUILDJOB [HOLDBUILDJOB ...]]
                     [--BaseFolder BASEFOLDER] [--BuildTime BUILDTIME]
                     [--MergeTime MERGETIME] [--jobArraySize JOBARRAYSIZE]
                     [--maxCurrentJobs MAXCURRENTJOBS]
                     [--Build_vmem BUILD_VMEM] [--Merge_vmem MERGE_VMEM]
                     [--nBuild_Cores NBUILD_CORES] [--mailTo MAILTO]
                     [--accountinggroup ACCOUNTINGGROUP] --engine
                     {SLURM,HTCONDOR,LOCAL,SGE} [--noBuildJob]
                     [--SingularityImage {x86_64-slc5-base.img,x86_64-slc5.img,x86_64-centos6-base.img,x86_64-centos7-base.img,x86_64-slc6-base.img,x86_64-slc6.img,x86_64-centos6.img,x86_64-centos7.img}]
                     [--BaseProject BASEPROJECT]
                     [--inputDS INPUTDS [INPUTDS ...]] [--RSE RSE]
                     [--RunTime RUNTIME] [--EventsPerJob EVENTSPERJOB]
                     [--FilesPerMergeJob FILESPERMERGEJOB]
                     [--FinalSplit FINALSPLIT] [--vmem VMEM]
                     [--HoldJob HOLDJOB [HOLDJOB ...]]
                     [--SpareWhatsProcessedIn SPAREWHATSPROCESSEDIN [SPAREWHATSPROCESSEDIN ...]]
                     [--outFile OUTFILE] [--analysis ANALYSIS] [--noSyst]
                     [--parseFilesForPRW] [--STConfig STCONFIG] [--testJob]
                     [--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
                     [--jobOptions JOBOPTIONS]
                     [--valgrind {,memcheck,callgrind}]

Named Arguments

--jobName, -J Specify the JobName
--HoldBuildJob

Specifiy job names which should be finished before the build job starts

Default: []

--BaseFolder

Changes the BaseFolder where the OutputFiles are saved.

Default: “/ptmp/mpp/none/Cluster/”

--BuildTime

Changes the RunTime of the BuildJob

Default: “01:59:59”

--MergeTime

Changes the RunTime of the merge Jobs

Default: “01:59:59”

--jobArraySize

The maximum size of the slurm job-array

Default: 2500

--maxCurrentJobs
 

The maximum size of the slurm job-array

Default: 400

--Build_vmem

Changes the virtual memory needed by the build job

Default: 8000

--Merge_vmem

Changes the virtual memory needed by the merge job

Default: 600

--nBuild_Cores

How many cores shall be used for the build job

Default: 2

--mailTo

Specify a notification E-mail address

Default: “none@rzg.mpg.de

--accountinggroup
 

If there is a special accounting group / project specify it here

Default: “”

--engine

Possible choices: SLURM, HTCONDOR, LOCAL, SGE

What is the grid engine to use

--noBuildJob

Do not submit the build job

Default: True

--SingularityImage
 

Possible choices: x86_64-slc5-base.img, x86_64-slc5.img, x86_64-centos6-base.img, x86_64-centos7-base.img, x86_64-slc6-base.img, x86_64-slc6.img, x86_64-centos6.img, x86_64-centos7.img

The migration to centos 7 rises the need of using singularity

Default: “x86_64-centos6.img”

--BaseProject

choose project containing file lists to adjust the filepath to the DS list (default is “XAMPPbase”)

Default: “XAMPPbase”

--inputDS

Input datasets

Default: []

--RSE

RSE storage element for files located via dcache.

Default: “MPPMU_LOCALGROUPDISK”

--RunTime

Changes the RunTime of the Jobs: default 19:59:59

Default: “19:59:59”

--EventsPerJob

Changes the Events per Batch job. Default: 10000

Default: 10000

--FilesPerMergeJob
 

Number of files per merge

Default: 8

--FinalSplit

How many files should be left after merge

Default: 1

--vmem

Virtual memory reserved for each analysis jobs

Default: 3500

--HoldJob

Specfiy job names which should be finished before your job is starting.

Default: []

--SpareWhatsProcessedIn
 

If the cluster decided to die during production, you can skip the processed files in the directories

Default: []

--outFile, -o

name of the output file

Default: “AnalysisOutput.root”

--analysis, -a

select the analysis you want to run on

Default: “MyCutFlow”

--noSyst

run without systematic uncertainties

Default: False

--parseFilesForPRW
 

flag to tune the PRW based on the MC files we encounter - only for local jobs

Default: False

--STConfig

name of custom SUSYTools config file located in data folder

Default: “SUSYTools/SUSYTools_Default.conf”

--testJob

If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file

Default: False

--dsidBugFix

Possible choices: , BFilter, CFilterBVeto, CVetoBVeto

Specify a string for the dsid bug fix in case someone messes it up again with the dsid

--jobOptions

The athena jobOptions file to be executed

Default: “XAMPPbase/runXAMPPbase.py”

--valgrind

Possible choices: , memcheck, callgrind

Search for memory leaks/call structure using valgrind

Default: “”