Running the framework

Running locally

XAMPPmonoH.runAthena.runAthena()

Run Athena with options defined via the command line arguments. This runs the XAMPPmonoH analysis code locally and processes DxAODs to generate XAMPP ntuples.

This script will launch a local run of the XAMPPmonoH code. Usually this is only done for testing during development or to check if the settings are correct before launching a production on the grid.

The three most useful options here are

• --testJob: runs a test job (only PRW info for sample is loaded) without systematics
• --testJobWithSyst: runs a test job (only PRW info for sample is loaded) with systematics
• --nevents: specifies number of events to be processed

This script starts the analysis code. For more help type “python XAMPPbase/python/runAthena.py -h”

usage: runAthena [-h] [--evtMax EVTMAX] [--skipEvents SKIPEVENTS]
                 [--filesInput FILESINPUT] [--outFile OUTFILE]
                 [--analysis ANALYSIS] [--noSyst] [--parseFilesForPRW]
                 [--STConfig STCONFIG] [--testJob]
                 [--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
                 [--jobOptions JOBOPTIONS] [--valgrind {,memcheck,callgrind}]

Named Arguments

--evtMax	Max number of events to process Default: 10000
--skipEvents	Number of events to skip Default: -1
--filesInput, --inFile
	Number of events to skip Default: “”
--outFile, -o	name of the output file Default: “AnalysisOutput.root”
--analysis, -a	select the analysis you want to run on Default: “MyCutFlow”
--noSyst	run without systematic uncertainties Default: False
--parseFilesForPRW
	flag to tune the PRW based on the MC files we encounter - only for local jobs Default: False
--STConfig	name of custom SUSYTools config file located in data folder Default: “SUSYTools/SUSYTools_Default.conf”
--testJob	If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file Default: False
--dsidBugFix	Possible choices: , BFilter, CFilterBVeto, CVetoBVeto Specify a string for the dsid bug fix in case someone messes it up again with the dsid
--jobOptions	The athena jobOptions file to be executed Default: “XAMPPmonoH/runMonoH.py”
--valgrind	Possible choices: , memcheck, callgrind Search for memory leaks/call structure using valgrind Default: “”

Running on the Grid

XAMPPmonoH.SubmitToGrid.SubmitToGrid()

Submit jobs to grid using XAMPPmonoH framework.

This script will launch a grid production. Usually this is done to produce a new set of ntuples for all samples listed on the sample lists in https://gitlab.cern.ch/atlas-mpp-xampp/XAMPPmonoH/tree/master/XAMPPmonoH/data/SampleLists

Before launching the script make sure that you have set up panda and rucio and have obtained a valid grid proxy:

setupATLAS
lsetup rucio
lsetup panda
voms-proxy-init -voms atlas

Please have a look in the section about ntuple production: Checklist for Launching a Ntuple Production

This script submits the analysis code to the grid. For more help type “python XAMPPmonoH/python/SubmitToGrid.py -h”

usage: SubmitToGrid [-h] [--noSyst] --list LIST [--jobOptions JOBOPTIONS]
                    --production PRODUCTION [--filesPerJob FILESPERJOB]
                    [--productionRole]

Named Arguments

--noSyst	No systematics will be submitted Default: False
--list	List of datasets given
--jobOptions	The athena jobOptions file to be executed Default: “XAMPPmonoH/share/runMonoH.py”
--production	Name of the production, must be less than 15 characters
--filesPerJob	Files per job to be submitted (warning: small value might drive job into exhausted state) Default: -1
--productionRole
	Use exotics production role for submitting jobs Default: False

Running on a batch system

XAMPPbase.SubmitToBatch.main()

This script will launch a production on a local batch system. This only makes sense if you want to process TRUTH DxAODs or you have the DxAODs required for production on some local group disk to which your batch jobs have access.

In order to be able to process the files it is expected that you have a folder:

XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK

or similar avaliable, where MPPMU_LOCALGROUPDISK is the name of your RSE disk. This folder should contain a text file for each sample that list the paths to all files that are associated to this sample. You will have to specify the project (in this case always XAMPPmonoH using the argument --BaseProject) to ensure that the right directory is scanned for the group disk lists.

Example:

XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0.txt

contains all DxAOD ROOT files associated to the sample 600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0.

To run the batch jobs you will in addition have to specify the batch system using the argument --engine.

Example command to process all sample lists whose DSID starts with 600:

python XAMPPbase/XAMPPbase/python/SubmitToBatch.py \
--engine SLURM --jobName monoSbb_grid --BaseProject XAMPPmonoH \
--jobOptions XAMPPmonoH/runTruth.py  \
--inputDS `ls XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/700* | xargs -n 1 basename | sed -e 's/\.txt*$//'`

This script provides the basic arguments to setup the cluster engine

usage: SubmitToBatch [-h] --jobName JOBNAME
                     [--HoldBuildJob HOLDBUILDJOB [HOLDBUILDJOB ...]]
                     [--BaseFolder BASEFOLDER] [--BuildTime BUILDTIME]
                     [--MergeTime MERGETIME] [--jobArraySize JOBARRAYSIZE]
                     [--maxCurrentJobs MAXCURRENTJOBS]
                     [--Build_vmem BUILD_VMEM] [--Merge_vmem MERGE_VMEM]
                     [--nBuild_Cores NBUILD_CORES] [--mailTo MAILTO]
                     [--accountinggroup ACCOUNTINGGROUP] --engine
                     {SLURM,HTCONDOR,LOCAL,SGE} [--noBuildJob]
                     [--SingularityImage {x86_64-slc5-base.img,x86_64-slc5.img,x86_64-centos6-base.img,x86_64-centos7-base.img,x86_64-slc6-base.img,x86_64-slc6.img,x86_64-centos6.img,x86_64-centos7.img}]
                     [--BaseProject BASEPROJECT]
                     [--inputDS INPUTDS [INPUTDS ...]] [--RSE RSE]
                     [--RunTime RUNTIME] [--EventsPerJob EVENTSPERJOB]
                     [--FilesPerMergeJob FILESPERMERGEJOB]
                     [--FinalSplit FINALSPLIT] [--vmem VMEM]
                     [--HoldJob HOLDJOB [HOLDJOB ...]]
                     [--SpareWhatsProcessedIn SPAREWHATSPROCESSEDIN [SPAREWHATSPROCESSEDIN ...]]
                     [--outFile OUTFILE] [--analysis ANALYSIS] [--noSyst]
                     [--parseFilesForPRW] [--STConfig STCONFIG] [--testJob]
                     [--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
                     [--jobOptions JOBOPTIONS]
                     [--valgrind {,memcheck,callgrind}]

Named Arguments

--jobName, -J	Specify the JobName
--HoldBuildJob	Specifiy job names which should be finished before the build job starts Default: []
--BaseFolder	Changes the BaseFolder where the OutputFiles are saved. Default: “/ptmp/mpp/none/Cluster/”
--BuildTime	Changes the RunTime of the BuildJob Default: “01:59:59”
--MergeTime	Changes the RunTime of the merge Jobs Default: “01:59:59”
--jobArraySize	The maximum size of the slurm job-array Default: 2500
--maxCurrentJobs
	The maximum size of the slurm job-array Default: 400
--Build_vmem	Changes the virtual memory needed by the build job Default: 8000
--Merge_vmem	Changes the virtual memory needed by the merge job Default: 600
--nBuild_Cores	How many cores shall be used for the build job Default: 2
--mailTo	Specify a notification E-mail address Default: “none@rzg.mpg.de”
--accountinggroup
	If there is a special accounting group / project specify it here Default: “”
--engine	Possible choices: SLURM, HTCONDOR, LOCAL, SGE What is the grid engine to use
--noBuildJob	Do not submit the build job Default: True
--SingularityImage
	Possible choices: x86_64-slc5-base.img, x86_64-slc5.img, x86_64-centos6-base.img, x86_64-centos7-base.img, x86_64-slc6-base.img, x86_64-slc6.img, x86_64-centos6.img, x86_64-centos7.img The migration to centos 7 rises the need of using singularity Default: “x86_64-centos6.img”
--BaseProject	choose project containing file lists to adjust the filepath to the DS list (default is “XAMPPbase”) Default: “XAMPPbase”
--inputDS	Input datasets Default: []
--RSE	RSE storage element for files located via dcache. Default: “MPPMU_LOCALGROUPDISK”
--RunTime	Changes the RunTime of the Jobs: default 19:59:59 Default: “19:59:59”
--EventsPerJob	Changes the Events per Batch job. Default: 10000 Default: 10000
--FilesPerMergeJob
	Number of files per merge Default: 8
--FinalSplit	How many files should be left after merge Default: 1
--vmem	Virtual memory reserved for each analysis jobs Default: 3500
--HoldJob	Specfiy job names which should be finished before your job is starting. Default: []
--SpareWhatsProcessedIn
	If the cluster decided to die during production, you can skip the processed files in the directories Default: []
--outFile, -o	name of the output file Default: “AnalysisOutput.root”
--analysis, -a	select the analysis you want to run on Default: “MyCutFlow”
--noSyst	run without systematic uncertainties Default: False
--parseFilesForPRW
	flag to tune the PRW based on the MC files we encounter - only for local jobs Default: False
--STConfig	name of custom SUSYTools config file located in data folder Default: “SUSYTools/SUSYTools_Default.conf”
--testJob	If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file Default: False
--dsidBugFix	Possible choices: , BFilter, CFilterBVeto, CVetoBVeto Specify a string for the dsid bug fix in case someone messes it up again with the dsid
--jobOptions	The athena jobOptions file to be executed Default: “XAMPPbase/runXAMPPbase.py”
--valgrind	Possible choices: , memcheck, callgrind Search for memory leaks/call structure using valgrind Default: “”