Running the framework¶
Running locally¶
-
XAMPPmonoH.runAthena.
runAthena
()¶ Run Athena with options defined via the command line arguments. This runs the XAMPPmonoH analysis code locally and processes DxAODs to generate XAMPP ntuples.
This script will launch a local run of the XAMPPmonoH code. Usually this is only done for testing during development or to check if the settings are correct before launching a production on the grid.
The three most useful options here are
--testJob
: runs a test job (only PRW info for sample is loaded) without systematics--testJobWithSyst
: runs a test job (only PRW info for sample is loaded) with systematics--nevents
: specifies number of events to be processed
This script starts the analysis code. For more help type “python XAMPPbase/python/runAthena.py -h”
usage: runAthena [-h] [--evtMax EVTMAX] [--skipEvents SKIPEVENTS]
[--filesInput FILESINPUT] [--outFile OUTFILE]
[--analysis ANALYSIS] [--noSyst] [--parseFilesForPRW]
[--STConfig STCONFIG] [--testJob]
[--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
[--jobOptions JOBOPTIONS] [--valgrind {,memcheck,callgrind}]
Named Arguments¶
--evtMax | Max number of events to process Default: 10000 |
--skipEvents | Number of events to skip Default: -1 |
--filesInput, --inFile | |
Number of events to skip Default: “” | |
--outFile, -o | name of the output file Default: “AnalysisOutput.root” |
--analysis, -a | select the analysis you want to run on Default: “MyCutFlow” |
--noSyst | run without systematic uncertainties Default: False |
--parseFilesForPRW | |
flag to tune the PRW based on the MC files we encounter - only for local jobs Default: False | |
--STConfig | name of custom SUSYTools config file located in data folder Default: “SUSYTools/SUSYTools_Default.conf” |
--testJob | If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file Default: False |
--dsidBugFix | Possible choices: , BFilter, CFilterBVeto, CVetoBVeto Specify a string for the dsid bug fix in case someone messes it up again with the dsid |
--jobOptions | The athena jobOptions file to be executed Default: “XAMPPmonoH/runMonoH.py” |
--valgrind | Possible choices: , memcheck, callgrind Search for memory leaks/call structure using valgrind Default: “” |
Running on the Grid¶
-
XAMPPmonoH.SubmitToGrid.
SubmitToGrid
()¶ Submit jobs to grid using XAMPPmonoH framework.
This script will launch a grid production. Usually this is done to produce a new set of ntuples for all samples listed on the sample lists in https://gitlab.cern.ch/atlas-mpp-xampp/XAMPPmonoH/tree/master/XAMPPmonoH/data/SampleLists
Before launching the script make sure that you have set up panda and rucio and have obtained a valid grid proxy:
setupATLAS
lsetup rucio
lsetup panda
voms-proxy-init -voms atlas
Please have a look in the section about ntuple production: Checklist for Launching a Ntuple Production
This script submits the analysis code to the grid. For more help type “python XAMPPmonoH/python/SubmitToGrid.py -h”
usage: SubmitToGrid [-h] [--noSyst] --list LIST [--jobOptions JOBOPTIONS]
--production PRODUCTION [--filesPerJob FILESPERJOB]
[--productionRole]
Named Arguments¶
--noSyst | No systematics will be submitted Default: False |
--list | List of datasets given |
--jobOptions | The athena jobOptions file to be executed Default: “XAMPPmonoH/share/runMonoH.py” |
--production | Name of the production, must be less than 15 characters |
--filesPerJob | Files per job to be submitted (warning: small value might drive job into exhausted state) Default: -1 |
--productionRole | |
Use exotics production role for submitting jobs Default: False |
Running on a batch system¶
-
XAMPPbase.SubmitToBatch.
main
()¶
This script will launch a production on a local batch system. This only makes sense if you want to process TRUTH DxAODs or you have the DxAODs required for production on some local group disk to which your batch jobs have access.
In order to be able to process the files it is expected that you have a folder:
XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK
or similar avaliable, where MPPMU_LOCALGROUPDISK is the name of your RSE disk. This folder should contain a text file for each sample that list the paths to all files that are associated to this sample. You will have to specify the project (in this case always XAMPPmonoH using the argument --BaseProject
) to ensure that the right directory is scanned for the group disk lists.
Example:
XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0.txt
contains all DxAOD ROOT files associated to the sample 600041.MadGraphPy8EG_A14NNP23LO_monoSbb_zp500_dm100_dh70q0p01x1p0
.
To run the batch jobs you will in addition have to specify the batch system using the argument --engine
.
Example command to process all sample lists whose DSID starts with 600:
python XAMPPbase/XAMPPbase/python/SubmitToBatch.py \
--engine SLURM --jobName monoSbb_grid --BaseProject XAMPPmonoH \
--jobOptions XAMPPmonoH/runTruth.py \
--inputDS `ls XAMPPmonoH/data/GroupDiskLists/MPPMU_LOCALGROUPDISK/700* | xargs -n 1 basename | sed -e 's/\.txt*$//'`
This script provides the basic arguments to setup the cluster engine
usage: SubmitToBatch [-h] --jobName JOBNAME
[--HoldBuildJob HOLDBUILDJOB [HOLDBUILDJOB ...]]
[--BaseFolder BASEFOLDER] [--BuildTime BUILDTIME]
[--MergeTime MERGETIME] [--jobArraySize JOBARRAYSIZE]
[--maxCurrentJobs MAXCURRENTJOBS]
[--Build_vmem BUILD_VMEM] [--Merge_vmem MERGE_VMEM]
[--nBuild_Cores NBUILD_CORES] [--mailTo MAILTO]
[--accountinggroup ACCOUNTINGGROUP] --engine
{SLURM,HTCONDOR,LOCAL,SGE} [--noBuildJob]
[--SingularityImage {x86_64-slc5-base.img,x86_64-slc5.img,x86_64-centos6-base.img,x86_64-centos7-base.img,x86_64-slc6-base.img,x86_64-slc6.img,x86_64-centos6.img,x86_64-centos7.img}]
[--BaseProject BASEPROJECT]
[--inputDS INPUTDS [INPUTDS ...]] [--RSE RSE]
[--RunTime RUNTIME] [--EventsPerJob EVENTSPERJOB]
[--FilesPerMergeJob FILESPERMERGEJOB]
[--FinalSplit FINALSPLIT] [--vmem VMEM]
[--HoldJob HOLDJOB [HOLDJOB ...]]
[--SpareWhatsProcessedIn SPAREWHATSPROCESSEDIN [SPAREWHATSPROCESSEDIN ...]]
[--outFile OUTFILE] [--analysis ANALYSIS] [--noSyst]
[--parseFilesForPRW] [--STConfig STCONFIG] [--testJob]
[--dsidBugFix {,BFilter,CFilterBVeto,CVetoBVeto}]
[--jobOptions JOBOPTIONS]
[--valgrind {,memcheck,callgrind}]
Named Arguments¶
--jobName, -J | Specify the JobName |
--HoldBuildJob | Specifiy job names which should be finished before the build job starts Default: [] |
--BaseFolder | Changes the BaseFolder where the OutputFiles are saved. Default: “/ptmp/mpp/none/Cluster/” |
--BuildTime | Changes the RunTime of the BuildJob Default: “01:59:59” |
--MergeTime | Changes the RunTime of the merge Jobs Default: “01:59:59” |
--jobArraySize | The maximum size of the slurm job-array Default: 2500 |
--maxCurrentJobs | |
The maximum size of the slurm job-array Default: 400 | |
--Build_vmem | Changes the virtual memory needed by the build job Default: 8000 |
--Merge_vmem | Changes the virtual memory needed by the merge job Default: 600 |
--nBuild_Cores | How many cores shall be used for the build job Default: 2 |
--mailTo | Specify a notification E-mail address Default: “none@rzg.mpg.de” |
--accountinggroup | |
If there is a special accounting group / project specify it here Default: “” | |
--engine | Possible choices: SLURM, HTCONDOR, LOCAL, SGE What is the grid engine to use |
--noBuildJob | Do not submit the build job Default: True |
--SingularityImage | |
Possible choices: x86_64-slc5-base.img, x86_64-slc5.img, x86_64-centos6-base.img, x86_64-centos7-base.img, x86_64-slc6-base.img, x86_64-slc6.img, x86_64-centos6.img, x86_64-centos7.img The migration to centos 7 rises the need of using singularity Default: “x86_64-centos6.img” | |
--BaseProject | choose project containing file lists to adjust the filepath to the DS list (default is “XAMPPbase”) Default: “XAMPPbase” |
--inputDS | Input datasets Default: [] |
--RSE | RSE storage element for files located via dcache. Default: “MPPMU_LOCALGROUPDISK” |
--RunTime | Changes the RunTime of the Jobs: default 19:59:59 Default: “19:59:59” |
--EventsPerJob | Changes the Events per Batch job. Default: 10000 Default: 10000 |
--FilesPerMergeJob | |
Number of files per merge Default: 8 | |
--FinalSplit | How many files should be left after merge Default: 1 |
--vmem | Virtual memory reserved for each analysis jobs Default: 3500 |
--HoldJob | Specfiy job names which should be finished before your job is starting. Default: [] |
--SpareWhatsProcessedIn | |
If the cluster decided to die during production, you can skip the processed files in the directories Default: [] | |
--outFile, -o | name of the output file Default: “AnalysisOutput.root” |
--analysis, -a | select the analysis you want to run on Default: “MyCutFlow” |
--noSyst | run without systematic uncertainties Default: False |
--parseFilesForPRW | |
flag to tune the PRW based on the MC files we encounter - only for local jobs Default: False | |
--STConfig | name of custom SUSYTools config file located in data folder Default: “SUSYTools/SUSYTools_Default.conf” |
--testJob | If the testjob argument is called then only the mc16a/mc16d prw files are loaded depeending on the input file Default: False |
--dsidBugFix | Possible choices: , BFilter, CFilterBVeto, CVetoBVeto Specify a string for the dsid bug fix in case someone messes it up again with the dsid |
--jobOptions | The athena jobOptions file to be executed Default: “XAMPPbase/runXAMPPbase.py” |
--valgrind | Possible choices: , memcheck, callgrind Search for memory leaks/call structure using valgrind Default: “” |