Simultaneous setup of MPI and SUSY group software

Current SUSYTools version: SUSYTools-00-00-63

This page details the steps currently needed to set up the MPI software for SUSY analyses, and run the default 4 lepton analysis. The key problems are that the MPI software runs in athena, while the SUSY group software (SUSYTools and the packages it depends on) are optimised for RootCore use, and often have limited athena compatibility.

For the moment, therefore, there is a lot of code editing etc needed in order to get everything working. The plan is to contact the developers of the various packages and try to persuade them to include the changes in future SVN versions, to make things easier in the future.

Max has pointed out that there is a slightly easier way of setting some things up by setting ROOTCOREDIR manually, even when running in athena. This makes some of the code changes unnecessary, and forms a valid alternative to some of the instructions on this page.

General note

Most lines can be copied and pasted, either onto the command line or into a file. There is one major exception: whenever you see svn+ssh://svn.cern.ch, this needs to be changed to svn+ssh://USERNAME@svn.cern.ch if your CERN user name is different to your username on the local system.

If there are questions, or you have trouble locating any of the files mentioned, please contact MichaelFlowerdew.

ChangeLog

11th June: Update to recommendations for DGTriggerReweight, removing the unnecessary linking in the InstallArea.

Initial setup

Every time you log in, set up athena using your favourite method. For example, using Mike's script:

source athenasetup.sh -t 17.0.4.8,AtlasProduction -a MPIsetup/

A lot of packages need to be checked out. To make this easier you can create an ssh key to give you password-less access to SVN. This procedure is stolen from here.

First, execute these lines on your local system:

ssh-keygen -t dsa       # Don't provide a password
scp ~/.ssh/id_dsa.pub USERNAME@lxplus.cern.ch:~

Then, log in to lxplus and do

cat ~/id_dsa.pub >> ~/.ssh/authorized_keys
/afs/cern.ch/project/svn/dist/bin/set_ssh

Still on lxplus, create a file ~/.ssh/config, with the following contents:

Host lxplus.cern.ch lxplus 
Protocol 2 
PubkeyAuthentication no 
PasswordAuthentication yes

Host svn.cern.ch svn 
GSSAPIAuthentication yes 
GSSAPIDelegateCredentials yes 
Protocol 2 
ForwardX11 no

Package checkout

Execute all commands in this section from $TestArea.

Check out the MPI analysis (This should now no longer require a password)

cmt co -r VersatileUserAnalysis17-00-00-18 svn+ssh://svn.cern.ch/reps/atlasinst/Institutes/MPI/MDT/analysis/VersatileUserAnalysis17

Fix up usage of SVNGRP, before checking out the other MPI packages

export SVNGRP=svn+ssh://svn.cern.ch/reps/atlasinst
cmt co -requirements VersatileUserAnalysis17/cmt/requirements

Check out SUSYTools and the packages it depends on. The package versions should match those in SUSYTools/python/install.py, but some are skipped either because they are unnecessary or already in the athena release.

cmt co -r SUSYTools-00-00-63 PhysicsAnalysis/SUSYPhys/SUSYTools
cmt co -r PileupReweighting-00-02-02 PhysicsAnalysis/AnalysisCommon/PileupReweighting
cmt co -r JetSelectorTools-00-00-16 PhysicsAnalysis/JetMissingEtID/JetSelectorTools
cmt co -r GoodRunsLists-00-00-96 DataQuality/GoodRunsLists
cmt co -r MuonMomentumCorrections-00-05-03 PhysicsAnalysis/MuonID/MuonIDAnalysis/MuonMomentumCorrections
cmt co -r MuonEfficiencyCorrections-01-00-10 PhysicsAnalysis/MuonID/MuonIDAnalysis/MuonEfficiencyCorrections
cmt co -r egammaAnalysisUtils-00-02-76 Reconstruction/egamma/egammaAnalysis/egammaAnalysisUtils
cmt co -r egammaEvent-03-06-20 Reconstruction/egamma/egammaEvent
cmt co -r JetResolution-01-00-00 Reconstruction/Jet/JetResolution
cmt co -r JetUncertainties-00-05-01 Reconstruction/Jet/JetUncertainties
cmt co -r MissingETUtility-00-02-14 Reconstruction/MissingETUtility
cmt co -r ReweightUtils-00-02-06 PhysicsAnalysis/AnalysisCommon/ReweightUtils
cmt co -r CalibrationDataInterface-00-01-02 PhysicsAnalysis/JetTagging/JetTagPerformanceCalibration/CalibrationDataInterface
cmt co -r ApplyJetCalibration-00-01-01 svn+ssh://svn.cern.ch/reps/atlasperf/CombPerf/JetETMiss/JetCalibrationTools/ApplyJetCalibration
cmt co -r ObjectSelectorCore-00-00-17 PhysicsAnalysis/AnalysisCommon/ObjectSelectorCore
cmt co -r TrigRootAnalysis-00-00-07 Trigger/TrigAnalysis/TrigRootAnalysis
cmt co -r DGTriggerReweight-00-00-04 svn+ssh://svn.cern.ch/reps/atlasinst/Institutes/UCIrvine/btoggers/DGTriggerReweight

Finally, check out the analysis packages

cmt co svn+ssh://svn.cern.ch/reps/atlasusr/goblirsc/SUSYAnalysis
cmt co svn+ssh://svn.cern.ch/reps/atlasinst/Institutes/MPI/MDT/analysis/UserCode/goblirsc/MyAnalysisUtilities

Code fixing.

The above code will not compile (or run) out of the box. This section contains the (near-to-minimal) changes that need to be made. All paths are relative to $TestArea.

SUSYTools

PhysicsAnalysis/SUSYPhys/SUSYTools/Root/

Remove some SUSYTools source files that do not compile in athena:

for f in HforToolD3PD.cxx Reco2Lep.cxx Reco_dpd2.cxx TJVFReweighting.cxx TopTag.cxx 
do
mv ${f} ${f}_backup
done

PhysicsAnalysis/SUSYPhys/SUSYTools/Root/FudgeMCTool.cxx

Replace [ptbins] with [9999] throughout.

        double x[9999]; double xerr[9999];
        double y[9999]; double yerr[9999];

PhysicsAnalysis/SUSYPhys/SUSYTools/Root/SUSYObjDef.cxx

After this block

  if (tmparea != NULL) {
    maindir = tmparea;
    maindir = maindir + "/../";
  }

insert these lines:

  else { // Athena
    tmparea = getenv("TestArea");
    if (tmparea != NULL) {
      maindir = tmparea;
      maindir = maindir + "/";
    }
  }

Then, comment out these two lines (they're near, but not next to, each other):

if (muon_sf_dir == "") muon_sf_dir = maindir + "MuonEfficiencyCorrections/share/"; // Default path
if (muon_momentum_dir == "") muon_momentum_dir = maindir + "MuonMomentumCorrections/share/"; // Default path

Next, add the "PhysicsAnalysis/SUSYPhys/" bits to these lines:

  if (multijes_file == "") multijes_file = maindir + "PhysicsAnalysis/SUSYPhys/SUSYTools/share/MultijetJES.config";
  if (jes_file == "") jes_file = maindir + "PhysicsAnalysis/SUSYPhys/SUSYTools/share/InsituJES2011_AllNuisanceParameters.config";

Finally, add "Reconstruction/Jet/" to this line:

  if (jer_file == "") jer_file = maindir + "Reconstruction/Jet/JetResolution/share/JERProviderPlots.root";

PhysicsAnalysis/SUSYPhys/SUSYTools/share/

In InsituJES2011_AllNuisanceParameters.config, replace the line for the JESUncertaintyRootFile with this:

JESUncertaintyRootFile:      ${TestArea}/Reconstruction/Jet/JetUncertainties/share/JESUncertainty.root

Similarly, for MultijetJES.config:

MultijetJESUncertaintyRootFile:      ${TestArea}/Reconstruction/Jet/JetUncertainties/share/MJESUncertainty.root
AnalysisRootFile:                    ${TestArea}/Reconstruction/Jet/JetUncertainties/analysisPlots/UnknownFlavourComp.root

PhysicsAnalysis/SUSYPhys/SUSYTools/cmt/requirements

Remove the final parts of the paths from these two lines:

use JetResolution JetResolution-* Reconstruction/Jet/JetResolution
use MissingETUtility MissingETUtility-* Reconstruction/MissingETUtility

so that they look like this:

use JetResolution JetResolution-* Reconstruction/Jet
use MissingETUtility MissingETUtility-* Reconstruction

Uncomment the "use CalibrationDataInterface" line in cmt/requirements, and add these packages:

use egammaEvent egammaEvent-* Reconstruction/egamma
use ObjectSelectorCore ObjectSelectorCore-* PhysicsAnalysis/AnalysisCommon
use TrigRootAnalysis TrigRootAnalysis-* Trigger/TrigAnalysis
use DGTriggerReweight DGTriggerReweight-* 
use ApplyJetCalibration ApplyJetCalibration-* 

Important: the last two lines have a space at the end of the line!

A little further down, under public, add these lines:

library  SUSYTools ../src/*.cxx ../Root/*.cxx
apply_pattern installed_library

Optional: Reflex dictionary generation

If you want to use the SUSYTools classes in standalone ROOT/pyROOT, you can achieve this with a few more modifications. For further details, see this page. This example just adds access to one class (CrossSectionDB), it is trivial to add in more.

Create a file SUSYTools/SUSYToolsDict.h:

#ifndef SUSYTOOLS_SUSYTOOLSDICT_H
#define SUSYTOOLS_SUSYTOOLSDICT_H

#include "SUSYTools/SUSYCrossSection.h"

#endif //> not SUSYTOOLS_SUSYTOOLSDICT_H    

Then, create a file SUSYTools/selection.xml:

<lcgdict>
<class name="SUSY::CrossSectionDB" />
</lcgdict>

Then, add these lines to cmt/requirements, just before the RTT section:

### dict generation ###
private
use AtlasReflex  AtlasReflex-*  External -no_auto_imports

apply_pattern lcgdict dict=SUSYTools \
 selectionfile=selection.xml \
 headerfiles="../SUSYTools/SUSYToolsDict.h"

end_private

To use the dictionaries in pyROOT (after compilation, of course!), do the following:

>>> import PyCintex
>>> PyCintex.Cintex.Enable()
>>> import ROOT
>>> mydb = ROOT.SUSY.CrossSectionDB('../data/susy_crosssections.txt') # Of course, adjust this to the correct path

Note that the middle "SUSY" is the namespace of the class.

Packages that SUSYTools depends on

PhysicsAnalysis/JetTagging/JetTagPerformanceCalibration/CalibrationDataInterface/cmt/requirements

Remove the "../src/*.cxx" bit from this line:

library CalibrationDataInterfaceLib "../Root/*.cxx ../src/*.cxx"

to leave just this:

library CalibrationDataInterfaceLib "../Root/*.cxx"

ApplyJetCalibration/cmt/requirements

This file needs to be created, with this content:

use AtlasPolicy         AtlasPolicy-* 
use AtlasROOT           AtlasROOT-*           External
use PathResolver        PathResolver-*        Tools
use GaudiInterface      GaudiInterface-*      External

apply_tag ROOTMathLibs
apply_tag ROOTBasicLibs

library ApplyJetCalibration "../src/*.cxx ../Root/*.cxx"
apply_pattern installed_library

## This is so that the JERProviderPlots.root file gets copied to the share area
apply_pattern declare_runtime files="*.root"

ApplyJetCalibration/Root/ApplyJetCalibration.cxx

Find this block:

  if (RootCoreDir != "") {
    _basePath=RootCoreDir+"/data/ApplyJetCalibration/";
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
    _basePath=RootCoreDir+"../ApplyJetCalibration/data/";
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
  }

and add this immediately after

  else { // Fix for athena
    RootCoreDir = gSystem->Getenv("TestArea");
    _basePath=RootCoreDir+"/";
    printf("Looking for file in %s\n", (_basePath+fn).Data());
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
    _basePath=RootCoreDir+"/ApplyJetCalibration/data/";
    printf("Looking for file in %s\n", (_basePath+fn).Data());
    if (gSystem->AccessPathName(_basePath+fn)==false) return _basePath+fn;
  }

DGTriggerReweight/cmt/requirements

This file also needs to be created, with this as contents:

use AtlasPolicy         AtlasPolicy-* 
use AtlasROOT           AtlasROOT-*      External
use PathResolver        PathResolver-*      Tools
use GaudiInterface      GaudiInterface-*   External
use ReweightUtils   ReweightUtils-*      PhysicsAnalysis/AnalysisCommon

apply_tag ROOTMathLibs
apply_tag ROOTBasicLibs

library DGTriggerReweight "../Root/*.cxx"
apply_pattern installed_library

## This is so that the root files get copied to the share area
apply_pattern declare_runtime files="*.root"

DGTriggerReweight/DGTriggerReweight/triggerReweight.h

Comment out this line:

  ClassDef(triggerReweight, 2); 

Reconstruction/MissingETUtility/src/METUtilityAthTool.cxx

Comment out these two lines in initialize():

  m_multiJES->includeFlavorComposition(m_doFlavourJES);
  m_multiJES->init();

This fix will be unnecessary when we switch to MissingETUtility-01-00-05.

MPI core code

NtupleWriter17/src/METCompositionDataSelectionTool.cxx

In the clean() method, after "delete m_my_met_composition", the pointer should be set to NULL:

  if(m_my_met_composition)
    {
      delete m_my_met_composition;
      m_my_met_composition = NULL;
    }

SUSY analysis code

SUSYAnalysis/cmt/requirements

Change the MyAnalysisUtilities line to this:

use MyAnalysisUtilities MyAnalysisUtilities-* 

Note that this has a space at the end of the line.

Comment out all the "use" statements under "ATLAS packages" except SUSYTools, which should be changed to look like this:

use SUSYTools SUSYTools-* PhysicsAnalysis/SUSYPhys

Compilation

MPI core packages

cd NtupleReaderAlg/cmt
cmt bro cmt config
source setup.sh
cmt bro cmt make -j4

The rest of the code

Everything else can be compiled from SUSYAnalysis.

cd SUSYAnalysis/cmt/
cmt bro cmt config
cmt bro source setup.sh
cmt bro cmt make -j4

Running

cd $TestArea/SUSYAnalysis/run

I recommend creating a working directory within run/ and copying your job options there. For example:

mkdir SUSYtest
cd SUSYtest
cp ../SUSY_Analysis_full.py .

File lists

To run, you need to provide a list of files. If you are running at RZG, you can do the following:

cp /ptmp/mpp/flowerde/MPIsetup/FileLists.tar.gz .
tar -xzf FileLists.tar.gz 

This creates a directory called VUA18_MC11c, containing the text files you need

Additional input files

You will need a couple of files from the multilepton twiki, to provide additional scaling and correction factors (and these will occasionally need to be updated). Scroll right down to the bottom (where it says "Attachments") and download these files:

mc11cpileup2201.root
ilumicalc_histograms_None_178044-191933.root
muon_triggermaps.root
electron_maps.root

You will also need a "Good Runs List" (GRL) from this page. At the moment, we are using

data11_7TeV.periodAllYear_DetStatus-v36-pro10_CoolRunQuery-00-04-08_Susy.xml

Right-click to select "Save As" or similar, depending on your browser.

Job options editing

To run a test job, you need to edit the job options in a few places. First, replace

NtupleReaderAlg.fileList = "filelist.txt"

with

NtupleReaderAlg.fileList = "VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt"

Then, set the number of events to a small number for testing (a few lines further down)

NtupleReaderAlg.numberOfEvents = 20

In addition, near the bottom (where the analysis tool properties are set), add this line:

analysisTool.GRL = "data11_7TeV.periodAllYear_DetStatus-v36-pro10_CoolRunQuery-00-04-08_Susy.xml"

Run it!

At last we get to run a test job:

athena SUSY_Analysis_full.py > firstrun.out 2>&1

The "2>&1" makes sure that both stdout and stderr end up in firstrun.out, so you can examine it at your leisure. An example file (run over 2000 events) is provided in /ptmp/mpp/flowerde/MPIsetup/firstrun.out.

Batch farm running

For this you will need four more files, all under /ptmp/mpp/flowerde/MPIsetup/: Note these will be slightly different to those sent around by Max, to make it easier to work at RZG.

• submit_array_dcap2_v3.py, the script you call to submit the analysis.
• batch_jobdcap2_v3.exe, the script that's executed on each worker node. It makes (and eventually removes) a temporary directory, runs the code and manages all necessary file transfers.
• merge_job.exe does exactly what the filename says, merging outputs into a single, final file.
• copy_files_dcap2.exe is used to actually copy the input files to the local batch node.
• copy_files_dcap2.py is a wrapper for the shell script with the same name, just to ensure it's executed in a different shell (this should be tidied up at some point).

These files need to be made executable:

chmod u+x *.exe submit_array_dcap2_v3.py copy_files_dcap2.py

Necessary edits

As with the code, these scripts will need a little tweaking for each user.

File locations

The batch jobs will need access to the files descibed earlier here. Relative to where you submit the jobs from, you will need to create these directories:

• GRL for the Good Runs List file(s) (*.xml)
• rootfiles for all other files (electron maps etc)

submit_array_dcap2_v3.py

If you are running at RZG, the edits below are optional. If you want to submit from MPI, some variables describing input and output locations will need to be set according to the needs of each user:

• MyOutDir: Somewhere on your local system with space for output to be stored, for example under.
• MyTempDir: Similar, but this is only used temporarily by the jobs, so must be local to RZG. The default is probably OK for most users.
• MyBatchLogs: Directory to store the job log files, could be in /ptmp or your home area at RZG.
• MyEmail: You will receive an email if a job fails or is aborted.
• q: "short" is OK most of the time. If your jobs take longer than two hours, you will need to change this to "standard".

If you are submitting from MPI, you will need to create output_dir remotely; uncomment this line and edit it appropriately:

# os.system(" ssh flowerde@pcatlas57.mpp.mpg.de mkdir -p "+output_dir)

Don't forget to remove the line after it!

batch_jobdcap2_v3.exe

Check that the variables ATHENAWORKDIR and INITSCRIPT are set correctly for you.

TMPDIR is set to be in /tmp by default (that is, on the worker node itself). This can apparently cause job crashes if other users' jobs do not remove their temporary files, so there is a suggested alternative under /ptmp that you could use instead.

merge_job.exe

Make sure that ATHSETUP does what you want it to do. Also, check the variable errorlocation, which is where logfiles will be put if the copy fails (should never happen if you're running at RZG). If you're submitting from MPI, you will need to uncomment all the lines referring to scp.

Job options

The input and output are given by the script executed on the batch node. To pick this up correctly, make sure your input/output lines look like this:

NtupleReaderAlg.fileList = "filelist.txt"
NtupleReaderAlg.numberOfEvents = -1
(...)
analysisTool.outputFileName = outputfilename

Submit!

The syntax for the submission script is this:

./submit_array_dcap2_v3.py <filelist> <joboptions file> <name for the job == output folder> [<files per job>] [<job number>]

The first two arguments are your file list and job options file, the third is the desired output location, which doubles as the job name. The last two arguments are optional: the first of these specifies how many files are processed in each job (default is 20), while the final argument tells the script to only submit the job with that particular number, obviously less than or equal to the length of the file list divided by the number of files per job. If it is the string "test", a local test job will be run instead.

An example command for running one local test (from SUSYAnalysis/run):

./submit_array_dcap2_v3.py SUSYtest/VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt SUSY_Analysis_full.py firstZZtest 1 "test"

A test batch job:

./submit_array_dcap2_v3.py SUSYtest/VUA18_MC11c/filelist_mc11_7TeV.105986.ZZ_Herwig.merge.AOD.e825_s1310_s1300_r3043_r2993.txt SUSY_Analysis_full.py firstZZtest 10

This will create 5 jobs (plus a merge job) because there are 50 ZZ files.

The batch backend is Sun Grid Engine (SGE), meaning that you can use the commands qstat and qdel to monitor and remove jobs, respectively. Refer to the command man pages for more details.

Alternative: UCI Ntuples + common software

This is apparently very quick and easy to install, let's see if it's true! The main packages are:

• The 2-3 lepton common code, with an email archive.
• The UCI ntuples.

Installation

Assuming you only want to read the ntuples, the following is sufficient (based on the README file). You should search for the latest tag, or else check out the trunk.

svn co svn+ssh://svn.cern.ch/reps/atlasinst/Institutes/UCIrvine/SUSYAnalysis/SusyNtuple/tags/SusyNtuple-00-00-21 SusyNtuple
# IMPORTANT: Edit install.sh, commenting or modifying the athena setup as appropriate
source SusyNtuple/scripts/install.sh
# At the end, this will complain about package Mt2 being missing. You can fix that like this:
svn co svn+ssh://svn.cern.ch/reps/atlasphys/Physics/SUSY/Analyses/WeakProduction/Mt2/tags/Mt2-00-00-01 Mt2
RootCore/scripts/build.sh

If you want to produce the ntuples, you will need the SusyCommon package, which has a similar install script. Please add notes if you try it!

Before you build, you need to fix one missing dependency in TriggerMatch/cmt/Makefile.RootCore:

PACKAGE_DEP      = SUSYTools

Regular use

Every time you start a new session, you need to do this, in addition to setting up ROOT:

cd RootCore
source scripts/setup.sh   
cd ..

To compile, run this script noting that this is RootCore compilation, not athena:

$ROOTCOREDIR/scripts/build.sh 

You can also compile an individual package if you need to:

cd ../cmt
make -f Makefile.RootCore

-- MichaelFlowerdew - 01-Jun-2012