The Professor's Almanac

I Really Want to Develop Athena Code in Release 22

Begin by forking ATLAS Athena into your private user area in CERN Gitlab:

https://gitlab.cern.ch/atlas/athena/

(select to "Fork" the project and put it in the space corresponding to your name. Forking takes time, so go get a coffee.)

The following is based on instructions from Carlo Varni:

• The basic instructions are available in a set of slides: https://twiki.cern.ch/twiki/pub/Atlas/BjetMigrationToAthenaMT/RunningAthena.pdf
  • For b-jet trigger-related developments the following packages should be compiled:
    • Trigger/TriggerCommon/TriggerMenuMT
    • Trigger/TrigHypothesis/TrigBjetHypo

Here is Carlo's package_filters_example.txt file contents in the source/ folder:

+ Trigger/TriggerCommon/TriggerMenuMT
+ Trigger/TrigHypothesis/TrigBjetHypo
- .*

Test Your Setup

After these packages have been compiled and everything is in place, according to Carlo's instructions, one should source the following bash file in order to run a b-jet trigger test:

source https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TrigValidation/TrigUpgradeTest/test/test_trigUpgr_slice_bjet_build.sh 

If you’ve done a full checkout you already have this package in your Athena folder. You don’t actually need to compile it.

CAVEAT: Be aware this test does not produce AODs!

The test will produce an athena.log file that you can check. This means that you need to put the OutputLevel for the algorithms you are interested in to DEBUG level.

Some Words on the Code

• The b-jet trigger code itself is available here: https://gitlab.cern.ch/atlas/athena/tree/master/Trigger/TrigHypothesis/TrigBjetHypo
• Right now, in AthenaMT, we only have a dummy BtagFexMT that only retrieve tracks, Primary vertex and jets: https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TrigHypothesis/TrigBjetHypo/src/TrigBtagFexMT.cxx
• The b-jet trigger sequence is actually being defined here: https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TriggerCommon/TriggerMenuMT/python/HLTMenuConfig/Bjet/BjetSequenceSetup.py
• The flavour tagging-specific sequence configuration will end up here: https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TriggerCommon/TriggerMenuMT/python/HLTMenuConfig/Bjet/BjetFlavourTaggingConfiguration.py

Bjet Monitoring Code

The Bjet Monitoring code lives in Athena under: https://gitlab.cern.ch/atlas/athena/tree/master/Trigger/TrigMonitoring/TrigBjetMonitoring. This is Offline Monitoring. The Online Monitoring is achieved by modification of the TrigBjetHypo code, https://gitlab.cern.ch/atlas/athena/tree/master/Trigger/TrigHypothesis/TrigBjetHypo.

Updated instructions on monitoring in Run 3/AthenaMT are available from this presentation by Frank Winklmeier:

• https://indico.cern.ch/event/883073/contributions/3762439/attachments/1996427/3330985/20200302-MTMonitoring.pdf

The documentation on using the R22/AthenaMT monitoring framework is here: https://atlassoftwaredocs.web.cern.ch/guides/trigger/monitoring/

I Really Want to Make Some Monte Carlo for a Student

So you have a student. And they need MC simulation for their research. However, their process is not part of the official ATLAS simulation. But you really want it now, and you don't want to wait for the grinding machines of bureaucracy to make the MC simulation for you. Well, here is how you might get started.

1. First, try to find an MC channel ID that is closest to what you want. For this example, I want to simulate a highly-boosted Higgs boson produced in gluon-gluon fusion (ggF) and then decaying to something crazy, like gluons. There is already a simulation of pp → j+H where the H → bb. That has channel ID 309450. That will be our starting point.
2. Once you have a channel ID, use the AMI tag browser to understand the provenance of how this MC is generated from the production and showering (e.g. POWHEG + PYTHIA8, resulting in the EVNT format of data); to simulating hits using a GEANT4 detector simulation (resulting in a HITS file format); to a full reconstruction of the event (e.g. resulting in an AOD file format or a derivation of that).
   1. Start from the top-level AMI page, http://ami.in2p3.fr.
   2. Then select "Dataset Browser".
   3. Set the "Data Type" to "Simulated Data"
   4. Select the appropriate simulation base project. For this tutorial, I will use mc16.
   5. Now check the "dataSetNumber" search field option and enter the channel ID: 309450.
   6. Be patient.
   7. Your patience should be rewarded by a green button and some text that appear indicating how many data sets match your search criteria.
   8. Click the "View Selection" button that has appeared.
   9. This pops up the view in a new "tab" inside the website.
   10. I looked for a full reconstructed AOD and found this: "

       mc16_13TeV.309450.PowhegPy8EG_NNLOPS_nnlo_30_ggH125_bb_kt200.recon.AOD.e7338_s3126_r10724

       ". I clicked the "Provenance" link to go to its information page. The Provenance will tell you the sequence of steps used to generate this sample.
   11. You can visually see how EVNT was generated and how that was then mixed with MinBias events to simulate pileup. Then this was all passed through simulation and reconstruction. The tags of interest are: e7338 (EVNT), e5984 (EVNT for MINBIAS), s3126 (DETECTOR SIMULATION), and r10724 (EVENT RECONSTRUCTION).

Now you have the tags. It's time to use the tag browser to find the command used to execute each step!

1. To reproduce the EVNT production of the primary interaction...
   1. Use AMI
      1. Go back to https://ami.in2p3.fr
      2. Click on the "Tag Browser" button
      3. Enter the tag of interest (e.g. e7338).
      4. This page now shows the command-line options passed to Generate_tf.py to make the sample. For instance from this tag we can reconstruct the ATLAS setup command needed to reconstruct the running environment is "asetup AtlasProduction,19.2.5.23". The command line option for running simulation has to be:

          Generate_tf.py --ecmEnergy 13000 --evgenJobOpts MC15JobOpts-00-12-79_v1.tar.gz <plus other options> 

      5. Note that, depending on the process in question, this step can take a very long time per event. To give you a sense of scale, this specific case of POWHEG + PYTHIA8 takes about 23-24 hours to generate 1000 events in EVNT format on a modern CPU.
   2. Use the command line:

       GetTfCommand.py --AMI xNNN 

2. To then convert EVNT to HITS...
   1. Again use the tag browser to look up the settings for this stage. For example, for the case above the tag of interest is s3126. An example command line based on this tag would be

      Sim_tf.py --maxEvents 50 --inputEVNTFile ../../MonteCarlo/MonteCarloProcess-000001/MonteCarloProcess-1.EVNT.root --outputHITSFile MonteCarloProcess-1.HITS.root --DBRelease "all:current" --DataRunNumber 284500 --conditionsTag "default:OFLCOND-MC16-SDR-14" --geometryVersion "default:ATLAS-R2-2016-01-00-01_VALIDATION" --physicsList FTFP_BERT_ATL_VALIDATION --postInclude "default:PyJobTransforms/UseFrontier.py" --preInclude "EVNTtoHITS:SimulationJobOptions/preInclude.BeamPipeKill.py,SimulationJobOptions/preInclude.FrozenShowersFCalOnly.py" --simulator FullG4 --truthStrategy MC15aPlus --AMITag s3126

   2. Mind the release that was used for this step, and make sure you execute asetup again to reset the release environment appropriate for the simulation (HITS production) step.
   3. If you thought the previous step took time, wait for this one! For this specific MC sample, HITS production averaged 233 seconds per event (averaged over about 350 events). You need to break this stage up into even smaller batches to process in finite time.
3. Now we can convert HITS to AOD. Lookup the options used for tag r10724 in the above example. Now run the reconstruction transform:

    Reco_tf.py --maxEvents 50 --inputHITSFile ../../jHgg-reco/MonteCarloProcess-{fileNumber:06d}/MonteCarloProcess-{fileNumber}.HITS.root  --outputAODFile AOD.pool.root --autoConfiguration everything --conditionsTag default:OFLCOND-MC16-SDR-25 --digiSteeringConf StandardSignalOnlyTruth --geometryVersion "default:ATLAS-R2-2016-01-00-01" --postExec "all:CfgMgr.MessageSvc().setError+=[\"HepMcParticleLink\"]" --postExec "ESDtoAOD:fixedAttrib=[s if \"CONTAINER_SPLITLEVEL = 99\" not in s else \"\" for s in svcMgr.AthenaPoolCnvSvc.PoolAttributes];svcMgr.AthenaPoolCnvSvc.PoolAttributes=fixedAttrib" --postInclude "default:PyJobTransforms/UseFrontier.py"  --preExec "all:rec.Commissioning.set_Value_and_Lock(True);from AthenaCommon.BeamFlags import jobproperties;jobproperties.Beam.numberOfCollisions.set_Value_and_Lock(20.0);from LArROD.LArRODFlags import larRODFlags;larRODFlags.NumberOfCollisions.set_Value_and_Lock(20);larRODFlags.nSamples.set_Value_and_Lock(4);larRODFlags.doOFCPileupOptimization.set_Value_and_Lock(True);larRODFlags.firstSample.set_Value_and_Lock(0);larRODFlags.useHighestGainAutoCorr.set_Value_and_Lock(True); from LArDigitization.LArDigitizationFlags import jobproperties;jobproperties.LArDigitizationFlags.useEmecIwHighGain.set_Value_and_Lock(False)" --preExec "ESDtoAOD:from TriggerJobOpts.TriggerFlags import TriggerFlags;TriggerFlags.AODEDMSet.set_Value_and_Lock(\"AODSLIM\");" --preInclude "HITtoRDO:Digitization/ForceUseOfPileUpTools.py,SimulationJobOptions/preInlcude.PileUpBunchTrainsMC16c_2017_Config1.py,RunDependentSimData/configLumi_run310000.py" --steering doRDO_TRIG --triggerConfig "RDOtoRDOTrigger=MCRECO:DBF:TRIGGERDBMC:2216,76,260"  --valid True --jobNumber 1 --inputHighPtMinbiasHitsFile ../{minbias_high_file} --inputLowPtMinbiasHitsFile ../{minbias_low_file} --AMITag r10724'.format(taskdirectory=taskdirectory, fileNumber=fileNumber, minbias_high_file=minbias_high_file, minbias_low_file=minbias_low_file

   1. What's the deal with those "Low" and "High" MinBias files? They are needed to mix in pileup background on top of the simulated primary hard-scatter collision. I had trouble trying to download files from those MinBias containers without specifying the GRID endpoint from which I intend to copy it (they are replicated across SO MANY GRID endpoints!). Here is what I did:

      rucio download --ndownloader 5 --nrandom 100 --rse SWT2_CPB_DATADISK --allow-tape mc16_13TeV.361238.Pythia8EvtGen_A3NNPDF23LO_minbias_inelastic_low.simul.HITS.e4981_s3087_s3111

      and

      rucio download --ndownloader 5 --nrandom 100 --rse SWT2_CPB_DATADISK --allow-tape mc16_13TeV.361239.Pythia8EvtGen_A3NNPDF23LO_minbias_inelastic_high.simul.HITS.e4981_s3087_s3111

4. The last step is AOD to DAOD conversion. This is easy. For each AOD file, execute the following:

   Reco_tf.py --passThrough True --preExec "default:from BTagging.BTaggingFlags import BTaggingFlags;BTaggingFlags.CalibrationTag = \"BTagCalibRUN12-08-49\"; from AthenaCommon.AlgSequence import AlgSequence; topSequence = AlgSequence(); topSequence += CfgMgr.xAODMaker__DynVarFixerAlg(\"InDetTrackParticlesFixer\", Containers = [ \"InDetTrackParticlesAux.\" ] );from AthenaMP.AthenaMPFlags import jobproperties as ampjp;ampjp.AthenaMPFlags.UseSharedWriter=True;import AthenaPoolCnvSvc.AthenaPool;ServiceMgr.AthenaPoolCnvSvc.OutputMetadataContainer=\"MetaData\";topSequence += CfgMgr.xAODMaker__DynVarFixerAlg(\"BTaggingELFixer\", Containers = [\"BTagging_AntiKt4EMTopoAux.\" ] );" --reductionConf EXOT8 --sharedWriter True --inputAODFile ../jHgg-aod/estt.AOD.pool.root --outputDAODFile test.DAOD.EXOT8.pool.root

   1. Again, the specific command-line options can be looked up from the DAOD tag. I used EXOT8 for this samply, so the tag was p4006.

-- StephenSekula - 2019-07-03