Cosmic Simulation in rel 14: How to

How to run simulation:

The new simulation transform is included in 14.5.1.4,AtlasProduction,runtime. Run without arguments for instructions - here are some examples anyway:

• • Pixel volume (= AND of TRT Barrel and pixel volume)

csc_cosmics_sim_trf.py outputHitsFile=sim.root maxEvents=5000 runNumber=123 randomSeed=1234 geometryVersion=ATLAS-GEO-07-00-00 CosmicFilterVolume=TRT_Barrel CosmicFilterVolume2=Pixel jobConfig=Cosmics.py

• • ID Volume (= OR of TRT Barrel and 2 TRT ECs)

csc_cosmics_sim_trf.py outputHitsFile=sim.root maxEvents=5000 runNumber=123 randomSeed=1234 geometryVersion=ATLAS-GEO-07-00-00 CosmicFilterVolume=TRT_Barrel CosmicFilterVolume2=TRT_EC jobConfig=Cosmics.py

• • Muon Volume (= whole detector)

csc_cosmics_sim_trf.py outputHitsFile=sim.root maxEvents=5000 runNumber=123 randomSeed=1234 geometryVersion=ATLAS-GEO-07-00-00 CosmicFilterVolume=Muon  jobConfig=Cosmics.py

• The above commands will automatically choose the correct condDB tag from the DetDescrVersion (condDB tag is magnetic field dependent - as is the DetDescrVersion)
• In production the runnumber and the first event number for each jobs can be set by input arguments to the transform:

 runNumber=XXX firstEvent=YYYY

• This is needed to be able to have unique run/event pairs when stuff is produced in production
• setting runnumber is important as having runnumber=0 (current default which i will change) causes monitoring to crash in reco step.

Trying to get digitization running from a transform

• John Chapman has got cosmic simulation digitization working out of the box in a fixed production release from a transform (so it can be run in production). (thanks John)
• Currently in 14.5.1.4 AtlasProduction can run like

csc_digi_trf.py inputHitsFile=rfio:/castor/cern.ch/grid/atlas/atlasgroupdisk/proj-simcos/rel14/ATLAS-GEO-03-00-00/sim/pixelbarrel_B-400299/sim.root outputRDOFile=test.RDO maxEvents=10 skipEvents=0 geometryVersion=ATLAS-GEO-03-00-00 jobConfig=SimuJobTransforms/CosmicsDigitConfig.py  digiSeedOffset1=0 digiSeedOffset2=0

• This works. Remember to set STAGE_SVCCLASS=atlasgroupdisk to use this example (if you want to access the HITS file from castor)
• The CondDB tag used here is automatically chosen from that used in the simulation step

Running cosmic sim reco with Reco_trf transform

• For most efficient help, write questions directly to hn-atlas-offline-commissioning HN
• Now use rel. 14.5.2.11,AtlasProduction
• Run transform like:

>Reco_trf.py inputRDOFile=dig.root maxEvents=10 preInclude=RecExCommon/RecoUsefulFlags.py,RecExCommission/CosmicSimulationRecoSetup.py,RecExCommission/RecExCommissionRepro.py,RecJobTransforms/UseOracle.py  beamType=cosmics outputESDFile=myESD.pool.root HIST=myMergedMonitoring.root  outputTAGComm=myTAGCOMM.root outputCBNT=myCommission.ntuple.root geometryVersion=ATLAS-GEO-07-00-00 --ignoreunknown

• Can change outputs by changing transform arguments (ie. if you dont give a outputCBNT argument wont get a CBNT) see this wiki for more details on Reco_trf
• Reco field setup and condDB should be autoconfigured by the DetDescrVersion in CosmicSimulationRecoSetup.py

• If you want to run all monitoring (RAW based and ESD based) all in the RAW step you can add to the JO:

from AthenaMonitoring.DQMonFlags import DQMonFlags
DQMonFlags.monType.set_Value_and_Lock('BSall')

• Or the above can be added as a preExec - in this case the DQMonFlags are already imported so you can just add:

preExec="DQMonFlags.monType.set_Value_and_Lock('BSall')"

• Or can run combined monitoring by giving an AODoutputfilename to the transform which means that art of the monitoring will be done in the ESD->AOD step and then the monitoring histograms merged
• Problem i see right now is that although the job runs through to completion with no problems but the transform returns non-0 exit code due to ERROR messages in the logfile which will be a problem for running in production. needs to be fixed!

Test of simulation step in Production

• Borut has kindly run 3 tasks (to give ~50k events each) of just the simulation step in production
• These all use GEO-03 DetDescrVersion (both solenoid and toroid on)
• The tasks are
  • pixel volume: http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=40593
  • trt barrel volume: http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=40594
  • muon volume: http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=40595
• You can download the HITS files to play with by doing using dq2-get eg.

dq2-get valid2.000700.Cosmic_test.simul.HITS.s507_tid040594

• A few performance plots on CPU, number of events etc are attached to this twiki here

Test of digitization step in production

• Borut has set off digitization tasks on the above simulated events
• Tasks are
  • http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=42839
  • http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=42840
  • http://panda.cern.ch:25880/server/pandamon/query?mode=showtask0&reqid=42841
• You can download the produced RDOs using dq2-get using eg.:

dq2-get valid2.000700.Cosmic_test.digit.RDO.s507_d155_tid042840

Current validation

• I have run digitization and reconstruction as described above over 689 events from the production tests (i chose the TRTBarrel, ATLAS-GEO-03-00-00 sample)
• You can see the produced monitoring histograms here
• You can see a couple of event displays here

Known problems

• Cant run trigger in reco - i suspect there are many problems for this. the first i get to is:

Shortened traceback (most recent user call last):
  File "/afs/cern.ch/atlas/software/builds/AtlasCore/14.5.1/InstallArea/share/bin/athena.py", line 459, in <module>
    include( script )
  File "/afs/cern.ch/atlas/software/builds/AtlasProduction/14.5.1.3/InstallArea/jobOptions/RecJobTransforms/skeleton.RAWtoESD.py", line 137, in <module>
    else: include( "RecExCommon/RecExCommon_topOptions.py" )
  File "/afs/cern.ch/atlas/software/builds/AtlasProduction/14.5.1.3/InstallArea/jobOptions/RecExCommon/RecExCommon_topOptions.py", line 540, in <module>
    protectedInclude ("AthenaMonitoring/DataQualitySteering_jobOptions.py")
  File "/afs/cern.ch/atlas/software/builds/AtlasEvent/14.5.1/InstallArea/jobOptions/AthenaMonitoring/DataQualitySteering_jobOptions.py", line 46, in <module>
    treatException("DataQualitySteering_jobOptions.py: exception when setting up L1 Calo monitoring")
  File "/afs/cern.ch/atlas/software/builds/AtlasEvent/14.5.1/InstallArea/jobOptions/AthenaMonitoring/DataQualitySteering_jobOptions.py", line 44, in <module>
    include("TrigT1Monitoring/TrigT1Monitoring_forRecExCommission.py")
  File "/afs/cern.ch/atlas/software/builds/AtlasTrigger/14.5.1/InstallArea/jobOptions/TrigT1Monitoring/TrigT1Monitoring_forRecExCommission.py", line 7, in <module>
    TTjob += LVL1__TriggerTowerMaker( 'TriggerTowerMaker' )
TypeError: attempt to redefine type of "TriggerTowerMaker" (was: LVL1__TriggerTowerMaker, new: TriggerTowerMaker_TTL1_Rel13)
Py:Athena            INFO leaving with code 8: "an unknown exception occurred"

• Muon timing wrong
  • Testing with UseTOF=False in digitization
  • see This talk for info on RPC timing (its wrong) and trigger issues...
• Some problems with calibration/alignment???
  • was a problem with TRT calibration - solved with new condDB tags i hope
• Cosmic charge assymmetry wrong in simulation
• Generator charge asymmetry is wrong
  • ~1.63 in generator, PDG value ~1.27

Current developments

New Geometry tags (done)

• Pixel need a new set of DetDescrVersions as their HV comes from the geometry DB (and is wrong in the old ones) so we will create

ATLAS-GEO-07-00-00, ATLAS-GEONSF-07-00-00, ATLAS-GEONF-07-00-00, ATLAS-GEONTF-07-00-00

• which are exactly the same as the GEO-03/04 ones but with the pixel HV fixed (this is bug report 47235 )
• These have been created and have the correct InnerDetector-GEO-00-01 tag which has the corrected ID setup in it (from Grant)
• needed for Attillio, made by Vahko
• with this we get the error:

PixelLorentzAng...  ERROR Conditions Summary Service required but not requested. Will use temperature and voltages from GeoModel or job options.

• Which we need to get rid of as causes transform problems (fixed by SiLorentzAngleSvc-00-00-07-02 which is in 14.5.1.4 nightly).

New conditions DB tags (done)

• We have created new condDB tags for sim, dig, reco of simulated cosmics (in rel. 14.5.X). Paul has created these tags

OFLCOND-SIMC-00-00-00   [Cosmic simulation - Solenoid off / Toroid off]
OFLCOND-SIMCS-00-00-00  [Cosmic simulation - Solenoid on / Toroid off]
OFLCOND-SIMCT-00-00-00  [Cosmic simulation - Solenoid off / Toroid on]
OFLCOND-SIMCST-00-00-00 [Cosmic simulation - Solenoid on / Toroid on]

• These tags have been locked (on 9 march 2009) and a DBrelease containing these changes and the updated Geometry tags is being built.
• experts, Vahko, Paul Laycock

New filter volume (done)

• The ID asked us to change from using TRTBarrel to ID volume (to increase TRT endcap tracks)
• Problem is ID volume is much less efficient (only ~30% of events have an offline track)
• Else is looking at other options but im not sure there will be something useful.
• Need to decide what to use soon....
• with tags: G4CosmicFilter-00-00-13 and a SimuJobTransforms-00-08-05-02 (both requested for 14.5.X.Y nightlies)
• driven by Christian

New pixel step size in pixel digitization (done)

• New tags from pixel

SiLorentzAngleSvc-00-00-07-01 
PixelDigitization-00-07-17-05

• Needed for bug: 47684
• These tags are in 14.5.X.Y nightlies
• Driven by Attillio

Muon digitization (not done - working on it for 15.1.0 i think)

• Muon group discussing if the MdtDigitization should have

MDTDigitization.UseTOF=False

• Need to know soon, and if there are any associated changes for cosmic sim reco (if so what) - not sure if this will get in for first production
• Driven by Niels
• Not sure of timescale for this. maybe deferred to second production (or redigitization/reco of first at a later time).

Get elavator shafts into GeoModel (not done for 15 based sim i hope)

• on page 16 of this talk you can see that the RPC see the effect of the elevator shafts in real data
• Andreas Korn is working of getting this into GeoModel but probably this wont get into the first production

Runnumber/Eventnumber issues (done)

• With the tag SimuJobTransforms-00-08-05-03 (requested for 14.5.1.4) one can specifiy optional arguments of runNumber and firstEvent to the simulation transform
• This means that we will be able to run with a single runnumber for all the jobs in a task (cosmic simulation configuration) and gaurentees unique run/event numbers
• The production system should be able to deal with this correctly

Extra functionality to get this working in production

* Runnumber, firstevent number setting for simulation step works, and should be able to be done by production system (Pavel tells me).

• Need to come up with different statistics for types of jobs (for each filter volume) - proposed initial sample sizes

1M Pixel Volume - both fields on
1M ID Volume - both fields on <-- maybve upto 5M evts????
0.5M Muon volume - both fields on
1M Pixel Volume - both fields off
1M ID Volume - both fields off
0.5M Muon volume - both fields off

• Validation of all of above changes - ongoing by Andreas Korn

Release issues

• Looks like we will run the simulation and digitization in 14.5.1.Y AtlasProduction (David Q proposed stopping 14.5.1.Y soon - we need to keep it open (but closed for developer submission) for cosmic sim)
• Then run reco in 14.5.2.Y AtlasProduction (but no tags into that until after the re-reprocessing release is fixed)
• Need to make sure all changes for this in branches are also in the head (so we can do a rel. 15 production one day).

Plan now (end of march 2009)

• 14.5.1.4 is out and Andreas is testing the full chain
• Then try to launch in production the following stats in 14.5.1.4 (reco in 14.5.2.Y)

1M Pixel Volume - both fields on
1M ID Volume - both fields on
0.5M Muon volume - both fields on
1M Pixel Volume - both fields off
1M ID Volume - both fields off
0.5M Muon volume - both fields off

Setup for special pixel digitization

• See this wiki

Future plans

• Things that have missed 14.5.1.4 but should go in for future(in no particular order!)
• Muon digitization fix(s)
• Muon dead-channel map from DB
• Trigger fix(s)
  • muon timing
  • trg configuration
• Use a calo volume (or with something else for ECs?) for muon volume (current muon volume too big and most events useless) - Claude (done for 15.2.0 with G4CosmicFilter-00-00-14 and SimuJobTransforms-00-09-03)
• testing - should have a full chain test for cosmic sim running (only a few events) in nightlies (zach has something for this here)
• revisit the idea of storing a large sample of cosmics at the entrance of cavern, and then run from these (with some small smearing) to save speed (This functionality is in place for 15.2.0)
  • This was worked on by Ahmet Bingul in 2007 and he started writing a package to do this: CosmicSmear but never finished this (added simple positional and directional smearing to CosmicGenerator - now being validated, but in place for 15.2.0)
  • May be easy to finish this work.... (swagato - looking into this)
• elevator shafts in rock description
• Momentum sliced samples (change surface area size and generator params) - (missing et reco issue??) (Done for 15.2.0; required modifications to cavern infrastructure GeoModel to change the size of the bedrock above ATLAS, so cannot be backported)
• physics lists used etc...

Description of how propagation through rock is done (from Zach - thanks)

In the logfile, for every type of material, we get a range cut printed out.

Region <SX1> -- appears in <World::World> world volume
Production cuts :  gamma 2 m      e- 2 m      e+ 2 m

Region <BEDROCK> -- appears in <World::World> world volume
Production cuts :  gamma 1 km     e- 1 km     e+ 1 km

Region <CAV_SHAFTS_CONC> -- appears in <World::World> world volume
Production cuts :  gamma 15 cm     e- 15 cm     e+ 15 cm

These are the cuts that Andrea mentioned.  They should be long enough that secondaries are not being created with any real frequency.  
The next thing to check is the length of the steps.  The muons take about 200-300 steps moving through the bedrock, mostly step-limited 
by interactions (>99%).  That means loosening stepping requirements or "turning off" the muon charge (for example) would not help speed 
things up much.  A typical physics event is several million steps, and a muon traveling straight through the ATLAS detector takes a few 
thousand steps, for comparison.

-- ElseLytken - 23 Feb 2009

• stats.ps: CPU performance for grid tests