Validation of Menu Run 2015 4.4

Size of streamer files

-rw-rw-rw- 1 daqlocal daqlocal 4.2M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsEGammaCommissioning/run256676_ls0001_streamPhysicsEGammaCommissioning_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 5.8M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsEGammaCommissioning/run256676_ls0002_streamPhysicsEGammaCommissioning_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 5.8M Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsEGammaCommissioning/run256676_ls0003_streamPhysicsEGammaCommissioning_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal    0 Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsEndOfFill/run256676_ls0001_streamPhysicsEndOfFill_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal    0 Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsEndOfFill/run256676_ls0002_streamPhysicsEndOfFill_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal    0 Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsEndOfFill/run256676_ls0003_streamPhysicsEndOfFill_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 4.4M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsHadronsTaus/run256676_ls0001_streamPhysicsHadronsTaus_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 4.1M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsHadronsTaus/run256676_ls0002_streamPhysicsHadronsTaus_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 3.7M Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsHadronsTaus/run256676_ls0003_streamPhysicsHadronsTaus_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 6.5M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsMuons/run256676_ls0001_streamPhysicsMuons_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 2.4M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsMuons/run256676_ls0002_streamPhysicsMuons_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 4.0M Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsMuons/run256676_ls0003_streamPhysicsMuons_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 1.6M Oct 14 09:53 /fff/BU0/output/run256676/streamPhysicsParkingScoutingMonitor/run256676_ls0001_streamPhysicsParkingScoutingMonitor_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 2.7M Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsParkingScoutingMonitor/run256676_ls0002_streamPhysicsParkingScoutingMonitor_hilton-c2f13-33-04.dat
-rw-rw-rw- 1 daqlocal daqlocal 3.3M Oct 14 09:54 /fff/BU0/output/run256676/streamPhysicsParkingScoutingMonitor/run256676_ls0003_streamPhysicsParkingScoutingMonitor_hilton-c2f13-33-04.dat

HLT rates (selected)

HLT Path Name                                                      L1 seed counts    Prescale counts    HLT path counts
HLT_Ele22_eta2p1_WPLoose_Gsf_v3                                       1695          1695             14 
HLT_Ele22_eta2p1_WPTight_Gsf_v3                                       1695          1695             13 
HLT_Ele30WP60_SC4_Mass55_v3                                           1105          1105              5 
HLT_Ele30WP60_Ele8_Mass55_v2                                          1105          1105              4 
HLT_Ele23_WPLoose_Gsf_v3                                              3130          3130             20 
HLT_Ele27_WPLoose_Gsf_v1                                              2408          2408             12 
HLT_Ele27_eta2p1_WPLoose_Gsf_v2                                       1571          1571             10 
HLT_Ele27_eta2p1_WPTight_Gsf_v2                                       1571          1571              9 
HLT_Ele32_eta2p1_WPTight_Gsf_v2                                        386           386              6 
HLT_HT470to500_v1                                                      655           655              4 
HLT_HT550to650_v1                                                      655           655              5 
HLT_IsoMu18_v2                                                         650           650              8 
HLT_OldIsoMu18_v1                                                      650           650              6 
HLT_IsoMu20_v3                                                         650           650              5 
HLT_IsoMu22_v2                                                         435           435              4 
HLT_IsoTkMu18_v2                                                       650           650             10 
HLT_OldIsoTkMu18_v2                                                    650           650              8 
HLT_IsoTkMu20_v4                                                       650           650              5 
HLT_IsoTkMu22_v2                                                       435           435              4 
HLT_QuadPFJet_DoubleBTagCSV_VBF_Mqq200_v4                             1209          1209              4 
HLT_QuadPFJet_DoubleBTagCSV_VBF_Mqq240_v4                             1209          1209              4 
HLT_BTagMu_DiJet20_Mu5_v2                                               84            84              5 
HLT_BTagMu_DiJet70_Mu5_v2                                              111           111              5 
HLT_Ele17_CaloIdL_TrackIdL_IsoVL_v2                                   1575           144              7 
HLT_Dimuon16_Jpsi_v2                                                   106           106              4 
HLT_Dimuon10_Jpsi_Barrel_v2                                            488           488              7 
DST_Physics_v1                                                       12548          1255           1255 
DST_HT250_CaloScouting_v2                                              655           655            150 
DST_HT450_BTagScouting_v1                                              655           655             15 
DST_HT450_PFScouting_v1                                                655           655             15 
DST_DoubleMu3_Mass10_BTagScouting_v1                                   112           112              6 
DST_DoubleMu3_Mass10_PFScouting_v1                                     112           112             18 
HLT_DoubleMu3_Mass10_v1                                                112           112             18 
AlCa_EcalPhiSym_v3                                                     651           651            651 
AlCa_Ele23_WPVeryLoose_Gsf_v1                                         2408          2408             28 
AlCa_RPCMuonNoTriggers_v2                                              801           801            168 
AlCa_RPCMuonNoHits_v2                                                  801           801            233 
AlCa_RPCMuonNormalisation_v2                                           801           158             98 
AlCa_LumiPixels_ZeroBias_v2                                            651            16             16 
AlCa_EcalPi0EEonly_v3                                                 5079          5079            617 
AlCa_EcalPi0EBonly_v3                                                 5079          5079           3122 
AlCa_EcalEtaEBonly_v3                                                 5079          5079           1575 
AlCa_EcalEtaEEonly_v3                                                 5079          5079            229 

Prescale checker

Invoked with: /nfshome0/hltpro/RateMon/CheckPrescales.py --ignore=0,1,12 /cdaq/test/tomei/Run2015/4.4/V1/HLT/V1 gt_2015_Stage1_Oct09_2b gtrs_2015_25ns_work51_1825b

List of triggers with non-sequential prescales:
hlt path: HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v3
l1t seed: L1_SingleEG10 OR L1_SingleEG5
total p.: [1, 1, 1, 1, 1, 1, 1, 8, 8, 8, 8, 8, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 1 ', ' 1 ', ' 1 ', ' 1 ', ' 8 ', ' 8 ', ' 8 ', ' 8 ', ' 8 ', ' 1  ']
l1t pre.: ['1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', 262139]

hlt path: HLT_Ele23_CaloIdL_TrackIdL_IsoVL_v3
l1t seed: L1_SingleEG20
total p.: [4000, 4000, 2000, 10000, 700, 400, 200, 140, 100, 70, 40, 20, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 1000 ', ' 700 ', ' 400 ', ' 200 ', ' 140 ', ' 100 ', ' 70 ', ' 40 ', ' 20 ', ' 1  ']
l1t pre.: [4000, 4000, 2000, 10, 1, 1, 1, 1, 1, 1, 1, 1, 262139]

hlt path: HLT_Ele23_CaloIdM_TrackIdM_PFJet30_v3
l1t seed: L1_SingleEG20
total p.: [4000, 4000, 2000, 6500, 530, 300, 150, 120, 80, 60, 32, 16, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 650 ', ' 530 ', ' 300 ', ' 150 ', ' 120 ', ' 80 ', ' 60 ', ' 32 ', ' 16 ', ' 1  ']
l1t pre.: [4000, 4000, 2000, 10, 1, 1, 1, 1, 1, 1, 1, 1, 262139]

hlt path: HLT_Ele12_CaloIdL_TrackIdL_IsoVL_v3
l1t seed: L1_SingleEG10 OR L1_SingleEG15
total p.: [20000, 15000, 10000, 1250000, 875, 500, 250, 175, 125, 50, 25, 1, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 125 ', ' 125 ', ' 125 ', ' 125 ', ' 175 ', ' 125 ', ' 50 ', ' 25 ', ' 1 ', ' 1  ']
l1t pre.: [20000, 15000, 10000, 10000, '7', '4', '2', '1', '1', '1', '1', '1', 262139]

hlt path: HLT_Ele23_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
l1t seed: L1_SingleEG20
total p.: [4000, 4000, 2000, 16000, 1100, 650, 325, 250, 180, 125, 75, 36, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 1600 ', ' 1100 ', ' 650 ', ' 325 ', ' 250 ', ' 180 ', ' 125 ', ' 75 ', ' 36 ', ' 1  ']
l1t pre.: [4000, 4000, 2000, 10, 1, 1, 1, 1, 1, 1, 1, 1, 262139]

hlt path: HLT_Mu8_v3
l1t seed: L1_SingleMu5 OR L1_SingleMu7
total p.: [262139, 250, 125, 7500, 4500, 2400, 1200, 800, 550, 330, 220, 110, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 60 ', ' 60 ', ' 60 ', ' 60 ', ' 80 ', ' 110 ', ' 110 ', ' 110 ', ' 110 ', ' 1  ']
l1t pre.: [262139, '250', '125', '125', 75, 40, 20, 10, 5, 3, 2, 1, 262139]

hlt path: HLT_Ele17_CaloIdL_TrackIdL_IsoVL_v2
l1t seed: L1_SingleEG15
total p.: [24000, 24000, 12000, 132000, 77, 44, 22, 15, 11, 8, 5, 3, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 11 ', ' 11 ', ' 11 ', ' 11 ', ' 15 ', ' 11 ', ' 8 ', ' 5 ', ' 3 ', ' 1  ']
l1t pre.: [24000, 24000, 12000, 12000, 7, 4, 2, 1, 1, 1, 1, 1, 262139]

hlt path: HLT_Mu8_TrkIsoVVL_v3
l1t seed: L1_SingleMu5 OR L1_SingleMu7
total p.: [262139, 250, 125, 3750, 2250, 1200, 600, 400, 275, 165, 110, 55, 262139]
hlt pre.: ['  1 ', ' 1 ', ' 1 ', ' 30 ', ' 30 ', ' 30 ', ' 30 ', ' 40 ', ' 55 ', ' 55 ', ' 55 ', ' 55 ', ' 1  ']
l1t pre.: [262139, '250', '125', '125', 75, 40, 20, 10, 5, 3, 2, 1, 262139]

hlt path: HLT_Photon22_R9Id90_HE10_IsoM_v2
l1t seed: L1_SingleEG20
total p.: [16000, 16000, 2000, 20000, 1200, 800, 400, 300, 200, 100, 80, 40, 262139]
hlt pre.: ['  4 ', ' 4 ', ' 1 ', ' 2000 ', ' 1200 ', ' 800 ', ' 400 ', ' 300 ', ' 200 ', ' 100 ', ' 80 ', ' 40 ', ' 1  ']
l1t pre.: [4000, 4000, 2000, 10, 1, 1, 1, 1, 1, 1, 1, 1, 262139]

hlt path: HLT_Photon22_v2
l1t seed: L1_SingleEG20
total p.: [144000, 144000, 18000, 100000, 8000, 5000, 2400, 1800, 1200, 1000, 500, 260, 262139]
hlt pre.: ['  36 ', ' 36 ', ' 9 ', ' 10000 ', ' 8000 ', ' 5000 ', ' 2400 ', ' 1800 ', ' 1200 ', ' 1000 ', ' 500 ', ' 260 ', ' 1  ']
l1t pre.: [4000, 4000, 2000, 10, 1, 1, 1, 1, 1, 1, 1, 1, 262139]

Updates on the prescale checker:

• HLT_Ele23_CaloIdL_TrackIdL_IsoVL_PFJet30_v3, HLT_Ele23_CaloIdL_TrackIdL_IsoVL_v3 and LT_Ele23_CaloIdM_TrackIdM_PFJet30_v3, as well as HLT_Photon22_R9Id90_HE10_IsoM_v2 and HLT_Photon22_v2, are all seeded by L1_SingleEG20. This column changed prescale (see CMSHLT-617, Column 3 (5E33) moved from 10 to 1, and the prescale went to the HLT instead), but that is not reflected in the L1 online menu that the Prescale Checker uses. So, these are fine!
• The Prescale Checker ran into a limitation on HLT_Ele8_CaloIdM_TrackIdM_PFJet30_v3, so it cannot retrieve correctly the L1 precales. I did it manually and it is fine!
• HLT_Ele12_CaloIdL_TrackIdL_IsoVL_v3 and HLT_Ele17_CaloIdL_TrackIdL_IsoVL_v2 started depending fully on the L1 prescales for Column 3 and above. This leads to the prescales not being sequential, but it is fine since the rates will not explode. Talk to Higgs with respect to that, relevant ticket is CMSHLT-520.
• HLT_Mu8_v3 and HLT_Mu8_TrkIsoVVL_v3 seem to really be problematic. They dropped the prescales on Column 2 (7E33). Unless the L1_SingleMu5 and L1_SingleMu7 seeds are changing prescale, these rates WILL blow up, giving hundreds of Hz! So, action is needed!!!

Things to follow:

• https://its.cern.ch/jira/browse/CMSHLT-628 - Fix HLT prescales for single electron control paths
• https://its.cern.ch/jira/browse/CMSHLT-627 - Unprescale dimuon non-DZ paths
• https://its.cern.ch/jira/browse/CMSHLT-626 - Incompatible prescales for SUSY Muon control triggers at 7E33
• Make sure that the new L1-HLT trigger mode picks up the new GTRS key: gtrs_2015_25ns_work51_1825b_v2 or later
• Make sure CMSSW_7_4_15 gets installed. INSTALLED
• Make the complete diff that Andrea Bocci asked for.
• Update the OTHER keys with the new menus (Cosmics, Stage1, Twinmux)
• Put all the changes in a nice ticket for Andrea Perrota.
• Finish the deployment of the HighPU menu online, with the new PDs prescaled to zero.
• The MT test

Updates:

• My online version /cdaq/test/tomei/Run2015/4.4/V2/HLT/V5 is in sync with the three tickets and passes all online tests. It was derived from the latest STEAM version: /online/collisions/2015/25ns14e33/v4.4/HLT/V2
• My online version /cdaq/test/tomei/Run2015/4.4/V2/HLT/V6 added the HLT prescales of https://its.cern.ch/jira/browse/CMSHLT-625.
• Still need to pass the L1 prescales to the L1 doc.

Updates 2015-10-19

Monday

Plan for the day is to make the high pu run asap (most likely will happen in the evening or tomorrow)... unfortunately this still did not happen. No beam all day.

Tuesday

Beam at around 4 in the morning , 600b but NOT the high PU one. Still, following what I thought was the plan (first the high PU fill, then the others) chose to NOT deploy the 4.4 menu, and went with the "HighPU" version prepared by Simone (still based on the 4.3). Mistake in the L1 prescales for the ZeroBias seed led to 2kHz of HLT_ZeroBias (it should have been adjusted for 600 bunches instead of 1800 bunches). I contacted DAQ shifter but they said it was fine.

LHC changed plans at the Daily Run meeting, and decided to go with the 1600 bunches fill before the HighPU. Prepared and deployed the v4.4 menu + four tickets that came later: CMSHLT-625, 626, 627, 628. Version deployed is /cdaq/physics/Run2015/25ns14e33/v4.4.1/HLT/V1. Full details in this ELOG: http://cmsonline.cern.ch/cms-elog/893029

Also, in a hurry I told the Trigger Shifter to start in Column 4 (1600 bunches so I was expecting lumi slightly above 3.5E33). Turns out that it was overconservative, she gave me a call at lunch time and I told her that she could follow the L1 Shifter Instructions and move to higher columns as the luminosity turned out to be less (we started at 3.3E33 I think).

As per TSG Coordination request, I will be rederiving the "HighPU" from the 4.4.

Finally: HLT_DoubleMediumIsoPFTau35_Trk1_eta2p1_Reg and HLT_AK8DiPFJet250_200_TrimMass30_BTagCSV0p45 are consistently below the rate predictions. This may be related to new filling scheme / new menu, but I add it here for the record.

Rederivation of the "HighPU" from 4.4: /cdaq/test/tomei/Run2015/4.4/V2/HLT/V17

Rederivation successful, new version in /cdaq/test/tomei/Run2015/4.4/V2/HLT/V17 but fails "numberOfPaths" because has 501 paths.

Sit together with L1 DOC and figured out the correct ZeroBias prescales for the 600 and 1600 bunches for the HighPU runs. New GT-RS keys made:

• For 1608 bunches : gtrs_2015_25ns_1608b_v1
• For 600 bunches : gtrs_2015_25ns_600b_v1
• Corresponding GT key is : gt_2015_Oct20_0x1043

LHC decided to do ~1800 bunches in the night, so moved back the GT-RS key to a 1825 one. Elog: http://cmsonline.cern.ch/cms-elog/893081 . Leaving for the night.

Wednesday

Ran nicely during the night with the 4.4.1 menu + 1825b GT-RS key.

New gospel, should be added to the Twiki:

In the daily, some discussion about "wrong beamspot in the online database". Question rendered moot as we started another run during the meeting.

Discussed with DAQ DOC on how to test the multithreading safely. Chose to deploy it in one BU appliance; we will ssh into that and check that it is going okay.

New 8b4e filling pattern:

• 25ns8b4e_529b_517_279_444_112bpi8inj (short fill)
• 25ns8b4e_1593b_1581_828_1327_112bpi17inj(long fill)

HighPU additions to the menu: /cdaq/test/tomei/Run2015/4.4/V2/HLT/V19. Four streams of ZeroBias, one PD per stream, one path per PD, one output module / endpath per stream. The paths are prescaled to give 1 kHz in each stream. I remembered to add the offsets. Version saved as /cdaq/physics/Run2015/25ns14e33/v4.4.2/HLT/V1

New keys with correct ZeroBias prescales

• For 25ns8b4e_1593b_1581_828_1327_112bpi17inj
  • TSC key : TSC_20151020_003550_collisions_BASE
  • GTRS : gtrs_2015_25ns_1608b_v6
• For 25ns8b4e_529b_517_279_444_112bpi8inj
  • TSC key : TSC_20151021_003553_collisions_BASE
  • GTRS : gtrs_2015_25ns_529b_v1

New filling pattern for the afternoon:

• 25ns_2041b_2029_1666_1710_144bpi17inj_sp

Questions to us:

• L1 ALways true flag when single beam is circulating: is it a problem for HLT? Why do we use it anyway? (From L1 DOC)
  • It is always true, the thing is that for the isolated bunch it has to be masked
• HighPU runs: is the HLT event size a problem? (From Silvia)
  • Shouldn't be, for ZeroBias and L1 accept events are smaller.
• The suggested columns seem wrong. Check this: http://cmsonline.cern.ch/cms-elog/893160 , 05:31 entry. Level 1 rate is around 80k, for ~ 850 Hz in Physics streams, even at a lumi of 3.8E33, which would correspond to column 4. (From Trigger Shifter)
  • No, for we need some contingency (we would be VERY close to the ~95 kHz L1), and above that we would incur deadtime.

Update, they moved to do the 600b 8b4e in the night. Changing the menu again, version 4.4.2/V2

Data taken correctly, but turns out I was overconservative with the prescales AND I was expecting too much luminosity. Got only 600Hz of HLT_Physics as opposed to 1 kHz.

Prepared 4.4.3/V3, for the 1600b 8b4e in the middle of the night. LHC lost the beams at 5AM, during Flat Top.

Thursday

No beams all day. Request from DAQ DOC and TRG COORD to hurry up with the migration of cosmics to 7_4_15. Some problems, email to FOG sent. Tonight we have 1825b, everything is set. Collisions key is l1_hlt_collisions2015/v309

Spent some time with Michele and Alberto trying to figure out what is happening with the paths. Right now I have a double hunch:

• Something wrong with HCAL, the MET paths are completely underperforming in run 259721
• Something wrong with PIXL, that would explain the step functions in run 259636

Will bug the HCAL and PIXL DOC tonight.

Friday

Tonight's fill was actually okay. Only three emails, all related to paths higher than expected (deviating from linearity).

Talked with PIXL DOC, he confirmed that 259636 indeed had a part of the pixel HV off. That solves one problem (run 259636).

Bugged the DQM shifter, HE still shows orange and yellow, and I am still told that it is normal. Run 259721 still not understood.

All menus migrated to 7_4_15, gave go-ahead to multithreading update.

Had some complaints about high rates with 259736. Traced it to two seeds, L1_SingleJet36 and 68. Already bugged Jim.

Will have 2000b tonight, coordinated with Jim.

And already changed... back to 1825b.

Run 259861 is going smooth.

Saturday

Beam around 18:30. Some problems with L1 Muon seeds due to CSC tests during stable beams [???]. Fine after that.

Beams dumped at 20:48.

Sunday

Magnet rumping down, Sam et. al. preparing the B=0T menu.

-- ThiagoTomei - 2015-10-14