%CERTIFY%

InstructionsPranavComments

Overall picture

The CALO-JET-MET expert shifter is the person on-call to deal with any problems in the trigger that come up during running at Point 1. The shifter is the first point of contact between Online Operations and CALO-JET-MET Trigger groups. In addition the shifter is responsible for marking data quality defects and signing off on runs. Via a mattermost channel and reports at meetings, the shifter also informs the HLT CALO, JET and MET groups about the status of the triggers.

Hi Pranav: this is how you can enter a comment

NOTE: This is a 7 day shift, TUES to TUES, including weekends.

NOTE: Shifters should be in the same/similar timezone as CERN.

How to book your shift

Shifters need to attend a Jet/MET/Calo training session beforehand. Get in touch with the signature coordinators in case interested.

Shifters may book their shifts via OTP. After login, please

1. click on "Book my shifts" and select "Slice experts on-call shifts" (click on the ID 571366 on the left)
2. click on "Task 529788 - Jets, MET and CALO Slices Expert On-Call" to show the calendar
3. make a rectangular selection over the weeks that have been assigned to you and press the "Save Shift Booking" button.

To have an idea about the shift coverage, you can click on "Browse", then select the "Trigger" tab. Click on the cell at the crossing between "Trigger Operation" and "General Tasks" to find a list in which "Task 529788 - Jets, MET and CALO Slices Expert On-Call" appears. Click on this item to check shifts.

Specific Instructions and duties:

Also see the Jet/MET/Calo training slides.

A shifter should do and sign up for the following:

• join the Jet/Met/Calo mattermost channel - you may have to join the ATLAS Software mattermost channel first (click here to join)
  • All communication during a shift to happen there!
• Meet with the previous shifter one day prior to the first day of shift to discuss ongoing issues.
• When contacted by Point 1, or when noticing problems in looking at histograms (see data quality below) the shifter should track down the sources of problems, with expert help when needed
• Attend the following meetings:
  • Daily
    • 09:00 (or exceptionally at 8:30): The daily Trigger operations meeting. You have to give a oral report if there are problems in this meeting. Attendance is mandatory. Indico link
    • 09:30 (or exceptionally at 09:00): The daily run meeting. The day run plan is discussed at this meeting. Indico link
  • Weekly
    • Tuesday 09:30, the weekly run meeting is held in lieu of the daily run meeting. This is a longer report of the past week and plan for the coming week. Indico link
    • Monday 14:00: The weekly CALO slice meeting: Indico
    • Monday 15:15: The weekly JET slice meeting: Indico
    • Monday 16:00: The weekly MET slice meeting: Indico
• Sign up for the following hypernews lists - use the following link, search up the full name of the hypernews list, and Subscribe to the lists: https://e-groups.cern.ch/e-groups/EgroupsSearchForm.do
  • atlas-dataQuality-automatic-notifications@cernNOSPAMPLEASE.ch sends notifications of "DQ runs to consider for <today>".
    • Please pay particular attention to lines like: Please review them before 4pm today
  • atlas-trigger-operation@cernNOSPAMPLEASE.ch is the "ATLAS Trigger Operation Communication" e-groups.
  • atlas-trigger@cernNOSPAMPLEASE.ch is the general trigger e-group
  • atlas-trig-hlt-calo@cernNOSPAMPLEASE.ch is the "ATLAS HLT CALO Trigger" e-group
  • atlas-trig-jet@cernNOSPAMPLEASE.ch is the "ATLAS JET Trigger" e-group
  • atlas-trig-met@cernNOSPAMPLEASE.ch is the "ATLAS MET Trigger" e-groups
  • atlas-trig-HLTJetMetCalo-elog@cernNOSPAMPLEASE.ch for all e-log entries related to CALO/Jet/MET slices
  • atlas-trigger-offline-monitoring-shifter@cernNOSPAMPLEASE.ch for trigger Monitoring and DQ
• Mark data quality on runs.
  • A daily message (sign up for e-group atlas-dataQuality-automatic-notifications@cernNOSPAMPLEASE.ch) will list the runs that need to be checked.
  • The shifter must enter defects for the HLT CALO & JET & MET Trigger slices and sign off on all the runs from the automatic DQ email (bullet above).
  • The trigger DQ/Debug expert will open a JIRA for the daily sign off. The shifter must report the sign off on this JIRA with the entered defects! This should be done by 15:00 every day.
  • A list of all the open DQ sign off and bulk processing jira tickets here.
  • See the detailed instructions below.
• On-call signature experts may be asked to check a new trigger menu/production cache and report their findings at the trigger operations meeting. Such requests are announced in the trigger operations meeting and made in JIRA. When requested to do so, report on DQ/monitoring and HLT cache tests and reprocessing exercises in the corresponding JIRA ticket.
• Document what is happening in the runs [eg. trigger rates, primary chains, etc] and any other relevant information in an email (send to the HLTJetMetCalo mailing list) and reports in the Jet and MET trigger meetings.

• see also TriggerExpertRules

Important links for Run 3

DQMD OHP Tier0 .

How to Check the CALO/Jet/Met trigger DQ

The customary means to finding information to consider in order to set the TRJET, TRMET and TRCAL DQ flags explained in this section.

Jets:

1. Go to the DQ Web display. For cosmics, tick "Include non-stable-beam runs"
2. Select the run and the stream you want to look (typically "express_express");
3. Click "Entire Run";
4. Expand "HLT". Expand "TRJET".
   This will display the jet trigger offline histograms, made for events in the express stream. Note that this includes only some triggers so that the distributions are biased. To find the triggers in this stream, go to the run query page and click on the prescale keys to reach the trigconf page.
5. Start by looking at the shifter folder.
   The shifter folder should be enough to give you a good idea if there are problems and possible causes (for instance disentangling if issues observed are due to noisy channels or to a more serious problem).
   There are sub-folders for jets that passed specific jet signatures (j25, j360) and others that contain all the reconstructed HLT jets (a4tcemsubjesFS). Check also the L1 distribution to disentangle if problems might be caused by L1.
6. You can better understand the origin of problems or ensure that everything is fine by looking at the expert folder.
   • Check the L1 experts folder to understand if problems may originate at L1.
   • Look at the single jet main signatures: do they look as expected? Do they show the same features as observed in j25 and j360?
   • Check that the containers with different calibrations are coherent with one another.
   • Check the a10 folders (a10tcemsubFS, j460_a10_sub_L1J100): do the distributions look as expected? Do they show the same features observed in the a4 signatures?
   • Check the triggers with eta cuts.
   • Same for the remaining folders.
7. If any issue is found, check the online histograms to have a different point of view/more information and to correlate what is seen.
8. From early 2018, some updates are made in jet triggers to have less histograms to check. Some histograms show empty because the web display is not updated consistently yet.

MET:

1. You are responsible to keep an eye on all the histograms in the Shifter folder.
2. The Shifter/Component histograms are to check the input to MET calculations. Problems in these histograms are indications of problems of calorimeter hardware. If you see something wrong, please contact experts.
   (HLT_MET_status) Normal: GlobalError bin is empty (not filled). Abnormal But Not Critical: 1. Bytestream conversion errors in some components (TileBar, Gap1,2,3, Ext1,2). 2. BadCellQuality in one or more components. Abnormal and Critical: Global Error bit is set GlobError or Missing Components - Cross check with compN_HLT_MET_status.
   (compN_HLT_MET_status) Normal: Processed Bit must always be set (and filled). Abnormal But Not Critical: 1. Bytestream conversion errors in some components (TileBar, Gap1,2,3, Ext1,2). 2. BadCellQuality in one or more components. Abnormal and Critical: Global Error bit is set GlobError or Missing components or Error in one or more components CompError.
   (compN_compEt) Normal: All components (X bins) are filled. Abnormal: One or more components are empty.
3. The Shifter/L1_ folders* contain basic histograms for the current L1 triggers. Currently L1_roi, L1_jFex and L1_gFexJwoj are filled.
   (Ex and Ey) Normal: No spikes, with possibly a single peak structure at zero. Abnormal: Spike or the peak shifted from zero, shifter should contact expert.
   (Et) Normal: No spikes, with possibly a single peak structure at low MET and a falling distribution. Abnormal: Spike in MET, shifter should contact expert.
   (sumEt) Normal: No spikes, with possibly a single peak structure at low sumEt and a falling distribution. Abnormal: Spike in sumEt, shifter should contact expert.
   (log plots) Should be able to see distribution changes clearer than corresponding linear plot.
4. The Shifter/cell, pfopufit, pfsum_cssk, pfsu_vssk, tcpufit, nn folders contain basic histograms for the current HLT triggers.
   (Ex and Ey) Normal: No spikes, with possibly a single peak structure at zero. Abnormal: Spike or the peak shifted from zero, shifter should contact expert.
   (Et) Normal: No spikes, with possibly a single peak structure at low MET and a falling distribution. Abnormal: Spike in MET, shifter should contact expert.
   (sumEt) Normal: No spikes, with possibly a single peak structure at low sumEt and a falling distribution. Abnormal: Spike in sumEt, shifter should contact expert.
   (log plots) Should be able to see distribution changes clearer than corresponding linear plot.
   (phi) Normal: No spikes. There could be a wavy structure. Abnormal: Spike, shifter should contact expert.
   (phi_etweight) Normal: No spikes. There could be a wavy structure. Abnormal: Spike, shifter should contact expert. When HLT_MET_phi show peaks, this plot is a good check if the peaks are problematic or not. If this plot has no peak, it means the peaks in HLT_MET_phi are from low energy hot sopts and has no real harm. When HLT_MET_phi does not have peaks, it is all right even if this plot show peaks because those peaks are due to a few very high pt jets.
   (eta_phi) Particular attention should be paid in checking hot regions. The distribution should not show peaks. Check if the same peak can be found in Calo or Jets.
   (eta_phi_etweight) Particular attention should be paid in checking hot regions. When HLT_MET_etaphi show peaks, this plot is a good check if the peaks are problematic or not. If this plot has no peak, it means the peaks in HLT_MET_etaphi are from low energy hot sopts and has no real harm. When HLT_MET_etaphi does not have peaks, it is all right even if this plot show peaks because those peaks are due to a few very high pt jets.
5. The Shifter/preSel contains histograms with L1XE50 preselection.
6. The Shifter/eff folder contains efficiency histograms for some trigger chains. Since those trigger chains change depending on the pileup, some chains may be missing. You may find them in Expert/eff folder.
   (all plots) Normal: Efficiency smoohtly reach 1 unless low statistics. Abnormal: Efficiency doesnot reach 1 shifter should check prescale. HLT chains are often with low stats and do not exactly reach 1 at high MET.
7. The Shifter/SignalEl folder contains histograms for selected events with signal-like electrons. The Shifter/SignalMu folder contains histograms for selected events with signal-like muons. These histograms are usually free from the lower energy activities such as hot spots. When you see lower efficiency or strange structure due to those low energy activities, please check these histograms. If the problem does not exist here, you can conclude that there is no data quality problem.
8. (This is not yet ready) The Shifter/mu50 folder contains events with pileup 47 < mu < 53. These histograms should be stable against pileup changes due to luminosity or bunch structure changes. If these histograms are empty, check the Shifter/Component/act_IPBC histogram to see if mu=50 is populated or not.
9. Plot *_phi shows a modulation in phi which varies from run to run and is not relevant (doesn't necessarily agree with the reference).
10. Plot *_etaphi_etweight may show some hotspots which are only relevant if peaks can be seen in plot *_etaphi; otherwise they are spurious spikes caused by a known effect of a few high energy jets. Check CaloMonitoring/CaloMonShift/CaloMonBAR/EMTopoClustersBAR if you see a hot spot in the opposite direction.
11. The *_etweight plots are often red when all others are green. This is not a problem as far as phi_etweight agreement is reasonable.
12. The Expert/Offline contains the histograms for the offline MET used as an reference for efficiency histograms. If you see some changes in efficiency histograms, check if they are coming from offline MET changes.
13. The Expert/HLT_ElMu contains the histograms for the HLT electrons and muons used to select signa-like events in Shifter/SignalEl and Shifter/SignalMu histograms. If you see changes in those signal-like histograms, check if the cause is the changes in HLT electrons or muons.

NOTE
The wavy structure in phi changes often. It depends on the position of beam collision. It moves from time to time.
The structure in eta is rather stable but details change often. Usually forward region in the calorimeter gets more energy deposit than central region, so it has peaks at high eta.

Hot to deal with hot spots or phi spikes for DQ sign off
There are 3 types of hot spots (phi spikes):
(1) Low energy hot spots
Many low energy activities due to noisy cells or other noise like problems.
You'll see hot spots in nominal eta-phi plots, but no corresponding hot spots in Et weighted eta-phi plots because they are low energy activities. Also, you don't see hot spots (phi spikes) in SignalEl and SignalMu nominal eta-phi histograms.
These hot spots (phi spikes) has no impact on the qualify of the data. Assign phi spike defect and sign off.
(2) A few miss-reconstructed energies
You don't see spikes in nominal eta-phi plots but spikes in Et weighted eta-phi plots.
Look at the Et weighted phi plot. You can tell that if only one of two events are causing the phi spike or many events are in the spike from the size of the error bar. If the number of events causing the phi peak is one or two events, these are likely miss-reconstructed high energy objects (jets). Usually only one or two extremely high pt jets.
This type of extreme objects are removed by event cleaning in physics analysis, so there is no problem in the data quality. Assign phi spike defect and sign off.
(3) High energy hot spots
You see hot spots both in nominal plots and Et weighted plots. From the Et weighted phi plot, you can tell many events are in the hot spot.
Look at physics_CosmicCalo stream to see if there is the same hot spot in this stream.
(a) If you find the same hot spot in the physics_CosmicCalo stream
The hot sot is coming from LAr noisy cell(s). These noisy cells are properly treated by LAr or CaloCombined people. So, you don't have to worry about them. Assign phi spike defect and sign off.
(b) If there is no hot spot at the same location in he physics_CosmicCalo stream
The hot spot may not be from LAr. Assign phi spike defect and sign off, and wait for BULK Phyiscs_Main.
If you see the hot spot in Physics_Main, you need to start investigate the source of the problem.
NOTE How to investigate is not yet determined but coming soon. For now, just contact MET signature coordinators.
One place to look at is CaloMonitoring/CaloMonShift/CaloMonBAR/EMTopoClustersBAR to see a hot spot in the opposite direction.

HLTCalo:

• There are four sets of plots, named
  • HLT_FastCaloEMClusters (related to fastCalo step of e/gamma signature)
  • HLT_TopoCaloClustersFS (related to topclusters in jet/met signature)
  • HLT_TopoCaloClustersLC (related to topclusters in tau signature)
  • HLT_TopoCaloClustersRoI (related to topclusters in precisionCalo step in e/gamma signature)
• Signing off Reprocessing:In general you will have a histogram lying on a reference (from previously signed off reprocessing). Luckily the algorithm used for HLT_Calo are quite stable. There in general very rare changes between nightlies. However, we do see changes often in histograms wrt reference due to changes in client signatures. E.g.
  • HLT_FastCaloEMClusters & HLT_TopoCaloClustersRoI : All changes to these plots come from changes introduced in e/gamma signatures. These changes in e/gamm are generally in HLT level but sometimes can go to L1 level where RoI sizes have been modified. E.g.
    • Changes in Entries to histograms wrt reference: mostly due to addition or removal of e/gamma items in menu. You can track these changes going to the "diff" link of a reprocessing. In general menu related changes show discrepancy in entries of histograms but the shape of the histograms remain same. * Changes in shape of the histograms can be introduced by change in RoI (eta/phi limit) in L1 input. This can also be tracked from the diff. * Few more changes other than shape/entries can come directly from HLT_Calo algorithm if some condition-data or calibration constants change * HLT_TopoCaloClustersFS: All changes to these plots come from changes introduced in jet/met signatures. The process to debug these changes are similar as mentioned above for HLT_FastCaloEMClusters & HLT_TopoCaloClustersRoI but you need to look for changes in jet/met signature menu. And again "diff" will be the place to look for the changes. Just a reminder jet/met are not reconstructed based on RoI they are full scan topoclustering (different than e/gamma and tau which are RoI based reconstruction)
  • HLT_TopoCaloClustersLC: All changes to these plots come from changes introduced in tau signatures. Finding the source of changes is similar to above mention methods, all you need to do is to concentrate on tau signature.
• As you see that HLT_Calo is quite dependent on the client signatures it is always a good strategy to wait for the client signatures to sign off first (but don't wait too long if you understand the bugs) then you can better strategize your method.
• Last but not least sometimes histograms can be wrongly flagged red which is why its better to double-check testing locally (with TH1F:Chi2Test/Kolmogorovtest). I am developing a code (will be attached soon) that should help

Pathological cases:

This page: JetMetCaloHLTPatologicalCases is intended as a catalogue of interesting pathological cases found in Jet/MET/HLTCalo monitoring, which can be useful information for other experts.

Known issues

How to fill the MET spreadsheet daily

The information about the peak luminosity, pile-up, integrated lumi and no. of events can be found via the run query webpage. For archiving the peak HLT_xe rates you can load the run in TRP, plot the relevant chains and note down the rates at beginning of the run. Another possibility is to go to Trigger Cost Browser, choose the right run, select the first lumi block (or HLT keys for the beginning of stable beams) and then 'HLT Chain Summary'. There you'll find the 'Execute rate' (=input rate) and the 'HLT pass fraction'.

There is also a nice tool for displaying the rates from recent runs, called xmon. Please display the rate vs. for recent runs in the summary at the end of the week.

How to check archived online histograms

To view online expert histograms for the ATLAS partition in a web browser, click on this link or MDA for archived online histograms. Archived histograms can also be found on /eos/atlas/atlascerngroupdisk/tdaq-mon/coca/ or COCA.

If you are given a specific reference CERN JIRA ticket to look through, find the "Web display" section, and click on the online link. This should take you to the web display for the various histograms that the JIRA ticket contains.

How to check the rates of the relevant CALO/Jet/MET HLT chains

You can also check the actual rates at P1 with the online version of the Trigger Rate Presenter here, where you can see the input/output rates and the prescale of each trigger. In the top part of this site you will find links to the plots of rate vs. time for the different chains. Clicking on the plot will lead you to the trigger menu for that lumi block.

To check live L1 rates: CTP web monitoring.

Rates are archived on MDA.

For live monitoring, it is possible to use the TRP either directly at P1 or via ssh (connection details above) with, after setup_TDAQ.sh, trpgui -p ATLAS -c /atlas/moncfg/tdaq-05-04-00/trigger/trp/trp_gui_conf.xml

How to signing off on runs and fill the defects database

The Defects web page is https://atlasdqm.cern.ch/defectentry/

• Choose Database 'Production', Tag 'HEAD'
• Contact Sebastian Prince/Antonia Strubig/Aparajita Dattagupta for the magic key word for signing off

To sign off a run:

1. Choose System 'TRIG_HLT_MET'
2. click on 'SIGN OFF RUN'
3. choose System 'TRIG_HLT_JET'
4. click on 'SIGN OFF RUN'
5. choose System 'TRIG_HLT_CALO'
6. click on 'SIGN OFF RUN'

after the sign off, the TRIG_HLT_*_UNCHECKED defect should disappear for the run. The same rule applies for BULK sign off (where defects have *_BULK_UNCHECKED label).

* The signoff of the runs has to be done for every run for which the CALO or JET or MET trigger is ok.

To specify defects:

• Select tab 'Upload'
• select TRIG -> TRIG_HLT -> TRIG_HLT_MET or TRIG_HLT_JET or TRIG_HLT_CALO
• tick the defect box(es)
• specify the run and luminosity block
• put in a comment
• put in your password and click 'Submit'

You can change the defect by doing the same thing and saying that the defect is absent.

More information about defects can be found on the TriggerDQFlagsRecipe page.

NOTE: The TRIG_HLT_JET_LOWSTAT flag should be applied when something seems wrong but the statistics are low, this flag should not be used to flag a good run that had lower statistics.

After setting the defects you should report it on the JIRA opened by the DQ/Debug expert for the daily sign off before 3PM!

This way the shifter is informed and can report in the daily DQ meeting at 4PM. The report should include the set defects (best with link to DQ display) and if the run can be signed off.

List of tolerable and intolerable defects

A list of all the existing defects with a description and information if a defect is tolerable or intolerable can be found on the DataQualityTriggerDefects page.

Please note, that if you put a intolerable defect, the run will be excluded from any GRL. So please check with the signature coordinators before setting an intolerable defect.

How to sign off on HLT Reprocessings

You may be asked to check a new Athena release in an HLT reprocessing on recorded data. QUESTION: I think shifters will be tagged directly in an HLT reprocessing in future?

• • Find a sample report at the ATR-25619: HLT Reprocessing for validation

Jet Sign-off

See the instructions on the Jet/MET/Calo training agenda.

How to summarize your shift

• Make an orderly transition to the next shifter:
  • Meet the next shifter (via mattermost or organise a skype chat) and relay any important information.

• Summarize your shift and the encountered problems
  • Prepare a report in form of slides or minutes with a summary of the week. The report should be presented in the Jet and the MET trigger meeting.
  • Post a summary report to the mattermost channel.
  • At the ATR-25619: HLT Reprocessing for validation site, you can find a report that has an example of "pinning" merge requests to help the readers find possible merge requests that could have caused the encountered problems.
  • These merge requests should be found under the subheading "Reference release". For example, in the JIRA ticket linked above (ATR-25619), the merge request site can be found on the following line: "Reference release: Athena...diff(master,16->23), diff(22.0,23->24)" Is this correct or is this specific to this JIRA ticket? If so, how would people go about finding the possible problematic merge requests?

Useful information

• Use the ATLAS e-log to get more information on possible problems you might see.
• To know who is in the control room and what are the phone numbers click here
• On-call experts phone numbers can be found on https://pptevm.cern.ch/mao/ui/TriggerList.html
• Lots of useful info from AtlasMetSliceOnCallExpert2010 is still valid
• Lots of useful info from JetSliceOnCallExpert is still valid
• Lots of useful info from HLTCaloExpertShifter is still valid
• Documentation for the Tier0 Jet Slice Histograms is here (but needs updating) - JetSliceTier0Histograms
• Documentation for the Tier0 MET Slice Histograms is here (but needs updating) - MetSliceTier0Histograms
• Information about the jet trigger chain naming scheme: Jet trigger note

CALO/JET/MET slice experts by subject

• MET software: Jon Burr
• MET menus: Jon Burr, Ben Carlson
• MET monitoring: Kenji Hamano, Charlie Chen
  

• JET software: Peter Sherwood, Claire Antel
• JET menus: Ayana Arce, Claire Antel
• JET monitoring: Ayana Arce, Peter Sherwood

• CALO: Denis Damazio

• L1: Ralf Gugel, Rhys Owen

Obsolete Information

https://twiki.cern.ch/twiki/bin/view/Atlas/CaloJetMetExpertShiftInstructionsObsolete

-- AyanaHollowayArce - 2022-06-16