LAr/Tile/Forward Detectors Whiteboard

Daily Plan and important messages from the RC

Shifters
  • At the start of each shift ensure you read the "Current Status" of the three sub-detectors. Known MRS/DCS warnings/errors are printed there.
  • Click here to see who is in shift in ACR at the moment. Clicking on the shifter's name, you can also see who are the next shifters.
  • Please report bugs or missing documentation on bug report page
  • Post suggestions or discussions on bug report page(new category)

LAr Calorimeter

  • Calibration period (experts are working with calibration)
  • 17:00 Join to combined run for data tacking
    • Run Parameters for these runs: samples: 5 latency: 94 first sample: 2 format: Format1/Results
  • Take LAr calibration (Daily/Weekly) according to available time (1h/2h). Please use LAr Calibration (beta) panel.
  • Notes:
    • If there are HV trips, shifters should mask them from the FSM Alarm Screen (later, the HW on-call will check the masked alarms)
  • Shifters please always study the whiteboard and check ATLAS and LHC programs (yellow box on the right).
  • Items to follow-up (experts only): Pt1:LArRunCoordination GPN:LArRunCoordination

Link to LAr Monitoring Spy

Tile Calorimeter
  • Notes: If a big part of the detector is OFF call immediately to the *Tile DCS on-call* and to the *Tile Emergency Phone*.

Call immediately to the *Tile DAQ on-call* and to the *Tile Emergency Phone* if you see the following error/alarm messages in MRS:

    • TileXXX_RODRCD rc:HardwareError TileXXX_RODX_ChanX_XXX-XXX_ROLXX TileXXX_RODRCD
    • TileXXX_RODRCD TileRCD::Issue TileXXX_RODX_ChanX_XXX-XXX_ROLXX has been disabled by TileCalXXX_RODModule_X and check with the Run Control shifter if he did the stopless removal

Until this message is deleted:
LBC41 sometimes does not recover properly after a trip (giving high fraction of digital errors). If this happen, try one power-cycle (calling DCS on-call). If this does not cure the problem DO NOT try to power-cycle again, but report in the elog for the Summary of run. Usually the module recovers itself at a new start of run. Check again the module at the next start of run and report in the elog if the module was recovered.

Until this message is deleted:
Do NOT submit one elog for every trip. Trips should be reported in the run summary elog and if there were particular problems with trip recovery, this information should also apper in the run summary elog.

From May 23, until this message is deleted:
Shifter should NOT log in into FSM panel, the ERRORs after a trip will be cleared automatically.

Until this message is deleted:
Do NOT call Tile DCS expert if shift leader sees "LHC TILE CurrentData no update" on DCS Alarm screen. This is a known problem that Tile shifter is not monitoring (only central DCS). There is people trying to solve this, it does not affect standard readout.

Forward Detectors
(Last Update: 10.10.2011)

  • Run Plan:
    • ALFA: standalone operation ocasionally by experts, usually in shutown state, in general out of the combined run until further notice. Around October 18: 2nd special ALFA run at beta* =90 m, ALFA included in the partition, dedicated trigger menue, latency for all other detectors shifted by 20 BX.
    • ZDC: in global partition (edited on 22.10.2011)
    • LUCID: in global partition.
  • Detector:
    • LUCID: all ON, DCS should be green.. In the unlikely event of LUCID becoming busy, a procedure for stopless removal is started. Tell RunControl shifter to always answer yes to the related pop-up window. This will NOT stop the measurement of luminosity.
    • ZDC: all ON, DCS should be green. If HV or Crates will turn orange/red please call the expert. If Scalers turn orange/red, check via the Log Manager whether there are any messages/errors from the application zdc_publish_scalers in the initial partition. If the problem persists for more than 5 minutes call the expert and indicate all messages you found. Data Quality: ZDC flags in the DQMD (Data Quality Monitoring Display) are under revision so they can be safely ignored until further notice.
    • RPO (Roman pots or ALFA): all OFF, DCS should be green. The default settings for RPO are OFF to minimize the risk of radiation damage of the electronics. DCS alarms should be (i) e-logged (under ALFA/RPO) and (ii) during working hours, reported by telephone to Sebastien Franz, 16-5217 while outside working hours the FD on call should be called. In particular, if the online value of the temperature exceeds 45 deg., call ALFA expert Sune Jakobsen at 16-8853. In case experts plan measurements they will contact the shift leader and LAr shifter in advance. When the measurements are finished or stopped for any reason the system has to be set back to the OFF state.
  • Calibrations:
    • LUCID: calibration ought to be performed once per day with the new automatic procedure LUCID Calibration (Beta). Instructions can be found in the Shifter's Manual
    • ZDC: calibrations ought to be performed twice per week, preferably Monday and Thursday with the new automatic procedure (Menu CAL->CalibrationMenu->LUCID Calibration Beta).
  • e-log policy:
    • When making an e-log entry that concerns ONLY forward detectors, please use the "Default" message type and select the relevant detector(s) as "System affected" (ALFA and/or LUCID and/or ZDC). Do NOT use the "LAr" message type, as this defaults to LAr (+ possibly others) as System Affected, and sends a lot of spam to the LAr experts and coordinators.

Weekly based calibration plan

Combined Calibration Tool Twiki

System Monday Tuesday Wednesday Thursday Friday Saturday Sunday
LAr Weekly(1) Daily - Daily - Daily Weekly
Tile - All - All - -
LUCID(2) Daily Daily Daily Daily Daily Daily Daily
ZDC YES - - YES - - -
L1Calo - - - - YES - -
L1Calo+LAr - - Combined: expert(3) - - - -
L1Calo+Tile YES - - - - - -
Notes:
  • (1) For LAr, take the full or remainder of the weekly set on Monday if it couldn't be completed on Sunday. Plan 1h30 for the full set.
  • (2) For LUCID, the default calibration is called 'Daily'; an attempt should be made to have one calibration everyday in between fills.
  • (3) For L1Calo+LAr combined calibrations, experts will be in charge of taking this set until the panels are fully automatized.
  • (4) For Tile: You can finish them all in less than 1 hour. The last laser calibration is the slowest. If there is not enough time before ATLAS needs TileCal again, take the fast calibrations first, and wait for another opportunity to take the long laser calibration. If ATLAS needs TileCal, stop whatever calibration you are doing, and close the TILE partition. Check if the shifter before you has already taken some of the calibrations.
Tools: these are the tools used for production. Unless the run coordinator or experts tells you otherwise, please use the following tools when a calibration is needed:
  • LAr: LAr Calibration panel, through CAL/Calibration Menu/LAr Calibration(Beta)
  • Tile: Tile Calibration panel, through Tile/Tile Calibration - More info in Tile Calibrations Twiki
  • LUCID:
  • L1Calo+LAr: calibration tool, through CAL/Calibration Menu/L1Calo Calibration), then 'LAr Energy Scan' in the bottom section

Elog Templates

elogShiftCombined.txt: Combined shift summary template
Template for the end of run Tile trip summary (physics)
For LAr HV trips, see ELOG template and instructions here.

Elogs for calibrations can be submitted from the combined Calibration Panel.

Important Phone Numbers

LAr Calorimeter Number
ACR Desk   7 1346  
SCR Desk   7 0949  
Hardware On-Call   70137  
Software On-Call   70128  
LAr Run Coordinator Olivier Simard 70136  
More numbers on LAr/Fwd Operations Page      
       
Tile Calorimeter Number Backup
Emergency Phone   16-2581 +41 76 487 2581
ACR Desk   71408 +41 22 767 1408
SCR Desk   71408 +41 22 767 1408
DAQ on-call Jalal Abdallah 16-0418 Oleg Solovyanov (16-3436)
DCS on-call Claudio Santoni 16-3476 Gabriel Ribeiro (16-9659)
Run Coordinator Paolo Francavilla 16-3155
Deputy Run Coord. Maria Fiascaris 16-1950
important Tile phone numbers      
       
Forward Detectors Number
Run Coordinator Davide Caforio 71122

ATLAS Daily Schedule

Important Links

Current Status of Detectors

LAr Current Status

Tile Current Status

FWD Current Status

LAr Specific Pages

This part should contain the links with the important information for the run coordination, and for the shifters. Maybe it can be divided into three session (different subsystems) + a general list of links

This list should contain the KNOWN issues for each subdetectors, the detailed description of the DCS status and the links for the documentation (twiki, training)

Shift Message Board

This section can be edited by ANYONE to post known or ongoing problems, after posting an entry to the ATLAS elog.


  • Shifters should begin with the Start of Shift Checklist (#2 under CAL/CALChecklists)
  • If you experience a problem and you do not find any instructions on the whiteboard & trouble shooting & shifter manual pages, please call the expert or the LAr run coordinator, and DO NOT try to fix the problem on your own!
  • LAr
    • LAr Busy procedure (modified by Nikiforos on October 19th, 2011):
      • In case LAr goes Busy without an underlying DAQ problem, the busy PU will get automatically disabled and the run control shifter will get just a notification pop-up window. They should inform you after the "stopless removal" that they have done so.
      • Look on the MRS for mesages like "HardwareError" and note the PU name since the RunControl shifter may neglect informing you. Call the LAr Run Coordinator and inform them of the issue.
      • If there is an underlying DAQ problem (Look at MRS for XOFF from the ROS, complains from ROSes and L2PUs etc) the RunControl desk should get a YES/NO window asking them to disable the PU. If they ask you what to do, tell them to click NO and ask them to call the TDAQ on call while you call the LAr Run Coordinator and SW on-call. In principle, RunControl should have initiated an investigation before even asking the LAr Shifter.
      • Instructions are also posted on the Troubleshooting Twiki
    • Comments by Nikiforos (current as of April 26, 2011):
      • Please monitor ROD Crate PS temperatures. Instructions:
        • Look at the FSM (without beeing logged in) under, for example: LAR-> EMB A -> EMB A ROD .
        • Notice the drawing of the 2 Racks with 2 crates in each rack.
        • Click on plot on to toggle display of the temp charts for that particular crate.
        • We are mainly interested in the PS Temperature (slot 6) curve.
        • You can go to other racks by clicking on the Change side entries.
        • If you see the temperature values increasing over ~a few hours/days, please post an elog with a screenshot and indicate which crate is affected.
    • Comments by Stephanie (current as of September 16, 2011):
    • Comments by Stephanie (current as of June 7, 2011):
      • In case of an HV trip, please follow the instructions on the Troubleshooting Twiki.
      • In case of any ERROR or FATAL states for the HV (in the FSM), or if anything seems strange, please call the HW on-call expert at 70137.
    • Comments by L. March (current as of April 26, 2011):
      • NO OPC TEST SERVERS DONE BY SHIFTERS UNTIL FURTHER NOTICE

  • OHP for Forward detectors
    • To get both the LAr & FW detectors OHP plots, this configuration should be set from the DAQ panel:
      • OHP Opt: -c $OHPSEARCHPATH/lar/ohp/CalMonitoringShifter.ohp.xml
      • Follow this link for more details.

  • LUCID
    • Due to a bug in the FSM, the secondary panels (bottom left corner of the FSM window) may not be correctly shown: in order to display them, right-click on the relevant FSM node name (LCD, ZDC or RPO) in the FWD FSM main panel (DC, 12 Mar 11)

  • ZDC
    • ZDC: no message at this time.

DAQ Panel Configuration

Check these values in the corresponding setup items of the DAQ Panel For the OHP configuration, Please synchronize your action with the other CAL shifter, in order to have the Tile OHP configured in one desk, and the LAr OHP configured on the other desk.

Global Setup
Version TDAQ-03-00-01, HLT-16.1.1.1
Setup Script /det/tdaq/scripts/setup_TDAQ.sh
Partition Name ATLAS
Database File /atlas/oks/tdaq-03-00-01/combined/partitions/ATLAS.data.xml
MRS Filter (*TIL*|*Til*|LAR|LAr|Lar|lar|*ZDC*|*Zdc*|*zdc*|*LUCID*|*Lucid*)&(^(*TIL*RODRCD|*Til*RODRCD|*Til*TTCRCD*|*TIL*TTCRCD*))&(ERROR|FATAL|WARNING)
OHP Opt -c /atlas/moncfg/tdaq-03-00-01/tile/ohp/Tile.ohp.xml
OHP Opt LAR
OMD Opt /atlas/moncfg/tdaq-03-00-01/daq/omd/OMD-generic-config.xml
TriP Opt -c /atlas/moncfg/tdaq-03-00-01/trigger/trp/trp_gui_conf.xml
Tile Remote Monitoring Setup
Instructions are in the TileRemoteMonitoring page (outside Point1)

Tile DAQ Configuration

DSP Configuration to be checked at the beginning of a run

DCS Status

CHECK FSM at least once every 30 minutes! If warnings come and go, during day shift call DCS-expert-on-call and make e-log entry, during night shift write e-log and instruct morning shifter to call DCS on-call.

Tile DCS

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng 2011_10_24_fsm.png r1 manage 116.4 K 2011-11-07 - 07:54 PaoloFrancavilla  
PNGpng DSP_Config.png r1 manage 12.4 K 2011-11-07 - 07:46 PaoloFrancavilla  
Texttxt elogShiftCombined.txt r1 manage 7.4 K 2011-11-12 - 15:57 UnknownUser Combined shift summary template
Edit | Attach | Watch | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r7 - 2011-11-14 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback