MDT Shift Manual

Phone Numbers

Introduction

In the commissioning run P3, the MDT system will include Sectors 1-12 from the Barrel, 1-16 sectors of the big wheel sides A and C from the Endcap, the EIL4 chambers of A and C sides, and the small wheel side C.

In order to run the system successfully, it is necessary to ensure that the chambers satisfy the following checks:

• Power Supply

• Gas

• Initialization

• TDAQ Software

• Monitoring (DQMF and GNAM)

Power Supply

The MDTs require Low Voltage (5V) for the electronics and High Voltage (3080V) that is applied to the tubes. These are supplied by power modules located in miniracks in UX15 and controlled from USA15. A PVSS interface integrated into the general ATLAS DCS controls and monitors the power supply remotely, recording the history of changes in voltage and current.

From the control room, use the DCS GUI that may be accessed directly from the DCS Menu:

DCS DCS FSM Screen

A link is also available from the Muons Menu:

Muons MDT DCS MDT DCS FSM

The system that will be used in the final configuration is the DCS FSM running on the machine cerntsatldcs01.cern.ch. If you select DCS FSM Screen, you should be connected remotely to the relevant machine.

If you are prompted to login to the remote machine and do not have an account, you can access the information about power supplies using:

Username: mdt-ps

Password: xxxxxx

Then select the FSM application and start the GUI.

The root account allows the user to view the status. However, in order to make changes, it is necessary to use the individual login and password (same as Point1 account). In the top left section of the GUI, a tree is displayed. Select MDT and then Barrel A, Barrel C, Endcap A, or Endcap C.

Turning ON Low Voltage

In P3 the LV is available for Sectors 1-12 in the Barrel A and C, all sectors of the Endcap A and C, all EIL4 chambers, and the small wheel C.

In the DCS GUI, there is a color scheme for the STATUS and the STATE:

The LV can be turned on by changing the state for the whole subsection next to the name of the subsection on the tree on the top left part of the GUI (this changes the state for all chambers in that subsection, so only chambers that should be modified should be included). Alternatively, the chamber state can be modified on an individual basis. The trip current for the low voltage should be set to 15A.

Turning ON High Voltage

Before turning on the HV, ensure that the gas situation is satisfactory. Please see the gas section in this manual for more detailed information.

Check with Joerg (Barrel) and Jainbei (Endcap) about the gas status of the MDT chambers.

If the HV is off and the chambers have adequate gas, the HV can be turned on using the DCS GUI. The trip current should be raised to A while turning on the HV. Once the full voltage is reached, the trip current should be lowered to A, and checks should be made to ensure that the current is close to zero.

In case the DCS GUI does not work, it is possible to access the mainframe directly from a terminal with the command:

muondaq@pc-atlas-cr-06> telnet 10.145.48.242 1527

When prompted to log in, use:

Login: xxxx

Password: xxxxx

Use the tab button to open the Menu and select channels. Then, scroll down (use the up/down arrows) to the desired module and channels and turn them on/off using the space bar. To exit, use the tab button to open the Menu again and select Log Out. It is advisable to contact an expert who can give directions on how to use the terminal interface.

Gas

The MDT chambers operate with a mixture of Argon (93%) and CO (7%) at a pressure of 3 bar. The gas is monitored in racks in UX15 that are connected directly to all the chambers, and these racks are then connected to the broader gas system. There is a PVSS interface in order to control and monitor the gas system remotely. This has been integrated into the general ATLAS DCS and may be accessed in the Control Room under the Muon Menu:

Muons MDT DCS MDT DCS FSM

It is important to keep under control chamber pressures, and gas inflow/outflow rates.

Initialization

The MDT chambers are initialized via DCS. The configuration parameters are read from the configurations database and input into a JTAG string that is sent out to the chambers. The chambers should be initialized once the electronics have power (Low Voltage ON, see previous section on power supply).

Nine PCs are used for initialization (4 for the barrel, 4 for the endcap and 1 spare). These PCs are called pcatlmdtmdm1 to pcatlmdtmdm9. These machines may be accessed remotely (in the control room, there is a link from the Muons menu) and the login for these machines is:

Username: mdmdcs

Password: xxxxxxx

NOTE: xxxxxx is SHIFT$+xxxxxxx$

The PCs work in parallel (so chambers may be initialized simultaneously from different PCs), but there is no single interface to all the PCs yet (so the initialization process should be initialized from all the PCs connected to the desired set of chambers). The chambers used in P3 that are connected to each one of the PCs are:

• Barrel Side A and C: pcatlmdtmdm1 to pcatlmdtmdm4 (sectors 3-10 A and C sides)

• Endcap Side A: pcatlmdtmdm5 (sectors 1-16 )

• Endcap Side C and EIL4 Side A: pcatlmdtmdm6 (sectors 1-16 )

• EIL4 Side C: pcatlmdtmdm7

Once logged into the relevant PC, the DCS initialization panel should be displayed. It may be necessary to select the FE control panel. Then select all the sectors and sides to be initialized, and click Init All. If the initialization fails, click Init Failed and this command will only initialize chambers that are in the FAILED state. It is possible to initialize a single chamber by selecting it and then clicking on Initialize. The state will be updated on the general panel.

In the case of the Barrel, an additional check on the initialization may be performed in USA15 . Once a chamber has been initialized, a green LED should appear next to the corresponding MROD channel (top six connectors of the MROD boards in USA15). The correspondance between the chambers and the MROD channels are described in the manual The MROD data format and the tower partitioning of the MDT chambers, available here. In the case of the Endcap, the green LEDs only turn on in the Configuration step of data acquisition.

For barrel sectors 3-6, the MRODs used are located in the top and middle crates of the first and third racks (counting from the entrance to USA15). The first two inputs from the top are the BI chambers, the middle two BM chambers and the bottom two the BO chambers.

TDAQ Software

The Trigger and Data Acquisition (TDAQ) system allows for taking data and selecting events using the trigger. During the commissioning runs, the shifter must configure, control and monitor the MDT system, and these tasks are carried out using the online software.

A feature of the system is that it is partitionable, meaning that it can be sub-divided into independent but fully-functional units. The MDTs can thus take data independently and in parallel with other sub-detectors (standalone mode) or combined with other parts of the detector (combined mode). During the commissioning milestone there are periods devoted to independent running, either to debug problems or to ensure stable and good quality data taking (e.g. calibration runs). However, since the goal is to run integrated with other parts of the detector, the combined software is typically run from the central station and the individual shifters may monitor the run in spy (observer) mode.

DAQ Panel Basics

On the left side of the DAQ Panel, specify the setup script used and the online database file used for the run. In particular for the M5 combined run select:

Setup scripts: /det/tdaq/scripts/setup_TDAQ_13.0.30-TR.sh

Database File: /db/tdaq-01-08-03/combined/partitions/combined-postM5.data.xml

Click on Get Partition, select the partition named muon_combined and then click Read In. Under the main menu, SpyIGUI opens a read-only copy of the Central TDAQ GUI.

The OKS button opens the OKS Data Editor with the current partition to edit the configurations database. If settings in the database are changed and the TDAQ GUI is running, it is necessary to reload the database from the command toolbar of the TDAQ GUI (blue arrow on the right of the partition name) when all components are in NONE or INITIAL state. The MRS button opens the Message Reporting System (this is also available from the TDAQ panel). This monitor presents all the error messages from all components in the DAQ system and provides capabilities to filter messages.

In the monitoring tab, the OHP button opens the Online Histogram Presenter and OH opens a root TBrowser with the GNAM histograms (more information below under Monitoring). The DQMF button runs the Data Quality Monitoring Framework that consists of quality algorithms that applied to GNAM histograms (see below the section under Monitoring, and the DQMF Manual). The IS Monitor (Information Service Monitor) button shows detailed information for run parameters, run control, monitoring, and PMG. This feature is also available from the TDAQ GUI. The Data Flow (DF) key provides similar information as the TDAQ GUI Data Flow tab (see next section on TDAQ Panel). The IS Logger opens a GUI for IS. Do not click the icon! The IS Logger crashes often leaving a window which cannot be closed. Finally, the BUSY button opens the CTP Busy Panel.

TDAQ Panel Basics

The TDAQ panel is divided into three main section. The left section is the run control panel that displays the state of the system, the commands that are available (Boot, Config, Start, Pause, Stop, Unconfig, Shutdown). This section also contains a summary of Run Information such as the Run type, the Run number and the rates and number of events at different stages.

The bottom section is the MRS window that displays the (error, info and so on) messages. In the GUI toolbar the user can select to display the messages in the window in LONG or SHORT (this is the default) form. A separate MRS window may also be opened from the main DAQ panel.

On the top right side, the TDAQ panel has multiple tabs with information for running and monitoring. The tabs that are relevant for the MDTs are:

• Run Control tab: This tab provides information about the run control. It is divided into two parts: the left side displays the tree of the segments in the partition and by expanding it the user can see the applications and controllers. The right side has three tabs on the bottom including the Status and Commands available to the user and information regarding the database configuration.
• Infrastructure tab: This tab appears at the startup of the TDAQ GUI. Right-click on field to view the setup and application logs.
• DataFlow tab: The information provided here is similar to Data Flow (DF) under the Information Service (IS) Monitor. All ROS segments show the number of L1 events. The Data Flow Manager (DFM) segment has the number of L2 accepts/rejects and L2 and Event Filter (EF) rates.
• Run Parameter tab: The Filename Tag is used to tag groups of runs. Typically during M5 the tag m5-combined is used. Use tab to Enable/Disable recording of data.
• MRS tab: This tab allows the user to configure filtering and formating of errors and warnings in TDAQ GUI lower screen.
• PMG tab: The Process Manager tab allows the user to view and manually start/stop processes on online computers.
• Segment &amp Resource tab: This tab can be used to enable/disable segments and subtrees by right-clicking on them. Note that the changes must be updated to the database (click on the button below) before running. Beware that it is crucial to check that the changes have actually been implemented in the database after updating since this operation sometimes fails without warning.

A more detailed Control Room Quick Guide to the IGUI is available with additional instructions.

MDT Standalone

In order to switch from a global run into MDT standalone mode running with the barrel, open a terminal as muondaq and type:

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt

muondaq@pc-atlas-cr-06> run

In order to run with the endcap, type instead:

muondaq@pc-atlas-cr-06> tcsh

muondaq@pc-atlas-cr-06> mdt2

muondaq@pc-atlas-cr-06> run

The Barrel and Endcap cannot run in combined standalone mode, but both these partitions can be run simultaneously. When the GUI has loaded, in the Segment &amp resources tab select the MDT RCD (select MROD channels), MDT ROS, TTC and EB (Event Builder) segments and channels that you want to run. Disable the other segments. Make sure that you have the MROD HW_InputChannels desired enabled. Also select the appropriate EB channels that are included in the run. The names of the enabled channels should match the MROD crate names. Commit the changes to the database. Then return to the Run Control tab and follow the steps boot, configure and start.

In order to enable/disable a chamber, the MDT Control Panel provides an extremely intuitive graphical interface. The user needs to select the MROD crate where the chamber is read out, mark the chamber of interest, and commit the changes made to the database. Alternatively, open the OKS data editor, select the MRODX object corresponding to the desired input channel, the mroChanMask is the mask of chambers to be read out, from the most significant to the least significant bit. Change accordingly and save changes.

Monitoring

A variety of tools are available to monitor chamber behavior and detect problems with the detector hardware or in data-taking. Histograms are produced online using GNAM and an Online Histogram Presenter (OHP) facilitates their viewing. In addition, quality algorithms are applied to the GNAM output using DQMF, providing error and diagnostic messages. The monitoring information is displayed in separate interfaces.

GNAM

GNAM is an Online Monitoring structure that allows monitoring of detector status and running conditions. For the MDTs during M5, GNAM has been implemented at the ROD level and provides histograms such as TDC and ADC spectra, hits per chamber and tube and so on. In order to setup GNAM it is necessary to provide chamber and run information. Before starting the run check and/or correct the files located in the directory:

/db/tdaq-01-08-04/muons/data/gnam_setup/mdt

The filenames are chamberList- followed by the name of an MROD crate (e.g., chamberList-EC03.txt). In particular, chambers not included in the run should be commented out and the missing chambers should be added. It is critical that the user modifies the runtype flag depending on the type of run ( 1= noise; 2= cosmics). GNAM histograms are available on the machine where the GNAM process is running. During M5, this machine was pc-tdq-onl-17.cern.ch; the GNAM root files were located in /data/gnam.

It is possible to automatically update the list of chambers utilized by GNAM with the autognamupdate script, located in atlasgw:~abelloni/autognamupdate/autognam.

DQMF and Online Histogram Presenter

The Online Histogram Presenter (OHP) is a stand alone application displays the GNAM histograms, enabling the user to browse the histograms and display selected groups of histograms to be checked during the run. In order to configure OHP, in the main DAQ Panel select the file:

OHP Opt: -c /db/tdaq-01-08-03/muons/segments/DQM/MDT.ohpconf.m5.xml -e ohpcommands

Select Read Info and select OHP in the monitoring tab. The OHP panel is organized in browser mode and with tabs which provide overview information per crate and per chamber.

ELOG

Run information is recorded in the ATLAS Elog, as well as a record of the start and the end of each run. The Elog can be accessed from outside CERN (read-only access) at:

https://pcatdwww.cern.ch/elog/ATLAS/ATLAS/

and from inside the CERN network at:

https://pc-atlas-www.cern.ch/elog/ATLAS/ATLAS/

For read-only access, the following credentials can be used:

Username: atlas

Password: inxer

In order to enter information request a password to Federico.Zema@cernNOSPAMPLEASE.ch.

Hardare Monitoring Guide

Power supply temperatures

Check the temperature for each module, and ensure that it always stays lower than 55 degrees C. Please check this a couple of times every hour. Also, note any large (>5 deg) rise in these values between checks.

Connect to the power supply mainframe as explained in the manual. For each module that is turned on, temperatures are listed (you’ll have to scroll to the right)

MRODs temperatures

When an MROD is in a high temperature state, it displays an over-temperature error in the TDAQ GUI. The easiest way to check for temperature issues with the MRODs is to keep your eyes peeled for these messages! You can also check the mrod log files from time to time.

High Voltage management

Before turning on, check that the ramp up / ramp down (neither should be higher than 100 V/s), trip time (not more than a few seconds) and voltage (3080 V) are acceptable. Confirm these with someone before turning on. Do not turn on high voltage without confirming with the gas people (Jianbei 162532, Joerg 72250). This is very important. While the chambers are on, check that the I0 is low (0-1 mu A) for the channels. When a HV channel trips, do not re-turn on immediately! Please check with the gas supply people first to see if the situation is ok.

Mezzanine and CSM temperatures

Login to the pcatlmdtmdm PCs (these are the machines used for initialization).

We can also perform more detailed monitoring of individual chambers; this takes a little more time, so do it for a sampling of chambers, say ten each hour. First, double click a chamber. You will see the following:

Click the CSM-ADC button. This shows information for the mezzanine cards and the CSM.

You can also click temperature. This just shows the values read out from the temperature sensors on the chambers, but is nice to look at as well.

#AMS.DataQuality

Data Quality Monitoring Framework (DQMF)

Getting Started

To have the DQMF implementation running properly and checking the correct histograms, it is essential to have the DQMF linked to the proper partition and to have the DQMagents running. However, this is best described by DQMF experts, and we defer here. We assume everything has already been setup and the DQMF agents are running. If the Java display is not yet running, it can be started with the command

dqmf display.sh $TDAQ PARTITION &

if the environment variable $TDAQ PARTITION has been set to the partition name.

The Display

In figure 1, we see a picture of the running DQMF java display. Note that the display may have changed slightly for M4, but the general features we discuss here will still be relevant. On the left hand side of the display is the set of histograms to be tested, organized in a tree-like structure. The highest region of the tree is MDT. One can then click on the small blue tab next to MDT to open the tree to the next level down, which should display the subdetector (EndCap or Barrel) along with the side (A or C). Again opening the tab for the subdetector reveals a list of the possible layers. If we continue opening the tree we find a list of the sectors, followed down in the tree by the chambers. Within a chamber is the list of histograms that are tested for the chamber. If you then click on one of the histograms (under the chamber regions on the left tree) the histogram will be displayed on the right hand side. Also on the rights hand side, under Configuration Parameters, one can see the name of the histogram and its reference histogram, the algorithm used to test this histogram, some of the input parameters to the algorithm (called algorithm parameters), and the thresholds which define a good and bad result. Under Results Produced, one can see the output values of the algorithm used to test the histogram. If a histogram has failed a given test, addition information as to why the histogram failed may often be present in the Results Produced section.

The Red, Green, and Yellow Lights

When the relevant histograms arrive on the OHS, DQMF will automatically check the histograms, and publish a result. This result can either be red, yellow, or green. Not surprisingly, red means a histogram has failed a test, yellow means the histogram has not yet failed but needs to be watched, and green means everything appears OK. If a result is simply gray, it means either the histogram never arrived on the OHS, or the histogram failed some initial test and a result was never produced. One example of a failed initial test is a failure of a minimum statistics requirement. An error message would be produced and sent to the specified output (probably a log file, but again, a question for a DQMF expert, it depends on some settings). The color of the result can be seen in the tree on the left hand side of the display. The small circle on the left hand side of the histogram name will turn to the result color. The display is set to the worst case summary, meaning that if any 1 histogram produces a red result, the entire chamber registers a red result, then the sector, and so on all, the way up to MDT. Thus you can simply track the red or yellow lights starting at MDT region all the way down to the offending histogram.

What to do when things change color

The general rule is that if a histogram turns red, you should seek an MDT expert and notify them. If the result is yellow, you should keep and eye on that result (a yellow light does not warrant the notification of an expert). If everything is green, you are in good shape. But, you still need to look around and keep and eye out for gray results. For this reason its a good idea to keep an eye on the file/output where the errors are being printed. If you see a gray result, its best to find out why and asses if one needs to contact an expert. If no error message is produced, the histogram never arrived, and its best to wait a bit to see if the histogram will come a bit later. If there is an error message, it may not be of immediate concern. For instance, if there is not enough statistics, you may need to wait a bit longer for a proper result. If a histogram never arrives after a significant period of time, or if the error message seems to be bad (or worse, incomprehensible) its probably best to contact a DQMF or MDT expert.

So what are we actually checking here?

I will describe briefly what we are checking on each chamber to give you a better idea of what is going on. The TDC histograms give the TDC spectrum of the chambers. We perform a fit to this spectrum and ensure that the rise time and drift time are close to their expected values. Next we check the number of hits per tube, and ensure the shape of this distribution is close to the expected reference histogram. If more than 24 bins deviate by more than 4 from the reference, or any one bin deviates by more than 7, a red result is produced. The exact same check is performed for the Noise Frequency per tube. Finally, we check that the mean and RMS of the ADC spectrum and Hit Multiplicity spectrum are close to their expected values.

-- AlbertoBelloni - 14 Feb 2008