MilliQan Shift Information

This page documents and provides information about the weekly shift position for the MilliQan 1% demonstrator.

Useful Information

• Elog: https://cmshead.mps.ohio-state.edu:8080/Millicharge/
• David Stuart's validation plots: http://cms2.physics.ucsb.edu/milliqanValidationPlots/index.html
• Channel map from Ryan

Quick Start

ssh -Y $USER@lxplus.cern.ch
ssh -Y $USER@milliqandaq-00.cern.ch
DAQCommand

Shifter Responsibilities

• Ensure the DAQ is operating as expected
• Ensure the data is being successfully transferred to UCSB
• Ensure that any requests for special data-taking are fulfilled in a timely manner
• Document all runs taken
• Report any problems
• Progress on list of code improvements
• Update this document

Details of Responsibilities

Ensure the DAQ is operating as expected:

• Has data-taking ceased at any point? Has the run number changed without providing an operation that would change it?
• Is the trigger rate reasonable?
  • Keep in mind that "triple coincidence" running is typically about 12-15 Hz
  • A trigger rate of 100 Hz would correspond to over 22.7 Gigabytes per hour
  • We observe a transfer rate of 5-6 MB/s which means that sustained running over 80 Hz means we cannot transfer faster than we record
  • /home directory have 1.3 TB available for data. So if we ran at 180 Hz, we would accumulate un-transferred events at 100 Hz and could only stay in this configuration for at most 57 consecutive hours, assuming all transferred files are deleted promptly.
• Check that DAQ is running
  • Check data files are being written
  • Do DAQCommand print info (see below for more details)
• Check that data transfer is running
  • Try to run the transfer script (see below for more details) so that there's no backlog of untransferred files.
• Try to record all relevant info in the elog
  • For example, record currents from the HV source when running at special HV (look at Oct 17 in this elog)

Operating MilliQan

High Voltage

High Voltage is set by hv command on the milliqandaq-00 machine. If you issue this command, you will see a window like this (a few seconds later)

Look at that last four lines. The normal operation HV and the max values are

To change the HV

1. Triple click the V0Set value of a HV channel, for example, 1600.0... for CH10
2. Change the number. Note that the interface is very slow. So, make sure that you do it very slowly and carefully.
3. Hit enter and the value will be set in a second. Before hitting enter make sure that the value is not above the maximum voltages.
4. To finish, ctrl+c or click in the x at the top right of the window

Function generator

Function generator is used to set hardware prescale. You can set the prescale on the milliqandaq-00 machine. To start, do

 FGCommand 

Then, you enter

 PULSFREQ [freq] 
 PULSWID [width] 

The [freq] is in Hz and [width] is in sec. The prescale = 1/(PULSFREQ*PULSWID) so, for example, if you want to use a prescale of 100, you can use 1000 for [freq] and 10e-6 for [width]

 PULSFREQ 1000
 PULSWID 10e-6 

Note that there is no feedback from the function generator after you sent the commands. You need some faith... If you don't, look at the trigger rate after starting a run. If the rate is consistent with typical values (see the table below), then you are good.

MilliDAQ

The core program recording data from the detector is MilliDAQ, running as a system service by the root user. It is constantly reading from the digitizer, and writing its output to the file ~milliqan/tmp/MilliQanEvents.root and renaming this file to ~milliqan/data/. To interact with the program:

 DAQCommand command (argument)

Show help message Hide help message

Usage: DAQCommand command (argument)

Commands available:
    quit                   -- terminate the MilliDAQ process
    pause                  -- pause then run
    unpause                -- unpause a paused run
    start                  -- start the run
    restart                -- stop, then start the run
    stop                   -- stop the run
    print [configuration | board | info]
          print configure  -- print the current V1743Configuration parameters: trigger mode, thresholds, etc
          print board      -- print information about the V1743 board: connection, firmware versions, etc
          print info       -- print DQM information: trigger rates, missed triggers, etc
    reconfigure </full/path/to/file.xml> -- stop the run, read/apply a new configuration file, then start
    dqm                    -- force an update of DQM plots and rates
    rotate                 -- force the current output file to be saved and move on to a new file
    peek                   -- save the last 10 events to a different file
InteractiveDAQ only commands:
    interactive_configure <file.xml> -- read/apply a new configuration while not running
    interactive_timed <nsec>         -- start a timed run lasting <nsec> seconds
    interactive_nevents <nevents>    -- start a run recording <nevents> events
    interactive_togglewaves          -- toggle the saving of waveforms on/off (not yet implemented)

To see the status of the DAQ program:

 DAQCommand print info

Show results Hide results

$ DAQCommand print info

Displaying log file...

Missed triggers for channel 12: 495(6.50802/s)
Missed triggers for channel 14: 269(3.53668/s)
Missed triggers for channel 15: 357(4.69366/s)
Average queue depth: 0 (max: 2)

Tue Oct 10 14:53:59 2017 GMT User command: [MilliDAQ]: Recieved print information command.
Tue Oct 10 14:53:59 2017 GMT Info: [DQM]: Since last update:
   Total board triggers: 28 (12.664/s)
   HLT fires: 28 (12.664/s)
   Cosmics: 28 (12.664/s)
   Cosmics after prescale: 28 (12.664/s)

Missed triggers for channel 1: 32(14.4731/s)
Missed triggers for channel 2: 5(2.26142/s)
Missed triggers for channel 3: 6(2.7137/s)
Missed triggers for channel 4: 13(5.87969/s)
Missed triggers for channel 5: 13(5.87969/s)
Missed triggers for channel 6: 5(2.26142/s)
Missed triggers for channel 7: 28(12.664/s)
Missed triggers for channel 8: 2(0.904568/s)
Missed triggers for channel 9: 3(1.35685/s)
Missed triggers for channel 10: 10(4.52284/s)
Missed triggers for channel 11: 6(2.7137/s)
Missed triggers for channel 12: 69(31.2076/s)
Missed triggers for channel 14: 9(4.07056/s)
Missed triggers for channel 15: 26(11.7594/s)
Average queue depth: 0 (max: 3)

To start a run:

 DAQCommand reconfigure /full/path/to/config

There are a few configuration files in MilliDAQ/config/.

Configuration file	What is it?	Rate	Comments
TripleCoincidence.xml	Unprescaled triple coincidence config	~13 Hz at normal HV	Used for collisions
DoubleCoincidence.xml	Unprescaled double coincidence config	50-150 Hz depending on HV
SingleChannelTrigger.xml	Prescaled single channel coincidence config	With prescale 100, 120-150 Hz depending on HV.	This should be used with prescale of 100 or higher in order to keep the trigger rate below 200 Hz.
prescaleDoubleCoincidence.xml	Prescaled double coincidence config		Same as DoubleCoincidence.xml except that it uses externalAndNormal trigger type. This means that function generator prescale is taken into account. The normal trigger type does not care about what the function generator is set with.
highThreshDoubleCoincidence.xml	Unprescaled double coincidence config with high threshold	180-190 Hz at normal HV	This uses higher threshold then DoubleCoincidence.xml in order to get more cosmics data.
prescaleHighThreshDoubleCoincidence.xml	Unprescaled double coincidence config with high threshold.	With prescale 5, 37 Hz at normal HV	This is a special config made to study the dead time of highThreshDoubleCoincidence.xml. Look at this elog.

 DAQCommand reconfigure /home/milliqan/MilliDAQ/config/TripleCoincidence.xml 

If you want to take a run after changing HV, do

 DAQCommand stop
 (adjust HV as described above)
 DAQCommand reconfigure /full/path/to/config 

All DAQCommand operations show "tail -f /var/log/MilliDAQ.log", so this view will continue to update; CTRL+C can be used to escape this view.

Data transfer to UCSB

Data transfer is done by /usr/local/bin/transferFiles.py. Keep this process running in the background. You should run this occasionally so that there's no backlog of untransferred files.

 nohup transferFiles.py & 

This copies files to UCSB, and moves originals to /home/milliqan/transferred/, and lists all files it has transferred in /home/milliqan/NewFilesAtUCSB.log

To check status of transferred files,

1. Kill transfer process (find PID with ps -u milliqan)
   • This action typically corrupts the current file at UCSB, which you will discover in subsequent steps
2. Execute ./checkFiles.sh
   • This will scp the NewFiles log to UCSB, and do some basic checks
3. To see the results, at UCSB do tail /homes/milliqan/transfersChecks.log
4. rsync missing files manually
   • e.g. rsync transferred/MilliQan_Run54.82_*.root milliqan@cms2NOSPAMPLEASE.physics.ucsb.edu:/net/cms6/cms6r0/milliqan/UX5/
5. When satisfied, rm -f transferred/*.root and rm NewFilesAtUCSB.log
   • Ideally, should make UCSB trees and David Stuart's plots before deleting originals

Trouble shooting

MilliDAQ crashes

• How to tell if it has crashed.
  • Print info fails (no response, strange response)
  • No data files being produced (based on time stamps of data files)
• How to recover
  1. Check if MilliDAQ process is running at all (ps)
     • If process exists: attempt reconfigure
  2. Kill by PID- will be resurrected 30 seconds later in externalTrigger
     • Afterwards: attempt reconfigure. Sometimes it takes several reconfigure commands to be successful.
     • Check trigger rate (print info) and root file names in data/ to confirm
  3. If it doesn’t resurrect: systemctl status MilliDAQ.service to look for clues

      sudo systemctl stop MilliDAQ.service sudo systemctl start MilliDAQ.service 

  4. Also try removing root file buffer in tmp/MilliQanEvents.root

Updating MilliDAQ

EXPERTS ONLY

Updating MilliDAQ should be a rare occurrence. If needed, check the status of the LHC to avoid downtime during collisions. If there are changes in gitlab not yet propagated to the DAQ PC:

# as user "milliqan"
cd ~milliqan/MilliDAQ/
sudo make uninstall 
# the above command issues "DAQCommand quit", disables the service from restarting, and removes executables and scripts
git pull
make clean all
sudo make install
sudo systemctl start MilliDAQ.service