GEM DAQ Development Work

Working on developing a uTCA based readout system for the CMS GEM project

Useful links

• HCAL ngFEC (GLIB based)
  • HCAL firmware
• AMC13 information
  • AMC13 firmware
• Using the CERN terminal services for FPGA design
• Kevin Sung's twiki about using the GLIB

Installing and using Xilinx tools on linux

Installation

• The Vivado, ISE and LabTools tarballs are stored on the NAS, you can obtain them from /data/bigdisk/GEMDAQ_Documentation/Xilinx on any of the GEM DAQ PCs
• Copy the source to the machine where you want to install it and untar the source package (I recommend you store it in /tmp, but you just need some place with ~25 GB to unpack the source, and another 20/6 GB to install ISE/!LabTools):

  cp /data/bigdisk/GEMDAQ_Documentation/Xilinx/Xilinx_ISE_DS_Lin_14.7_1015_1.tar /tmp
  cd /tmp
  tar xf Xilinx_ISE_DS_Lin_14.7_1015_1.tar
  sudo ./xsetup
     

• You will need two additional packages for getting the USB platform cable working on Linux fxload and usb-driver,
  • fxload is available from sourceforge or,
  • on the GEM DAQ machines, the fxload source is located in the same location as the other tar files (you may need the libusb-devel or similar libraries), so you can copy it from there and then do:

    cp /data/bigdisk/GEMDAQ_Documentation/Xilinx/fxload-2008_10_13.tar.gz /tmp
    cd /tmp
    tar xzf fxload-2008_10_13.tar.gz
    cd fxload-2008_10_13
    make -j4
    sudo make install
         

• • The usb-driver source can be obtained from github, and I recommend you put the source in the same directory that your ISE/LabTools installation is made), you will need the libusb-devel or similar libraries:

    cd /opt/Xilinx
    cd usb-driver
    sudo git clone git://git.zerfleddert.de/usb-driver
    sudo make -j3
         

• Some final steps are:

  source /opt/Xilinx/14.7/ISE_DS/settings64.sh
  ## or for LabTools
  source /opt/Xilinx/14.7/LabTools/settings64.sh
  
  cd /opt/Xilinx/usb-driver
  sudo ./setup_pcusb $XILINX
  sudo cp $XILINX/bin/lin/xusb*.hex /usr/share/
  sudo sed -i -e 's/TEMPNODE/tempnode/' -e 's/SYSFS/ATTRS/g' -e 's/BUS/SUBSYSTEMS/' /etc/udev/rules.d/xusbdfwu.rules
     

• Xilinx ISE 14.7 on CentOS 6.5
• Useful hints on getting the USB Platform Cable II working

Usage

• Set the CERN license server

  export XILINXD_LICENSE_FILE="2112@lxlic01,2112@lxlic02,2112@lxlic03"
  source /opt/Xilinx/14.7/ISE_DS/settings64.sh
  ise
       

• Useful command for unblocking a non-recognized cable

  echo -e 'cleancablelock\nexit' | impact -batch
       

• Different device IDs

  more /opt/Xilinx/gem904daq01.cdb
  # Cable Database File
  # Format:
  # ESN,HostName,CableName1,CableName2...
  #  - Set HostName to ' ' (SPACE Character) if the cable is connected locally.
  000015E0F9E001, ,OptoHybrid
  0000150F493601, ,GLIBs
  
  
  GLIB in slot 9
  '1': IDCODE is '8424a093' (in hex).
  '1': : Manufacturer's ID = Xilinx xc6vlx130t, Version : 8
  
  OptoHybridV2A on table
  '1': IDCODE is '6424a093' (in hex).
  '1': : Manufacturer's ID = Xilinx xc6vlx130t, Version : 6
       

Using the Digilent basys2 test board:

• http://sadgeeksinsnow.blogspot.ch/2012/01/using-xilinx-ise-webpack-ide.html
• http://sadgeeksinsnow.blogspot.ch/2012/01/installing-digilent-adept-2-on-linux.html
• Using Adept to program the Digilent test board

  djtgcfg
  djtgcfg enum
  djtcgcfg init -d Basys2
  djtgcfg prog -d Basys2 -i 0 -f image.bit
       

Using the uFedKit

• UFEDKIT
• CMSFedkitManual

Setting up the system to read events off the GLIB

• 1 In order to establish reliable communication between GLIB and OH, I need to flash GLIB FW, then flash OH FW and then powercycle the GLIB (clock recovering issue?)
• 2 after the powercycle GLIB has no clocking signal from AMC, its DAQ counters looks good (also checked with chipscope). No extra fake triggers/fake data present.
• 3 AMC 13 can generate triggers in en X t mode, however the TTS should be disabled before by running wv CONF.AMC.TTS_DISABLE_MASK 0xfff
• 4 If I enable the fake data generation, I see sometimes some chunks of fake data dumped by FEROL card. Sometimes, but not always - I didn't find a good pattern yet.
• 5 After enabling the slot with GLIB in AMC13, it starts to receive the clocks and also start to receive some strange events. The triggers are disabled and the counters aren't incrementing everywhere in the system, but it starts to built events.
• 6 If one enables AMC13 triggers, they go until 60 and then stop - same as at 904.
• 7 GLIB never sends an event to the AMC13, at least the counter of sent events is always 0.
• 8 The AMC_LINK_READY flag is always 0.

Compiling the Mellanox driver

• Find the device information

  lspci -v | grep Mellanox
  01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
          Subsystem: Mellanox Technologies Device 000d
       

• Obtain the latest drivers from here linux

  # Unpack the downloaded tarball
  tar xzvf mlnx_en-<version>.tgz
  
  #Change the working directory.
  cd mlnx_en-<version>
  
  # Run the installation script.
  ./install.sh
  
  # Load the driver.
  /etc/init.d/mlnx-en.d restart
       

Recompiling the driver

# Enter the source directory.
cp -a /usr/src/mlnx-en-<version>/ /tmp
cd /tmp/mlnx-en-<version>

# Apply kernel backport patch.
scripts/mlnx_en_patch.sh

# Compile the driver sources.
make -j4

# Install the driver kernel modules.
make install
     

Installing and using the management tools

Find the management tools here

Configuring the 10 GbE NIC

• The 10 GbE NIC should be configured to have a static IP address of 10.0.0.5, as per the twiki above. To do this, you'll need to modify the ifconfig-device file as follows:

  cat /etc/sysconfig/network-scripts/ifcfg-eth2
  DEVICE="eth2" #this should match the device reported by ifconfig that corresponds to your Mellanox card
  HWADDR="E4:1D:2D:29:E8:E0" #this should be your MAC address
  BOOTPROTO="none"
  ONBOOT="yes"
  TYPE="Ethernet"
  IPADDR="10.0.0.5"
  NETWORK="10.0.0.0"
  NETMASK="255.255.255.0"
  USERCTL="no"
  NM_CONTROLLED="no"
  IPV6INIT="yes" # probably not necessary
       

• Set up the firewall to allow all requests on this interface (eth2, but again, make sure it is the correct one for your setup)
• The rule needs to be inserted before the first REJECT rule in the INPUT chain, to find this, look at the output of sudo iptables -L -v

  sudo iptables -I INPUT 5 -i eth2 -j ACCEPT #accept all incoming requests on the 10 GbE NIC, insert rule to position 5 of the INPUT chain
  # alternatively, just append the rule to the end of the chain, but IF there is a REJECT that is satisfied by the traffic, it won't get to the rule you appended
  sudo iptables -A INPUT -i eth2 -j ACCEPT #accept all incoming requests on the 10 GbE NIC, append rule to the end of the INPUT chain
  #sudo iptables -A OUTPUT -j ACCEPT #accept all outgoing requests
       

• Maybe the following if you need more specific filtering, accept all traffic with source/destination of IP address supplied, insert rule into position 5 of the INPUT chain

  sudo iptables -I INPUT 5 -i eth0 -d 10.0.0.5 -j ACCEPT
  sudo iptables -I INPUT 5 -i eth0 -s 10.0.0.5 -j ACCEPT
  sudo iptables -I INPUT 5 -i eth0 -d 10.0.0.4 -j ACCEPT
  sudo iptables -I INPUT 5 -i eth0 -s 10.0.0.4 -j ACCEPT
       

• Save the firewall rules if you want them to be picked up automatically

  sudo /etc/init.d/iptables save
       

Recompiling the xpci driver for use with the uFEDKIT

If you are using a different kernel than the one "officially" supported by the CMS DAQ group, you will have to rebuild the xpci driver. To do this, follow the below procedure (as a non-root user) do the following

# get src rpm
yumdownloader --source daq-xpcidrv

# get deps
sudo yum-builddep daq-xpcidrv-<version>.src.rpm

# install rpm 
rpm -ihv daq-xpcidrv-<version>.src.rpm
cd ~/rpm # or ~/rpmbuild depending on setup
rpmbuild -bb SPECS/daq-xpcidrv.<version>.spec
cd RPMS/x86_64/

# remove the old packages
sudo yum remove daq-xpcidrv daq-xpcidrv-debuginfo  daq-xpcidrv-devel

# install the newly built versions
sudo yum install kernel-module-daq-xpcidrv-<version>.rpm
sudo yum install daq-xpcidrv-<version>.rpm
sudo yum install daq-xpcidrv-debuginfo-<version>.rpm
sudo yum install daq-xpcidrv-devel-<version>.rpm

#check that the driver is installed
sudo /sbin/lsmod

#load the driver if not installed
sudo modprobe xpci
     

Other tools

Thomas' NodeJS web-daq

1. Install NodeJS on a computer
2. git clone http://iihe-daq.iihe.ac.be:333/tlenzi/gem-daq.git
3. cd gem-daq/web-daq
4. npm install
5. Open app/ipbus.py and modify line 4 with the correct GLIB address
6. run node app.js
7. go to http://localhost:3000 to see the web daq system
     

Setting up macvlan for docker hosted AMC emulator

nmcli c e
Valid connection types: adsl, bluetooth, bond, bridge, cdma, dummy, generic, gsm, infiniband, ip-tunnel, macsec, macvlan, 802-11-olpc-mesh (olpc-mesh), ovs-bridge, ovs-interface, ovs-port, pppoe, team, tun, vlan, vpn, vxlan, wimax, 802-3-ethernet (ethernet), 802-11-wireless (wifi), bond-slave, bridge-slave, team-slave
Enter connection type: macvlan

===| nmcli interactive connection editor |===

Adding a new 'macvlan' connection

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, macvlan, 802-3-ethernet (ethernet), ipv4, ipv6, tc, proxy
nmcli> set connection.id utcamacvlan
nmcli> set connection.interface-name utcamacvlan
nmcli> set connection.zone trusted
nmcli> set ipv4.method manual
nmcli> set ipv4.addresses 192.168.0.190/16
nmcli> set ipv4.ignore-auto-dns yes
nmcli> set ipv4.never-default yes
nmcli> set ipv4.may-fail yes
nmcli> set ipv6.method ignore
nmcli> set ipv6.ignore-auto-dns yes
nmcli> set ipv6.may-fail yes
nmcli> set ipv6.never-default yes
nmcli> set macvlan.parent em2
nmcli> set macvlan.mode 2
nmcli> set macvlan.promiscuous yes
nmcli> save persistent
Saving the connection with 'autoconnect=yes'. That might result in an immediate activation of the connection.
Do you still want to save? (yes/no) [yes] yes
Connection 'utcamacvlan' (54b739d2-baa8-4760-bf65-30a34969230f) successfully saved.
nmcli> q

firewall-cmd --zone=trusted --add-interface=utcamacvlan
firewall-cmd --zone=trusted --add-interface=utcamacvlan --permanent

Links

• nodejs
• emacs cheat sheet
• nodejs overview
• nodejs tutorial
• nodejs tutorial 2
• nodejs overview 2
• dynamic web pages
• ajax intro W3C
• ajax overview
• ajax with php

-- JaredSturdy - 2015-06-24