GOAL

The ultimate goal would be to integrate the PCI-Express functionality useful for the MTF6/7 boards within HAL

Current status

• First prototype ready. Still far from an optimal integration, nevertheless managing to write/read registers and blocks of memory into the MTF6/7 prototype in UF

Where is the Code?

• Not in any official repository
• It lives in endcap-tf1.phys.ufl.edu at /home/username/projects/pcie_hal/
• It also copied in /afs/cern.ch/user/d/digiovan/public/backup_hal

To Do List

• Commit the code in some repository
• Remove the dependency from /dev/xpci driver. Not really needed as this driver is for PCI
• Code properly the raising exception part

Before Starting

• Assumption you are working on the UF machine endcap-tf1, see how to connect to endcap-tf1
• Assumption you are working on a SLC6 machine and have superuser rights
• Read the users manual and mostly look at the general stuff and the PCI related stuff, see here.
• Read the documentation about the pci-express driver written by the UF group, see here.
• Install XDAQ environment:
  • Example of xdaq.repo file to be copied into /etc/yum.repos.d/xdaq.repo for XDAQ 11 and SLC6
  • Type the following commands:

sudo yum groupinstall "extern_coretools"
sudo yum groupinstall "coretools" 
sudo yum groupinstall "extern_powerpack"
sudo yum groupinstall "powerpack"
sudo yum groupinstall "database_worksuite"
sudo yum groupinstall "general_worksuite"
sudo yum groupinstall "hardware_worksuite"

• Install the HAL suite:

svn co https://svn.cern.ch/reps/cmsos/releases/baseline11/trunk/daq/hal daq/hal
cd daq/hal
source environment.sh
make

• NB : Th environment .sh file is not present in the svn repository. You can find an example in the attachment below. [environment.sh]
• The code is currently installed in /home/username/project/pcie_hal/daq/hal
• Optional: you may want to learn how to read PCI configuration space or block of memory in one of your PCI card, see here.

How To Connect to UF Machines

• You must ask Alex Mardorsky to give you a working account (and password)
• The code is installed and the hardware in currently hooked in the endcap-tf1.phys.ufl.edu
• This computer is not accessible from outside directly
• If you are not based at UF, the best way to connect to the computer is:
  • Install vncserver (http://realvnc.com/download/viewer/)
  • Double ssh tunnel via cosmic3 as shown below:

ssh -t -L 30105:localhost:30105 username@cosmic3.phys.ufl.edu ssh -L 30105:localhost:5905 username@endcap-tf1.phys.ufl.edu

• Set the vnc server name to localhost::30105 (same password as for ssh)

Example on How to Use the Standalone Code

• This is the code upon which the example for PCIe running in HAL is currently working
• Code living on endcap-tf1.phys.ufl.edu
• Code written using QT

cd /home/username/project/sp12_qt/
./sp12_qt #(This should open cosmic3 desktop on your screen.)
This will open my test project for SP12 prototype. 
In the program's window, press PCI express Loopback button (top right of the window). 
The program runs 1000 read/write cycles, then 1000 write cycles, and 1000 read cycles, measures the performance and prints out results.

The code that Loopback button calls is located in this file loopback_dma.cpp

Note that it calls macros mwrite and mread, that are defined in this file sp12_macros.h

These macros are just wrappers for pwrite and pread functions that check the return value and print error message if needed.

Example on How to Use the Code into HAL

• Prerequisites: Get familiar with the standalone code example above
• The example replicates the functionality of loopback_dma.cpp
• The code is located in /home/username/project/pcie_hal/daq/hal/

cd /home/username/project/pcie_hal/daq/hal
source environment.sh
cd examples/PCIExpr
make
./runnit.sh

Example on How to Interact with one of Your PCI Card with HAL

• Disclaimer: Do it at your risk and read only: if you attempt to write a wrong block of memory you may end up crashing your pc and need to reboot
• Read a bit about PCI configuration space at http://en.wikipedia.org/wiki/PCI_configuration_space
• Check the /dev/xpci exist. If the driver does not exist you may get error like

terminate called after throwing an instance of 'xpci::exception::OpenFileFailed'
what():  Cannot open file/dev/xpci

• xpci is the universal driver which talks to the PCI hardware. Given a driver is kernel code it must be compiled with the same exact kernel where it is suppose to be used. The default driver in HAL may not be compiled with the kernel you are using, so you need to recompile it. Below you will find some guidelines. Unfortunately exact instructions depend on the SLC release you may be using:

Look for the SOURCE rpm with the exact version number as the one installed on your computer at  https://xdaq.web.cern.ch/xdaq/repo
Example: I am on SLC6 machines, 64-bit, with XDAQ 11 installed I would probably pick something like
https://xdaq.web.cern.ch/xdaq/repo/11/slc6x/x86_64/updates/SRPMS/daq-xpcidrv-1.2.0-1.cmsos11.slc6.gcc4_4_6.src.rpm

One way to identify the driver currently installed is
> rpm -qa | grep -i kernel-module-daq-xpcidrv

To recompile:

> wget https:/path/xpci_blabla.src.rpm
> rpm -i xpci_blabla.src.rpm
> cd /root/rpmbuild/ (in other OS it could be in /usr/src/redhat/...)
you will find folders like RPMS, SPEC, BUILD...

> rpmbuild -bb SPEC/{specfile del driver}
> rpm -i RPMS/{rpms created}
It is probably best to remove the existing rpms with the same name as the ones you just created and want to install

load the driver
> sudo /sbin/service xpci stop
> sudo /sbin/srevice/xcpci start

If everything went fine you should find your driver using
> lsmod
or by typing
> ls /dev/ | grep xpci

Good luck...

• Assuming you have a well compiled driver to run the example

cd /home/hvuser/projects/pcie_hal/daq/hal/
source environment
cd examples/PCI
!!!Before running it make sure the device and vendor IDs in src/common/PCIExample.cc 
correspond to the one of the card you would like to test!!!
make
./runnit.sh

Modifications w.r.t. the Default HAL Release

• Before starting, Christoph Schwick sent several guidelines to help and here it is one of the most important one:

I checked once more the interface you would have to implement:

hal/generic/include/hal/PCIBusAdapterInterface.hh

This interface actually DOES contain the readBlock  and writeBlock calls.

For your work you could start to look at the implementation of the PCILinuxBusAdapter.cc
in hal/busAdapterpci/src/common/PCILinuxBusAdapter.cc

In this implementation we use the generic xdaq xpci-driver which only supports single accesses 
and replaces the block transfers by a loop with single accesses. But it is a good starting point to see how things work.

For playing you can use the dummybusadapter in hal/busAdapter/dummy/pci/ but it is better to play with real hardware and 
the standard PCILinuxBusAdapter (you can always pick any PCI device in your PC and read out some ConfigurationSpace 
registers which are the same for all PCI devices. 
Therefore you do not need to know the address tables but just write some basic configuration-space items in the addresstable.  

• Written additional set of busAdapter code in daq/hal/busAdapter/pciexpr/, which is based on daq/hal/busAdapter/pci:
  • The most important class which does all the job is src/common/PCIExprLinuxBusAdapter.cc. This class contains the call to read/write register and memoryBlocks (readBlock/writeBlock)
  • The additional functionality "*findDeviceByIndex*" has been added.
  • The PCI driver is a general one, while the PCI-express one is tailored for the MTF6/7 boards the functionality findDeviceByBus, findDeviceByVendor, findDevice would require more work and are no supported right now.

Driver Differences

The xpci driver is a general-purpouse driver which allows the user to talk to any kind of PCI hardware. 
Therefore it contemplates the possibility to identify a PCI board via its vendor, device IDs as well as its position on the BUS.

The UF driver does look at these registers as well: it indeed reads Vendor and Device ID to make sure it talks to appropriate boards. 
It also reads BAR registers to learn the starting addresses and sizes of address space windows allocated for each board.

HOWEVER, the user-mode software is isolated from these details. 
As soon as one opens a utca_sp12X device (where X=0,1,... number of connected boards), he/she knows that this is one of the MTF6/7 boards. 

The starting physical addresses that the driver reads out from each board's BAR registers are not available to the user as in the case for the xpci driver. 
User-mode software will operate in terms of "offset" addresses from the start of the address space of each board. The latter is true also for the xpci driver.
 

Useful Links

• Main HAL web page
• XDAQ project

-- GianPieroDiGiovanni - 24 Jan 2014

• environment.sh: environment.sh