This page describes the PhEDEx testbed setup for ANSE

Testbed components

There are several components of the testbed. The basic installation for read-only access is described, developers who will be working on this should read the section at the end before doing any of these steps. PhEDEx itself consists of an oracle database, a single set of 'central' agents, and 'site agents' which are run at every site. There is also a website for user-interaction with the database.

We also use vagrant to manage VM images. The vagrant box we are using is http://vagrant.phys.uvic.ca/scientificlinux-57.box (thanks Ian). Currently PhEDEx is only supported on SLC5 or SLC6 systems, so you need something close to that as a starting point.

The testbed configuration software is in the ANSE git repository at https://github.com/anse-lhc/PhEDEx-Testbed.

The database

The Oracle database is at CERN, and is one of the standard PhEDEx testbed instances. Currently only ANSE is using it, though it may be shared with other users (Hepix IPv6 group) if that can be done cleanly. Anyone wanting access to the database (i.e. anyone wanting to participate in the testbed) should mail me with their grid certificate's public key (usually ~/.globus/usercert.pem). I will then send you an email with the encrypted connection parameters and instructions for decrypting them.

Please treat the DBParam file with the care you would use for any database connection parameters. I.e. the file should not be stored anywhere public, and should not be readable by anyone who does not need to use it.

The database is maintained by Tony, nobody else should need to interact with it except via the PhEDEx tools.

Website

Andrew has set up a website for us at https://brazil.accre.vanderbilt.edu:4443/phedex. People can set up another one if they wish, it won't do any harm, but there's no need for it.

I have not provided detailed instructions for installing the website. It's based on a pre-installed image at http://wildish.home.cern.ch/wildish/ANSE/OVA/PhEDEx-web.ova, which can be run with VirtualBox.

Central agents

There is a VM running at CERN, phedex-anse.cern.ch, maintained by me. We can use that for the central agents. In principle the only people who will need access to it are the people who will be working directly on the PhEDEx code. Mail me if you want access for any reason.

Central agents should also be installable via the same mechanism as for site agents, described below. Providing only one instance of the agents is running at any one time, this should work.

Site agents

Site agents are available via a VagrantUp virtual machine (thanks to Andrew and Ian). The startup scripts will need to be customised for each site to specify the node-name, which must match that in the PhEDEx database (see below). You will also need database connection parameters, which will only be sent by encrypted email. Again, mail me with your public certificate and I will mail the parameters to you. The exact procedure is:

Install the VM with Vagrant

Assuming you have vagrant installed on your system, this should be as simple as:

git clone git@github.com:anse-lhc/PhEDEx-Testbed.git ANSE-PhEDEx-Testbed
cd ANSE-PhEDEx-Testbed/VagrantUp
vagrant up

The first boot will take a long time. First it downloads the image, then boots and does the relevant system updates. There are a lot, so this can take 10-15 minutes. Then it installs a few RPMs that are expected by the CMS installation process but which are missing in this image, which takes another 5 minutes or so. Finally, it installs the PhEDEx software in TESTBED_ROOT (relative to the vagrant user home directory, /home/vagrant), this also takes several minutes. Expect 20-30 minutes for your first boot.

When the installation finishes, reboot the VM to pick up the latest kernel:

vagrant reload

Changing the VM parameters

The VM comes out of the box with 512 MB RAM and 1 CPU. The VagrantFile reconfigures it to have 2 GB RAM and 2 CPUs, for more realistic performance. You can reduce this to 1 GB RAM and 1 CPU while testing with fake transfers, but lower than that is probably not workable. OTOH, there's little point in giving your VM more than 2 GB RAM and 2 CPUs. You'll probably only want to lower these values if you're testing from a laptop.

Configure your site

You need to know the name of your site. The database has been configured with a fictitious Tier-0 (T0_Test_Buffer), four test nodes (T2_Test1_Buffer to T2_Test4_Buffer), and a node for each participating site: T2_ANSE_Caltech, T2_ANSE_CERN, T2_ANSE_Michican, T2_ANSE_UTA, T2_ANSE_Vanderbilt.

You need to vagrant ssh into your VM and set the PHEDEX_SITE variable in TESTBED_ROOT/env-anse.sh.

You also need that DBParam file from me. When you get it, copy it to TESTBED_ROOT/

Check your installation

In TESTBED_ROOT/ANSE-PhEDEx-Testbed/PhEDEx-site-agents, you will find a script, check-installation.sh. This will perform some simple checks to make sure your environment is configured correctly and that you can contact the database at CERN. If that works, you can start your agents. If it reports an error, fix it, and run this script again.

Starting/stopping agents

In TESTBED_ROOT/ANSE-PhEDEx-Testbed/PhEDEx-site-agents, you will find two more scripts: start-agents.sh and stop-agents.sh. No prizes for guessing what they do! You can start or stop the agents as often as you like, whenever you like. They maintain only as much state as they need locally and can recover from random restarts.

Once you have started your agents you can check they are running in several ways. Obviously you can check for the processes, but you can also look at the logfiles. They are in TESTBED_ROOT/Testbed2_$PHEDEX_SITE/logs/. You won't normally see much activity unless there are transfers active, but you should see a status line every hour or so.

You can also check the website 'component status' page, at https://brazil.accre.vanderbilt.edu:4443/phedex/prod/Components::Status. That has a bit of a delay, so it may be a few minutes before your agents show there. Also, when you stop your agents, the page will carry on showing them in green for some time. We expect people to stop and re-start agents from time to time and don't want to alarm central operators too readily for no reason.

N.B. You should only have one set of agent-processes running at a time. If you want to delete your virtual machine and start from scratch, that's fine, you can install the agents anywhere, as often as you like, as long as you only have one set running at any time.

Updating your installation

To update your installation without completely re-installing the virtual machine, do this:

cd $TESTBED_ROOT/ANSE-PhEDEx-Testbed/PhEDEx-site-agents
./stop-agents.sh

Then check manually (ps auxww | grep TESTBED_ROOT/T2) if any agent processes are still running. Kill them manually if they are. Then do this:

git pull
../PhEDEx-admin/re-install.sh

N.B. If you've modified any files in the ANSE-PhEDEx-Testbed area you will need to make a backup of them first.

After that, you can simply re-start your agents.

What next?

That should be enough to get a testbed working where we can exercise fake transfers. Once the agents are running for your site, let me know, and I can set up some fake transfers to check things out.

The next concrete steps in terms of PhEDEx development are:

• update the LifeCycle agent to inject real files. These can either be pre-prepared files in a SE if we have a big enough machine at each site, or it can be fake files, made by soft-linking to a single real file. That options will allow us to fake several hundred TB on a single disk without elaborate hardware.
• choose a transfer-protocol between sites and start real transfers. Configure the agents to automatically delete the files on transfer in order to avoid filling the disk
• make some baseline measurements on how well PhEDEx performs over timescales of 12-24 hours between representative sites
  • may need to enhance PhEDEx monitoriing if it's not detailed enough
• enable circuits to see how much performance we gain from that alone, without any interaction between the circuit management and PhEDEx itself. That should show the upper bound of what we can achieve
• then start the integration with PhEDEx to manage the circuit lifetime

For developers

If you intend to do any development in this environment there are a few things you should do to make things easier.

During the installation step, after you should not clone the official git repository, but make a personal fork of it instead, and clone that. Then you should change the ANSE-PhEDEx-Testbed/VagrantUp/install-phedex.sh boot script to point to your personal fork of the ANSE git repository before starting the virtual machine. You'll need to use a read-only path, because your VM will not have credentials on the first boot.

Note that the git repository is cloned twice. The first clone is to get the VagrantFile and the system & PhEDEx installation scripts. The second installation is in the home directory of the virtual machines user, to serve as a working directory. After the VM has been fully installed, you can log in and edit the ~/TESTBED_ROOT/ANSE-PhEDEx-Testbed/.git/config file and change the git URL to a read/write path, so you can commit your modifications.

Installing on other VM systems

If you want to use a pre-existing SLC5 image instead of using vagrant, you can try this recipe, which works on a standard CERN SLC5 machine:

Clone the git repository as before, directly in your VM

git clone git@github.com:anse-lhc/PhEDEx-Testbed.git ANSE-PhEDEx-Testbed

In the ANSE-PhEDEx-Testbed/VagrantUp directory, execute the install-system.sh script as root to make sure your basic system has the right RPMs on it.

Next, decide where you want to install PhEDEx. By default, it will install into your home directory, under $HOME/TESTBED_ROOT. You can edit install-phedex.sh and change that, it's right at the top of the file. Then execute the install-phedex.sh script as the user you want to run PhEDEx as. Then proceed with the configuring your site, checking your installation, and starting your agents.

Automating the installation on other SLC5 systems

It should be simple enough to automate the installation on a non-vagrant box providing it's running something close to SLC5. First, log in and clone the repository, as shown above. Then, there are two places to modify things, both in ANSE-PhEDEx-Testbed/VagrantUp/:

1. bootstrap.sh steers the whole process. Change the definition of the bootstrap variable to point to where the git repository was cloned.
2. install-phedex-boot.sh shows the installation sequence for PhEDEx if you are running as root. You can either edit it appropriately, or just run install-phedex.sh as the appropriate user.

If you wish to install the software somewhere other than the home directory of the VM user, edit the install-phedex.sh script too, and change the parts where $HOME is used.

Then you should be able to run bootstrap.sh as root and with luck, it will all just work.

Installing on SLC6 systems

For SLC6, you will need to install by hand after tweaking a few scripts. I've only installed on CERN-standard SLC6 systems, so this may take longer, but it should work eventually.

First, clone the git repository as described above. Then, in ANSE-PhEDEx-Testbed/VagrantUp/, edit install-phedex.sh and change

export SCRAM_ARCH=slc5_amd64_gcc461

to

export SCRAM_ARCH=slc6_amd64_gcc461

then execute the script. The chances are that it will complain about missing dependencies, things which the CERN SLC6 systems have by default. You will need to install the relevant RPMs on your system one by one as you go through the process. Eventually, you should get something that works.

If you want to contribute a script that installs those RPMs automatically, like install-system.sh does, that may make the process easier for the next person who tries.

Setting up real transfers

Setting up real transfers requires several things:

• a pool of files to transfer
• a Trivial File Catalogue to map Physical File Names to Logical File Names
• a correctly configured download agent
• a subscription from the source to the destination

Creating a pool of files

Presumably we don't all have petabytes of disk space lying around, but we want to have a huge amount of data to pump through PhEDEx in order to test things at high rates. Fortunately, PhEDEx is happy to follow soft links, so we can fake the presence of a large volume of data with just one file. So, we need to create lots of soft-links and register them as data in PhEDEx.

Assuming we're using FDT for the transfers, the pool of files must be on a disk system accessible to the FDT server. If you're exporting files only, and not importing them, that can be a separate machine from the site agents that are running for that site. However, it's better to keep everything together if you can, so the site-agents and FDT server should all be running on the same machine. I will assume that from here on.

In ANSE-PhEDEx-Testbed/PhEDEx-central-agents, there's a file, Injection.conf, that you can feed to the Lifecycle agent. The only parts you will need to modify are in this section:

  Workflows => [
    {
      Name                      => 'Raw data',
      Template                  => 'RAW',
      Suspend                   => 0,

      InjectionSite             => 'T2_Test1_Buffer',

# 10 injections per block & 250 blocks per dataset gives:
# 625K files per dataset
# 1250 TB per dataset
#    5 TB per block
      InjectionsPerBlock        =>        10,
      BlocksPerDataset          =>       250,
      DropClosedBlocks          =>         1,

#      T2Subscribe      => {
#        Nodes       => 'T2_Test2_Buffer',
#      },

#     Initial parameters for the generator
      Dataset   => '/data/fakeData-02-%04x/RAW',
      Datasets  =>     1,
      Blocks    =>     1,
      Files     =>   250,

      AutoRestart => 1,

      DBS       => 'http://cmsdoc.cern.ch/cms/aprom/DBS/CGIServer/query',

#     Hard or soft links?
      makeLinks => {
        LinkStyle => 'soft',
        LinkDir   => $ENV{TESTBED_ROOT},
        SrcFile   => {
          Name     => $ENV{TESTBED_ROOT} . '/data/file-2000-a.gz',
          Size     => 2097152000,
          Checksum => 'cksum:1526619636',
        }

The SrcFile has a Name which must correspond to a physical file on your system. It should be about 2 GB in size. This is the file you will soft-link to to create your registered data.

The LinkDir is a directory under which the links will be stored. The links will be stored in a subdirectory structure under this root. The subdirectory structure is store/$Dataset, where the $Dataset is the name you gave to the Dataset field, at the top of that stanza. So if your dataset were called /data/my-fake-data/RAW, your links would be stored under TESTBED_ROOT/store/data/my-fake-data/RAW/. The links themselves are further nested under that, mimicking the LFN of the files.

You should change the InjectionSite field to be the name of the site you are injecting data at.

You should choose a descriptive name for your dataset, probably based on the site you are injecting it at. E.g, /data/fake-Michigan-01/RAW.

The Size and Checksum are obvious. The LinkStyle can be hard or soft, but with hard-links you will not be able to create many files, most filesystems have a hard limit on the number of hard-links you can create, only about 32K or so.

You will need to have a valid proxy, then just run

LifeCycle.pl --config Injection.conf --log Injection.log

and it should run fine. It may take several hours, but you will be able to see links appearing in your $TESTBED_ROOT/store directory.

You can run the script again, if you need to, to create a new dataset. Just change the Dateset name to something new and run it.

The Trivial File Catalogue

The Trivial File Catalogue (TFC) is what maps Logical File Names (LFNs) to Physical File Names (PFNs). The LFN is the same everywhere, and is a unique filename-like string that identifies the file to the PhEDEx system The PFN for a given copy of a file is probably different at all sites, containing the protocol and local path to the actual copy of the file. So PhEDEx only deals with LFNs, but the transfer needs PFNs to get the actual file. The mapping between the two is maintained by the TFC. The testbed site-agents come with a sample TFC (ANSE-PhEDEx-Testbed/PhEDEx-site-agents/tfc.xml) which looks something like this:

<storage-mapping>
  <lfn-to-pfn protocol="direct" destination-match=".*" is-custodial="y"
    path-match="(.*)" result="/data/custodial/$1"
    space-token="CMSCUSTODIAL" />
  <lfn-to-pfn protocol="direct" destination-match=".*"
    path-match="(.*)" result="/data/default/$1"
    space-token="CMSDEFAULT" />
  <lfn-to-pfn protocol="srm" destination-match=".*" chain="direct"
    path-match="(.*)" result="file:///$1"/>

  <pfn-to-lfn protocol="direct" destination-match=".*"
    path-match="/data/*(/.*)" result="$1"/>
  <pfn-to-lfn protocol="srm" destination-match=".*" chain="direct"
    path-match="file:/+(/.*)" result="$1"/>
</storage-mapping>

This is basically a set of substitution regular expressions and a set of rules to apply them in the right order. Ignoring the 'is-custodial' setting at the top, You can see how it works by using the test script that comes with the PhEDEx installation:

$PHEDEX_ROOT/Utilities/TestCatalogue -c tfc.xml -p srm -L /a/b/c/d

The -c tells it where to find the TFC. The -p tells it to use the SRM protocol for name lookups, and the -L tells it the LFN to look up. So for the catalogue above, the script first looks for the rule to convert an LFN to a PFN for the SRM protocol, and it finds one. That rule has a chain element, so it follows the chain to the next protocol (direct) and applies the rule there too. It only applies it to the path part of the PFN, not to the protocol. So, the output from that command should look like this:

Testing file name mappings in tfc.xml using protocol srm
LFN: /a/b/c/d
PFN: file:////data/default//a/b/c/d
TKN: CMSDEFAULT
Re-LFN: /default//a/b/c/d *** ERROR: result different from /a/b/c/d (/default//a/b/c/d)

Ignore the error message, that's OK in this situation. The key part is that you can see the LFN /a/b/c/d has been matched to a PFN file:////data/default//a/b/c/d

Now the fun part: Assuming you know what your PFNs should look like to get your FDT server to transfer them, you have to mess with the rules to make the PFN come out correctly for your LFNs. You can add new rules for new protocols, and the usual cut-n-paste, trial-and-error approach will probably work well enough. If not, send a mail to the ANSE list with details of your setup and someone can help you.

Configuring your download agent

You also need to configure a real download agent to perform the transfers. This has a number of steps.

You will need to modify the Config.ANSE file for your destination site so that it uses the FDT protocol. Start from the example that has LABEL=download-fts and modify that (you may want to make a backup first). Change the backend from FTS to FDT and remove the -server option. You should also set -jobs and -batch-files to one while debugging.

You need to edit the fake-validate.pl script that the agent uses. This is used by the download agent both before and after transferring the file (the pre and post conditions mentioned in the file). The idea is that it checks before a transfer, in case the file is already there and doesn't need to be sent. Then it checks again afterwards, to make sure the transfer really worked.

In the post condition, you want to actually delete the file, to prevent your disk from filling up. You will have the PFN of the file, so you can probably recover the LFN by a simple regular expression, then just unlink $file to delete it, and return zero to indicate success.

Subscribing the data

So far, so good, but the data won't move until you place a transfer request. At the moment, the web-server in Vanderbilt is only configured to recognise me, so I'll have to do that. If you're running a private testbed (that's you, Vlad!), go to your private web-server and create a transfer request for that dataset. Specify the dataset name fully, pick an arbitrary group for the transfer to belong to, and make the request. If the request works, you'll see a link to the page where you can then approve that request. Do that, then go for coffee, and in 20-30 minutes you should see transfer attempts.

After that, it's all about looking in the error logs and debugging things to see what happens. Go to the state directory of your download agent and look in the subdirectories there. The archive directory will hold the output from transfer jobs, but only for a few hours, they're purged after that. If things aren't working you should be able to find out why by poking around there. Ask on the list if you need help, there's more than one PhEDEx expert there so it's likely to be faster than mailing me directly.