Xrootd Server

This is a guide to using the xrootd server on the titan UMass cluster. Xrootd distributes data files from a dataset over all available nodes, which allows each node to take responcibility for passing around data, rather than just the "head" node. This reduces or eliminates wait time, among other benefits.

I will be adding to this wiki as our experience with the server grows, please email me if you have any questions (ameade@physicsNOSPAMPLEASE.umass.edu).

Viewing Files

/titan/atlas/ is a read only version of the xrootd server, for example you can use "ls" to look at what directories and datasets exist. How the datasets are actually distributed between the nodes is hidden. If you want this information, you can look at /titan/atlas/titanXX, where XX should be 11-26. If one of the nodes is NOT listed here, it isn't properly configured, and you shouldn't go forward, instead contact bbrau@physicsNOSPAMPLEASE.umass.edu and myself.

UPDATE

Currently, you should be using /titan_rw/ to download files using dq2-get and cp. As far as I can see, this works fine without the xprep step listed below. Make sure all the Nodes are up before you start copying files, otherwise the files will not be evenly distributed, and performance will suffer.

When running over files, all you need to know is that when accessing the files you need to use root://titan.physics.umass.edu//atlas/ instead of /titan_rw/atlas/.

Contact Ben or I if there are problems.

Most of the below information is obselete or depreciated.

Copying Data

Copying a dataset is a two step process, you use "xprep" to prepare the server for what files are being added, and then you use "xrdcp" to copy the dataset.

xprep -w -f /tmp/files.txt titan.physics.umass.edu

where /tmp/files.txt has one line per file listing where in the xrootd filespace the files are to go, e.g.:

/atlas/local/ameade/mc10_7TeV.107660.AlpgenJimmyZmumuNp0_pt20.merge.NTUP_SMWZ.e737_s933_s946_r2302_r2300_p605_tid441395_00/NTUP_SMWZ.441395._000643.root.1
/atlas/local/ameade/mc10_7TeV.107660.AlpgenJimmyZmumuNp0_pt20.merge.NTUP_SMWZ.e737_s933_s946_r2302_r2300_p605_tid441395_00/NTUP_SMWZ.441395._000644.root.1

etc.

Then:

xrdcp /data/atlas/data/mc10b_7TeV/WZ_D3PD/mc10_7TeV.107660.AlpgenJimmyZmumuNp0_pt20.merge.NTUP_SMWZ.e737_s933_s946_r2302_r2300_p605_tid441395_00/ root://titan.physics.umass.edu//atlas/local/ameade/mc10_7TeV.107660.AlpgenJimmyZmumuNp0_pt20.merge.NTUP_SMWZ.e737_s933_s946_r2302_r2300_p605_tid441395_00/

Some datasets have already been copied to /titan/atlas/local/mc10b_7TeV/WZ_D3PD/.

In order to produce the list, one approach is to use ls /path/to/dataset/ &> file.txt and then use perl -i.old -pe 's/old/new/g' filetochange.txt, in order to do an inplace edit. Keep in mind you need to use a backslash in order to edit forward slashes (i.e. "\/").

dq2-get

So far we have two ways of using dq2-get to write directly into the xrootd server.

r/w xrootdfs mount

/titan_rw/atlas/common has been mounted read/write. This means in this space you can write and delete files as you wish. So you can use dq2 as usual directly into this space. This works, but it is unclear right now if there will be any performance impact due to the fact that no "xprep" step is performed. Update As far as I can tell, this way of getting files still has very good performance. We should use this method as our primary method of getting files for now, and put all files under /common in order to reduce duplication of datasets.

setupATLAS
localSetupDQ2Client --skipConfirm
cd /titan_rw/atlas/common
dq2-get mc11_valid.113642.Pythia8JetsJ7.recon.log.e789_a124_tid365181_00

Note - if when running the dq2-get command and you see this error -

Querying DQ2 central catalogues for files in dataset...
[Errno 110] Connection timed out: 'mc11_valid.113642.Pythia8JetsJ7.recon.log.e789_a124_tid365181_00'
No download attempts made

It means that there is a problem with the xrootdfs mount. (likely a data server went down and xrootdfs lost communication with it).

dq2-get + asyncronous transfer + xprep

Doug Benjamin wrote a development version of dq2-get under /data/share/atlas/DQ2ClientsTesting/. This version does xprep on the fly. It also uses an asyncronous transfer, which means the files are not downloaded immediately as the command is made. Rather, they are downloaded in the background, and the process continues even if the terminal session ends, in principle until the dataset is complete. It is not yet clear where/if there is a log of the process to confirm when the process is completed, this is being looked into.

You can get any dataset that is under the xrootd federation. Soon this will be all Tier 1 and Tier 2 sites in the US.

setupATLAS
localSetupDQ2Client --skipConfirm
source /data/share/atlas/DQ2ClientsTesting/setup.sh
dq2-get  -L ROAMING  -q XPREP -S titan:1094 mc11_valid.113642.Pythia8JetsJ7.recon.log.e789_a124_tid365181_00

Notice where the file is placed:

ls  /titan/atlas/dq2/mc11_valid/log/e789_a124/mc11_valid.113642.Pythia8JetsJ7.recon.log.e789_a124_tid365181_00/

Running Over Xrootd Data

If you use TFile, you need to you a TXNetFile instead, which is a specialization:

TXNetFile* temp = new TXNetFile(filepath,"read","", 1, 0, kFALSE, 0);

You can use TChain::Add as usual:

mychain.AddFile(filepath);

In both cases, "filepath" must be of the following form:

root://titan.physics.umass.edu//atlas/local/mc10b_7TeV/WZ_D3PD/mc10_7TeV.107661.AlpgenJimmyZmumuNp1_pt20.merge.NTUP_SMWZ.e737_s933_s946_r2302_r2300_p605_tid441396_00/NTUP_SMWZ.441396._000001.root.1

Acknowledgements

Much of the info on this page is thanks to Ben Brau and Doug Benjamin, who have done the bulk of the work setting up the xrootd server.

More information

-- AndrewMeade - 21-Sep-2011