THIS PAGE IS BEING DECOMMISSIONED - PLEASE DON'T EDIT - USE ITS SUCCESSOR HERE

Joining a HDFS-based SE to the Xrootd service.

This document covers joining the USCMS Xrootd service based on the redirector xrootd-itb.unl.edu for HDFS sites. This assumes you are using Hadoop 0.20 or later.

Installation

You must already have HDFS working and configured on the node. A FUSE mount is not needed; however, you should be able to use hadoop -put and hadoop -get to move files in and out of HDFS on the node. If the node is already a functioning HDFS GridFTP server, then it probably meets these requirements.

First, install the OSG software repository. For SL6:

rpm -Uhv http://repo.grid.iu.edu/osg-el6-release-latest.rpm

For SL5:

rpm -Uhv http://repo.grid.iu.edu/osg-el5-release-latest.rpm

Next, install the xrootd RPM. This will add the xrootd user if it does not already exist - sites using centralized account management may want to create this user beforehand.

yum install --enablerepo=osg-testing,osg-contrib cms-xrootd-hdfs

The version should be at least 3.2.2.

Warning: The CMS transition to 3.1.0 from previous versions is not a clean upgrade (as we switched to the CERN-based packaging). We believe this is a one-time-only event. Unfortunately, folks will need to remove all local copies of xrootd, lcmaps, lcas, xrootd-lcmaps, xrootd-cmstfc, and lcas-lcmaps-gt4-interface before installing.

If your CMS namespace is not truly trivial (i.e., if the CMS top-level directory in Hadoop is not /store), copy your storage.xml to /etc/xrootd/storage.xml. Then install the CMS TFC parser:

yum install --enablerepo=osg-contrib xrootd-cmstfc

Make sure your storage.xml exports a hadoop protocol (which should provide the PFN relative to your storage system; see Nebraska's TFC as inspiration if necessary). If you aren't using the hadoop protocol in your TFC, you can edit the sample configuration file to pick a protocol of your liking.

Copy the template config file, /etc/xrootd/xrootd.sample.hdfs.cfg to /etc/xrootd/xrootd-clustered.cfg. If your site requires storage.xml, uncomment (and possibly update) the oss.namelib line.

Finally, create a copy of the host certs to be xrootd service certs:

mkdir -p /etc/grid-security/xrd
cp /etc/grid-security/hostcert.pem /etc/grid-security/xrd/xrdcert.pem
cp /etc/grid-security/hostkey.pem /etc/grid-security/xrd/xrdkey.pem
chown xrootd: -R /etc/grid-security/xrd
chmod 400 /etc/grid-security/xrd/xrdkey.pem # Yes, 400 is required

Integrating with GUMS, Argus, or SCAS

In order to integrate xrootd with GUMS (v1.3 or higher), Argus, or SCAS, install the following RPM:

yum install xrootd-lcmaps

This will bring in several dependencies, including Globus libraries, from the OSG. These do not appear to conflict with gLite installs of these libraries, but please be careful.

Next, copy/paste the following line from /etc/xrootd/lcmaps.cfg into /etc/xrootd/xrootd-clustered.cfg:

# sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/xrd/xrdcert.pem -key:/etc/grid-security/xrd/xrdkey.pem -crl:3 -authzfun:libXrdLcmaps.so -authzfunparms:--osg,--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0|useglobals -gmapopt:10 -gmapto:0

Uncomment the line in xrootd-clustered.cfg, of course.

For GUMS or SCAS, update the /etc/xrootd/lcmaps.cfg provided in the RPM so the endpoint properly references your server's XACML endpoint. For Argus, use the attached lcmaps.cfg.

If this is a brand new host, you may need to run fetch-crl to update CRLs before starting Xrootd.

Operating xrootd

There are two init services, xrootd and cmsd, which must both be working for the site to participate in the xrootd service:

service xrootd start
service cmsd start

Everything is controlled by a proper init script (available commands are start, stop, restart, status, and condrestart). To enable these on boot, run:

chkconfig --level 345 xroot on
chkconfig --level 345 cmsd on

Log files are kept in /var/log/xrootd/{cmsd,xrootd}.log, and are auto-rotated.

After startup, the xrootd and cmsd daemons drop privilege to the xrootd user.

If you used the RPM version of fetch-crl, you will need to enable and start the fetch-crl-cron and fetch-crl-boot services. To start:

service fetch-crl-cron
service fetch-crl-boot # This may take awhile to run

To enable on boot:

chkconfig --level 345 fetch-crl-cron on
chkconfig --level 345 fetch-crl-boot on

Port usage:

The following information is probably needed for sites with strict firewalls:

• The xrootd server listens on TCP port 1094.
• The cmsd server needs outgoing TCP port 1213 to xrootd.unl.edu.
• Usage statistics are sent to xrootd.t2.ucsd.edu on UDP ports 3333 and 3334.

Testing the install.

The newly installed server can be tested directly using:

xrdcp -d 1 -f xroot://local_hostname.example.com//store/foo/bar /dev/null

You will need a grid certificate installed in your user account for the above to work

You can then see if your server is participating properly in the xrootd service by checking:

xrdcp root://xrootd-itb.unl.edu//store/foo/bar /tmp/bar2

where /store/foo/bar is unique to your site

THIS PAGE IS BEING DECOMMISSIONED - PLEASE DON'T EDIT - USE ITS SUCCESSOR HERE