Enable HTTP protocol for XRootD

Configuration

The xrootd daemon can export both the xrootd protocol and the https protocol; this page covers the configuration changes required for enabling HTTPS.

For clients to successfully read from the regional redirector, HTTPS must be enabled for the data servers and the site-level redirector.

Versions and Platforms

• XROOTD 5.X

You can follow one of the installation instructions in one of the links appropriate for your backend SE in the 'For Admin' section in CMSXrootDArchitecture

• XROOTD 4.X

We strongly suggest utilizing RHEL7 and XRootD 4.9.1.

The latest version of xrootd-lcmaps (1.3 or later) has features required by CMS and doesn't support RHEL6.

Xrootd Config Changes

Verify you have the most up-to-date version of the sec.protocol line:

• XROOTD 5.X

  sec.protocol configuration with lcmaps 

sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates \
                        -cert:/etc/grid-security/xrd/xrdcert.pem \
                        -key:/etc/grid-security/xrd/xrdkey.pem \
                        -crl:3 \
                        -authzfun:libXrdLcmaps.so \
                        -authzfunparms:lcmapscfg=/etc/lcmaps.db,loglevel=3 \
                        -gmapopt:10 -gmapto:0

  sec.protocol configuration without lcmaps (e.g. OSG 3.6) 

sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates \
                        -cert:/etc/grid-security/xrd/xrdcert.pem \
                        -key:/etc/grid-security/xrd/xrdkey.pem \
                        -crl:1 \
                        -gmapopt:trymap \
                        -gridmap:/etc/grid-security/grid-mapfile \
                        -vomsfun:default

In the xrootd daemon specific configuration, enable HTTPS:

  https/davs configuration with lcmaps 

if exec xrootd
   xrd.protocol http:1094 /usr/lib64/libXrdHttp.so
   http.cadir /etc/grid-security/certificates
   http.cert /etc/grid-security/xrd/xrdcert.pem
   http.key /etc/grid-security/xrd/xrdkey.pem
   http.secxtractor /usr/lib64/libXrdLcmaps.so
   http.listingdeny yes
   http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
   http.desthttps yes
   # Enable third-party-copy
   http.exthandler xrdtpc libXrdHttpTPC.so
   http.exthandler xrdmacaroons libXrdMacaroons.so
   # Pass the bearer token to the Xrootd authorization framework.
   http.header2cgi Authorization authz
   #http.trace all
fi

  https/davs configuration without lcmaps (e.g., OSG 3.6)

if exec xrootd
   xrd.protocol http:1094 /usr/lib64/libXrdHttp.so
   http.cadir /etc/grid-security/certificates
   http.cert /etc/grid-security/xrd/xrdcert.pem
   http.key /etc/grid-security/xrd/xrdkey.pem
   http.listingdeny yes
   http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
   http.desthttps yes
   # Enable third-party-copy
   http.exthandler xrdtpc libXrdHttpTPC.so
   http.exthandler xrdmacaroons libXrdMacaroons.so
   http.header2cgi Authorization authz
   http.gridmap /etc/grid-security/grid-mapfile
   #http.secxtractor /usr/lib64/libXrdVoms.so
   http.secxtractor /usr/lib64/libXrdHttpVOMS.so
fi

• XROOTD 4.X

sec.protocol /usr/lib64 gsi \
   -certdir:/etc/grid-security/certificates \
   -cert:/etc/grid-security/xrd/xrdcert.pem \
   -key:/etc/grid-security/xrd/xrdkey.pem \
   -crl:1 \
   -authzfun:libXrdLcmaps.so \
   -authzfunparms:--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0 \
   -gmapopt:10 \
   -gmapto:0

In the xrootd daemon specific configuration, enable HTTPS:

if exec xrootd
  xrd.protocol http:1094 /usr/lib64/libXrdHttp-4.so
  http.cadir /etc/grid-security/certificates
  http.cert /etc/grid-security/xrd/xrdcert.pem
  http.key /etc/grid-security/xrd/xrdkey.pem
  http.secxtractor /usr/lib64/libXrdLcmaps.so
  http.listingdeny yes
  http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
  http.desthttps yes
fi

Updates to Auth database contents

The /etc/xrootd/Authfile should contain these lines:

g /cms /store lr

Other VOs may need additional authorizations; note that the VOMS attributes are mapped to the XRootD group structure. The above example only covers reads; writes for CMS are covered later in the document.

In particular, it should not have a line that looks like this:

u * /store lr

Create robots.txt

Add the file robots.txt to /etc/xrootd with these contents:

User-agent: *
Disallow: /

We recommend doing this in order to prevent search engines from trying to index your site.

Configuration of storage.xml for davs protocol Mapping

Update storage.xml to have a davs protocol. Add a line that looks like this:

  <lfn-to-pfn protocol="davs"  destination-match=".*" chain="direct" path-match="(.*)" result="davs://xrootd-local.unl.edu:1094/$1"/>

  <lfn-to-pfn protocol="davs"  destination-match=".*" chain="direct" path-match=".*/LoadTest07_Nebraska_(.*)_.*_.*" result="davs://xrootd-local.unl.edu:1094/user/uscms01/pnfs/unl\.edu/data4/cms/(.*)/$1"/>

Update as appropriate for your site. Remember to synchronize this with CVMFS and any other place you store your storage.xml.

Testing the configuration

Make sure you see file not found from any browser session, even if you have your certificate loaded. This should fail because a VOMS extension isn’t present in your browser; we want to require VOMS.

From the terminal, generate a CMS VOMS proxy and attempt to use davix-get to copy from your XRootD host:

davix-get https://cmstest2.rcac.purdue.edu:1094/store/user/goughes/test.root -P grid

Do not use curl as it does not support proxy certificates.

Enable HTTP-based Writes

Xrootd Authorization changes

The primary changes are to the /etc/xrootd/Authfile; you will need to add several a (all) authorizations to where users need to be able to write. Here's an example:

t writecmsdata /store/backfill/          a \
               /store/data/                a \
               /store/express/          a \ 
               /store/generator/       a \
               /store/group/rucio/     a \
               /store/hidata/            a \
               /store/himc/              a \
               /store/hirelval/          a \
               /store/mc/                a \
               /store/relval/             a \
               /store/results/          a \ 
               /store/temp/             a \
               /store/test/loadtest/  a \
               /store/test/rucio/       a \
               /store/unmerged/     a \
               /store/user/rucio/      a

t readcmsdata  /store/                   lr

# cmsprod and PhEDEx have full access to managed CMS data, and read for CMS
u cmsprod  writecmsdata readcmsdata
u cmsphedex writecmsdata readcmsdata

# lcgadmin can write into /store/user/sam and /store/unmerged/SAM.
u lcgadmin /store/user/sam/ a /store/unmerged/SAM a readcmsdata

# CMS users have full access to their own directory, and read for CMS
# While xrootd allows the user to *attempt* any operation - even in other user's
# home directories - the underlying filesystem also has its internal permissions and will further
# limit things.
g /cms /store/user a /store/temp/ a readcmsdata

Notes:

• This guide assumes that you have an underlying filesystem (HDFS, Lustre, etc) implementing filesystem permissions for /store/user directories. If not (pure Xrootd, files are written as the same user), you will need one line per CMS username.
• List authorizations from most specific to least specific.
• The first two columns must be unique; if multiple authorizations are needed for a user, add them to the same line.
• t is a template
• u lines are for users as mapped by LCMAPS (such as cmsprod). These should correspond to Unix usernames at your site.
• g lines refer to VOMS groups (such as /cms). These do not correspond to Unix group names at your site.
• If you have u *, recall this allows ALL users, including unauthenticated. This includes random web spiders!

The upstream documentation has further information on the Authfile format.

Xrootd Checksum changes

In order for the checksumming functionality to work, additional RPM upgrades may be needed:

• xrootd-4.9.1 or later is needed (as of April 2019, this is in the OSG release repo)
• xrootd-hdfs-2.1.3 or later. Currently in the OSG release; only necessary for HDFS sites.

Additionally, we need to enable checksums in the configuration file:

xrootd.chksum max 2 md5 adler32 crc32
# Below line is only necessary for HDFS sites
ofs.ckslib * /usr/lib64/libXrdHdfs.so
# For POSIX filesystems, uncommand the following and comment above
#ofs.ckslib * libXrdMultiuser.so

CMS-specific changes

• Update storage.xml to have a davs protocol. Add a line that looks like this:

    <lfn-to-pfn protocol="davs"  destination-match=".*" chain="direct" path-match="(.*)" result="davs://xrootd-local.unl.edu:1094/$1"/>
  

• Update the site-local-config.xml to point the stageout protocol to davs. Inside the <local-stage-out> stanza, convert the catalog protocol to davs. Example:

     <catalog url="trivialcatalog_file://etc/cvmfs/SITECONF/PhEDEx/storage.xml?protocol=davs"/>
  

It is best to test this out on a few worker nodes before deploying widely. SITECONF update example from Caltech: https://gitlab.cern.ch/SITECONF/T2_US_CALTECH/commit/edc766852de59923c075061e04e8e8d572e94276

Testing the configuration

From the terminal, generate a CMS VOMS proxy and attempt to use davix-get to copy from your XRootD host:

davix-put /tmp/hello_world.txt https://cmstest2.rcac.purdue.edu:1094/store/user/goughes/test.root -E /tmp/x509up_u28772 --capath /etc/grid-security/certificates

OR

gfal-copy  /tmp/hello_world.txt https://transfer-1.ultralight.org:1094//store/temp/user/localtest.1/hello_world.txt
Copying file:///tmp/hello_world.txt   [DONE]  after 4s

Do not use curl as it does not support proxy certificates.

Enable Third-Party-Copy

The OSG has third-party-copy support for XRootD; this section walks through enabling the support.

Install RPMs

You will need the XRootD 4.9.1 (or later) release.

If you are a HDFS site, you will also to have xrootd-hdfs version 2.1.3 or later; this is only in OSG 3.4.

Configuration Changes

If you have a line like this in your XRootD config, remove it:

xrootd.fslib /usr/lib64/libXrdOfs.so

This is no longer necessary and breaks the configuration for the third party copy daemon.

Add the following lines for the xrootd daemon:

if exec xrootd
# Enable third-party-copy
http.exthandler xrdtpc libXrdHttpTPC.so

# Pass the bearer token to the Xrootd authorization framework.
http.header2cgi Authorization authz
fi

# Enable Macroons-based mappings; if no token is present, then the GSI certificate will be used.
ofs.authlib libXrdMacaroons.so

Follow the Macaroons support configuration section below to finalize the Macaroons pieces.

Testing Setup

You may utilize the FTS server at https://fts3-devel.cern.ch:8446.

fts-transfer-submit --overwrite -s https://fts3-devel.cern.ch:8446 \
    https://xrootd-local.unl.edu:1094//user/uscms01/pnfs/unl.edu/data4/cms/store/mc/RunIISpring18wmLHEGS/TTJets_DiLept_TuneCP5_13TeV-madgraphMLM-pythia8/GEN-SIM/100X_upgrade2018_realistic_v10-v1/100000/FC337F4C-BB22-E811-B941-0242AC130002.root \
    davs://transfer-8.hep.caltech.edu:1094//store/user/bbockelm/test/writes_new/scitokens.8 \
    --blocking --verbose

Macaroons Support

Macaroons are a token format that allows delegation of authorization and attenuation. They are a convenient way to enable a user to upload or download a single file without having a grid proxy and are used by FTS to perform third party copies.

Install the plugin

This is installed by default with XRootD 4.9.1 or later.

Configuration

Add the following lines to your xrootd configuration:

http.exthandler xrdmacaroons libXrdMacaroons.so
macaroons.secretkey /etc/xrootd/macaroon-secret
ofs.authlib libXrdMacaroons.so libXrdAccSciTokens.so

The secret key is a symmetric key necessary to verify macaroons; the same key must be deployed to all XRootD servers in your cluster (so puppetize its distribution).

The secret key must be base64-encoded. The most straightforward way to generate this is the following:

openssl rand -base64 -out /etc/xrootd/macaroon-secret 64

NOTE: The current implementation is sensitive to errant newline characters. Use the openssl command above and try to avoid editing with a text editor.

Usage

To generate a macaroon for personal use, you can run:

macaroon-init -v
Usage: macaroon-init URL validity_min ACTIVITY ...

Here's an example. You will need to change the parameters for your needs. Make sure you have a voms proxy first.

macaroon-init https://g34n02.hep.wisc.edu:1094/store/user/cvuosalo 60 DOWNLOAD,UPLOAD

This will generate a macaroon with 60 minutes of validity that has upload and download access to the path (/store/user/cvuosalo in this example), provided that your X509 identity has that access. Note that there must be only one slash between the 1094 and the directory name (store in this example). There was an earlier version of macaroon-init that allowed two slashes (e.g. https://g34n02.hep.wisc.edu:1094//store/user/cvuosalo), but the current version allows only one slash. The macaroon-init CLI can be found as part of the x509-scitokens-issuer-client package.

The output will look like the following:

MDAxZGxvY2F0aW9uIFQyX1VTX1dpc2NvbnNpbgowMDM0aWRlbnRpZmllciBlMjE5MjU0Ni1hOWE1LTQwMDYtODRhNS1kYTFjOTgxNzFhOTIKMDAxNWNpZCBuYW1lOmNtc3Byb2QKMDA1MmNpZCBhY3Rpdml0eTpSRUFEX01FVEFEQVRBLFVQTE9BRCxET1dOTE9BRCxERUxFVEUsTUFOQUdFLFVQREFURV9NRVRBREFUQSxMSVNUCjAwMjFjaWQgYWN0aXZpdHk6RE9XTkxPQUQsVVBMT0FECjAwMjJjaWQgcGF0aDovc3RvcmUvdXNlci9jdnVvc2FsbwowMDI0Y2lkIGJlZm9yZToyMDIyLTA5LTA5VDE3OjUxOjMzWgowMDJmc2lnbmF0dXJlIE9Bufe3-36Rwo3tGZbD8spG-iCs5IkoaFy0yrnRuxfgCg

The contents of the macaroon key is your new security token. Anyone you share it with will be able to read and write from the same path. You can use this token as a bearer token for HTTPS authorization. For example, it can authorize the following transfer:

curl -v -H "Authorization: Bearer MDAxZGxvY2F0aW9uIFQyX1VTX1dpc2NvbnNpbgowMDM0aWRlbnRpZmllciBlMjE5MjU0Ni1hOWE1LTQwMDYtODRhNS1kYTFjOTgxNzFhOTIKMDAxNWNpZCBuYW1lOmNtc3Byb2QKMDA1MmNpZCBhY3Rpdml0eTpSRUFEX01FVEFEQVRBLFVQTE9BRCxET1dOTE9BRCxERUxFVEUsTUFOQUdFLFVQREFURV9NRVRBREFUQSxMSVNUCjAwMjFjaWQgYWN0aXZpdHk6RE9XTkxPQUQsVVBMT0FECjAwMjJjaWQgcGF0aDovc3RvcmUvdXNlci9jdnVvc2FsbwowMDI0Y2lkIGJlZm9yZToyMDIyLTA5LTA5VDE3OjUxOjMzWgowMDJmc2lnbmF0dXJlIE9Bufe3-36Rwo3tGZbD8spG-iCs5IkoaFy0yrnRuxfgCg
" https://g34n02.hep.wisc.edu:1094//store/user/cvuosalo/testfile.txt

For debugging, you can read a macaroon using this web page: https://www.base64decode.org/

Enable SciTokens

Please refer to the documentation here: https://twiki.cern.ch/twiki/bin/view/CMSPublic/XRootDAndTokens