How to set up a DPM pool node from scratch as used by Edinburgh

Guide for a head node here: https://twiki.cern.ch/twiki/bin/view/Main/DPMEdinburghManualInstall

Short summary

Need to setup and install: xrootd , httpd , gsiftp , rfiod

This means that the config files for each need to be setup (see below).

Trust (shift.conf) between the storage node and the dpm head node needs to be setup (again see below).

In order to support this `/etc/grid-security` needs to be setup such the voms information as well as the ca's and host cert need to be installed.

This means that `fetch-crl` needs to be run relatively frequently on the hosts and this is best done via cron.

After this you will only need to be running the 4 services mentioned above.

Setup the needed user

Add the dpmmgr account for managing/running dpm and dpns

groupadd -g 151 dpmmgr
useradd -c "DPM manager" -g dpmmgr -u 151 -r -m dpmmgr

Install needed packages

Compared to the head node we only need a subset of programs installed on the pool nodes for dpm:

cd /etc/yum.repos.d/
wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo
yum install epel-release
yum update
yum install dpm-xrootd dpm-rfio-server dpm-dsi dmlite-plugins-adapter lcg-CA python-dmlite fetch-crl edg-mkgridmap

Configure the needed services

Edit the following configs:

/etc/sysconfig/rfiod

RUN_RFIOD="yes"
export DPNS_HOST="srm-test.gridpp.ecdf.ed.ac.uk"
export DPM_HOST="srm-test.gridpp.ecdf.ed.ac.uk"
RFIOLOGFILE=/var/log/rfio/log

/etc/gridftp.conf

inetd 0
daemon 1
detach 1
chdir 1
fork 1
single 0

cas 1
secure_ipc 1
ipc_auth_mode host
allow_anonymous 0

log_level ERROR,WARN,INFO,TRANSFER
log_single /var/log/dpm-gsiftp/gridftp.log
log_transfer /var/log/dpm-gsiftp/dpm-gsiftp.log
disable_usage_stats 1
usage_stats_target usage-stats.globus.org:4810

data_node 0
stripe_blocksize 1048576
stripe_layout 2
stripe_blocksize_locked 0
stripe_layout_locked 0

blocksize 262144
sync_writes 0

port 2811

control_preauth_timeout 120
control_idle_timeout 600
ipc_idle_timeout 600
ipc_connect_timeout 600

banner_terse 0
login_msg "Disk Pool Manager (dmlite)"

load_dsi_module dmlite
use_home_dirs 1
debug 0

/etc/shift.conf

RFIOD TRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
RFIOD WTRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
RFIOD RTRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
RFIOD XTRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
RFIOD FTRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
DPM   TRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
DPNS  TRUST srm-test.gridpp.ecdf.ed.ac.uk pool90-glite.rdf.ac.uk
DPM PROTOCOLS rfio gsiftp xroot https
RFIO DAEMONV3_RDMT_BUFSIZE 524288
RFIO DAEMONV3_RDSIZE 524288

/etc/dmlite.conf.d/adapter.conf

LoadPlugin plugin_adapter_dpm /usr/lib64/dmlite/plugin_adapter.so
LoadPlugin plugin_fs_rfio /usr/lib64/dmlite/plugin_adapter.so
DpmHost srm-test.gridpp.ecdf.ed.ac.uk
NsHost srm-test.gridpp.ecdf.ed.ac.uk
ConnectionTimeout 15
RetryLimit 3
RetryInterval 2
TokenPassword supersecretweasel
TokenId ip
TokenLife 1000
AdminUsername

/etc/xrootd/xrootd-dpmdisk.cfg

ofs.trace all
xrd.trace all
cms.trace all
oss.trace all

all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
xrd.network nodnr


xrootd.monitor all flush 30s fstat 60 lfn ops xfr 5 window 5s dest fstat info user redir atlas-fax-eu-collector.cern.ch:9330
xrootd.async off

if exec xrootd
 xrootd.seclib /usr/lib64/libXrdSec.so
 sec.protocol /usr/lib64 gsi -crl:3 -key:/etc/grid-security/dpmmgr/dpmkey.pem -cert:/etc/grid-security/dpmmgr/dpmcert.pem -md:sha256:sha1 -ca:2 -gmapopt:10 -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -gridmap:/etc/lcgdm-mapfile
 sec.protocol /usr/lib64 unix
 xrootd.export /
 xrd.port 1095
 xrd.report atl-prod05.slac.stanford.edu:9931 every 60s all -buff -poll sync
 ofs.osslib libXrdDPMOss.so.3
 ofs.authlib libXrdDPMDiskAcc.so.3 
 ofs.authorize
 ofs.persist auto hold 0
 ofs.tpc pgm /usr/bin/xrdcp --server
 all.role server
 xrootd.chksum max 3 adler32 /usr/bin/chksumadler32.sh
fi


if exec cmsd
all.role server
fi

dpm.nohv1
if exec xrootd
fi
dpm.dmconf /etc/dmlite.conf

httpd

This is setup the same way as the head node with very little modifications to the settings. The service should run as `dpmmgr` and the dpm config should handle the ssl and davs.

/etc/xrootd/dpmxrd-sharedkey.dat Needs to be copied from the head node (in this case srm-test.gridpp.ecdf.ed.ac.uk)

/usr/bin/chksumadler32.sh

#!/bin/bash

touch /tmp/outputlog
date &>> /tmp/outputlog
# Define local prefix used with DPM
PREFIX="/dpm/ecdf.ed.ac.uk/home"
whoami >>/tmp/outputlog
# LFN to PFN conversion, needs dpm-dpns-to-disk tool
# If dpm-dpns-to-disk not installed locally, connect to one that has (srm.glite here which is DPM headnode)
prefixpresent=$(echo $1|grep -c "$PREFIX")
if [ "$prefixpresent" == "1" ];
then
 localfile=$(ssh root@srm.glite "dpm-dpns-to-disk $1|cut -d: -f2")
else
 localfile=$(ssh root@srm.glite "dpm-dpns-to-disk ${PREFIX}$1|cut -d: -f2")
fi
echo $1 >>/tmp/outputlog
echo file=$localfile>>/tmp/outputlog
#calculate checksum and store in array
chksumoutput=($(xrdadler32 $localfile 2>>/tmp/outputlog))

#give out checksum which is in first field of array
echo chksum=${chksumoutput[0]}>>/tmp/outputlog
echo file=${chksumoutput[1]}>>/tmp/outputlog
echo ${chksumoutput[0]}

Edit the iptables

On Centos7 first install iptables-services and disable firewalld:

yum install iptables-services
systemctl stop firewalld
systemctl disable firewalld
systemctl enable iptables
systemctl start iptables

Add the appropriate firewall rules:

iptables -A INPUT -p tcp -m multiport --dports 5015 -m comment --comment "050 allow DPM" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 5010 -m comment --comment "050 allow DPNS" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 1094 -m comment --comment "050 allow cmsd" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 2811 -m comment --comment "050 allow gridftp control" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 20000:25000 -m comment --comment "050 allow gridftp range" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 80,443 -m comment --comment "050 allow http and https" -j ACCEPT
iptables -p tcp -m multiport --dports 5001 -m comment --comment "050 allow rfio" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 5001 -m comment --comment "050 allow rfio" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 20000:25000 -m comment --comment "050 allow rfio range" -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 1095 -m comment --comment "050 allow xrootd" -m state --state NEW -j ACCEPT
iptables -S
service iptables save

Adding the filesystem to the dpm head node:

From the dpm headnode:

dpm-addfs --poolname atlas --server POOL_SERVER --fs /PATH/TO/STORAGE --st 0

Testing the protocols

First we need to setup a proxy for authentication. This needs to be for the dteam VO:

voms-proxy-init --cert=/root/robs-cert/usercert.pem --key=/root/robs-cert/userkey.pem --voms=dteam

To test all of the supported protocols for this setup:

voms-proxy-init --cert=/root/robs-cert/usercert.pem --key=/root/robs-cert/userkey.pem --voms=dteam

-- RobCurrie - 2017-04-02