Tips on using the Grid

Getting ready with your Grid certificate

It is very likely that once you have obtain you grid certificate, you end up with a .p12 file (after you have exported your grid-certificate from your web browser). I copy here the instruction suggested CERN-CA (How to use your certificate with grid-proxy-init ):

openssl pkcs12 -in myGridCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
openssl pkcs12 -in myGridCert.p12 -nocerts -out $HOME/.globus/userkey.pem
chmod go-rw $HOME/.globus/userkey.pem

Copying files using srmcp (SRM Storage Resource Management)

There are many ways to copy files across the Grid. I describe here an example on using srmcp:

• get on lxplus (or the UI) and set your grid environment: source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.csh
• initiate a voms proxy: voms-proxy-init -voms cms
• issue the command: srmcp options source target

If you type srmcp --help you will get a full description on the available options (help-srmcp.txt).

The options are the key point. We have to use SRM V.2 and tell srmcp to use a script that comes with the distribution (url-copy.sh):

-srm_protocol_version=2 -use_urlcopy_script=true -urlcopy= ${SRM_PATH}/sbin/url-copy.sh

• Example 1: if you want to copy a file located at CERN ( srm://srm-cms.cern.ch:8443/ ) you will do:

<lxplus231> srmcp -srm_protocol_version=2 -use_urlcopy_script=true -urlcopy=${SRM_PATH}/sbin/url-copy.sh \
srm://srm-cms.cern.ch:8443/castor/cern.ch/user/r/rebassoo/CSAMuonStewCombinedWCSA/MuonStewNtupleCombined2.root \
file:////tmp/aosorio/test.root 

• Example 2: if the file is at FNAL ( srm://cmssrm.fnal.gov:8443/srm/managerv2\?SFN= ):

<lxplus231> srmcp -srm_protocol_version=2 -use_urlcopy_script=true -urlcopy= ${SRM_PATH}/sbin/url-copy.sh \
srm://cmssrm.fnal.gov:8443/srm/managerv2\?SFN=/2/rebassoo/JR/simulation-4mu-expoPt-30-0-200-1000-72.root \
file:////tmp/aosorio/testfile.root

• This last command has been tested in our Tier-3

From FNAL to CERN

• Example 3: from local FNAL to CERN Castor

You will first to configure Castor so it is able to receive the files. You need to create a special directory, call it gridfiles in your personal area and give it some special permissions:

<lxplus231> nsmkdir /castor/cern.ch/user/a/aosorio/gridfiles
<lxplus231> nschmod 775 /castor/cern.ch/user/a/aosorio/gridfiles

Now, you should be able to copy files across using the srmcp tool as described before:

<cmslpc07> srmcp -srm_protocol_version=2 -use_urlcopy_script=true -urlcopy= ${SRM_PATH}/sbin/url-copy.sh \
file:////uscms_data/d1/rebassoo/LM1SusyTree/job/LM1_SUSY_Tree.root \
srm://srm-cms.cern.ch:8443/castor/cern.ch/user/a/aosorio/gridfiles/LM1_SUSY_Tree.root

From CERN to Uniandes

• Example 4: from local CERN to Uniandes

You will first to configure your user area at the Tier-3 Uniandes so it is able to receive the files. You need to create a directory, for instance call it gridfiles in your personal area and give it some special permissions (you need first to start a grid-proxy):

<lxplus231> rfmkdir /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles
<lxplus231> rfchmod 775 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles

• Now, you should be able to copy files across using the srmcp tool as described before. We first try with a local file in your castor area:

<cmslpc07> srmcp -srm_protocol_version=2 -use_urlcopy_script=true -urlcopy= ${SRM_PATH}/sbin/url-copy.sh \
file:////afs/cern.ch/user/a/aosorio/Work/grid/Tier3-data/rpctt_results.root \
srm://moboro.uniandes.edu.co:8446/srm/managerv2\?SFN=/dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/file1.root

Things would work silently. You can then list the contents of your remote area:

<lxplus209> srmls --count=10 srm://moboro.uniandes.edu.co:8446/srm/managerv2\?SFN=/dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles
WARNING: SRM_PATH is defined, which might cause a wrong version of srm client to be executed
WARNING: SRM_PATH=/afs/cern.ch/project/gd/LCG-share/3.1.25-0/d-cache/srm
  0 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/
      0 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/SUSY/
      4166 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/file1.root
      4166 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/file2.root
      0 /dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/ganga/

Note the "--count=10 option": this tells srmls a maximum number of lines to display.

• Example 5: from your CERN Castor area to Uniandes

The same principle applies here. However this time you will need use the SRM protocol at both ends:

<lxplus209> srmls srm://srm-cms.cern.ch:8443/castor/cern.ch/user/a/aosorio/gridfiles/tier3/rpctt_results.root
WARNING: SRM_PATH is defined, which might cause a wrong version of srm client to be executed
WARNING: SRM_PATH=/afs/cern.ch/project/gd/LCG-share/3.1.25-0/d-cache/srm
  4166 /castor/cern.ch/user/a/aosorio/gridfiles/tier3/rpctt_results.root

For the moment this does not work ... sorry.

• Example 6: from your Tier-2 to Uniandes

lxplus230> ./copy-files-SRM-2-SRM.py susy_PatLayer1_LM0.txt 

Copying files: 

srmcp -srm_protocol_version=2 -use_urlcopy_script=true -urlcopy= ${SRM_PATH}/sbin/url-copy.sh \
srm://grid-srm.physik.rwth-aachen.de:8443/srm/managerv2\?SFN=/pnfs/physik.rwth-aachen.de/cms/store/group/user/susy/fronga/SUSY_LM0-sftsht/SUSY_LM0_229_SUSYPAT_V5_v1/00a24c1fa2b61c755cc684c50e9bc48c/patLayer1_9.root \
srm://moboro.uniandes.edu.co:8446/srm/managerv2\?SFN=/dpm/uniandes.edu.co/home/cms/user/a/aosorio/gridfiles/SUSY/Summer09/SUSY_LM0_229_SUSYPAT_V5_v1/patLayer1_9.root

this procedure seems to be working fine!

Places to look for data

The logical file name is not the complete story, and if you want to perform a srmls command, there is usually some path you have to prepend to the lfn. Here is a list of some of the sites and their particular path where the actual files are located.

Copying files using LCG commands

The following is a brief description on using the LCG commands in order to replicate / copy files to our Tier-3 in Bogota. This was done with the kind help from Michael Perez our storage expert and suport in Uniandes (thanks). The first thing you need to do is to make sure you have the following environment variables at your UI:

[michael@yali ~]$ echo $LFC_HOST; echo $LCG_GFAL_INFOSYS; echo $LCG_CATALOG_TYPE;
prod-lfc-cms-central.cern.ch
exp-bdii.cern.ch:2170
lfc

• Create then a valid grid proxy for this session. We will use the command lcg-cr (have a look to the LCG-2 manual pages or just type lcg-cr to get some information):

[aosorio@yali ~]$ lcg-cr --vo cms -l LFN -d moboro.uniandes.edu.co TURL

where LFN is the logical file name that YOU will assign to your file. TURL is the URL to your file using a Transport protocol. In order to get this, you will need to get this information using the command lcg-gt:

• Example 1: the goal here is to copy files stored at the Aachen Tier-2 to our Tier-3:

lcg-gt \
srm://grid-srm.physik.rwth-aachen.de:8443/srm/managerv2\?SFN=/pnfs/physik.rwth-aachen.de/cms/store/mc/Summer08/SUSY_LM0-sftsht/GEN-SIM-RECO/IDEAL_V11_v1/0009/FE53DD4E-2FF0-DD11-B7FB-001A645D07C8.root gsiftp
gsiftp://grid-se102.physik.rwth-aachen.de:2811//pnfs/physik.rwth-aachen.de/cms/store/mc/Summer08/SUSY_LM0-sftsht/GEN-SIM-RECO/IDEAL_V11_v1/0009/FE53DD4E-2FF0-DD11-B7FB-001A645D07C8.root
-2137491635

gsiftp is the transport protocol to be used. lcg-gt read a SURL (that points to the path to this file on the Grid) and returns something that starts with gsiftp.

• Now we got what we need:

[aosorio@yali ~]$ lcg-cr --vo cms -l  lfn:/grid/cms/test_lm0_tier3 -d moboro.uniandes.edu.co \
> gsiftp://grid-se102.physik.rwth-aachen.de:2811//pnfs/physik.rwth-aachen.de/cms/store/mc/Summer08/SUSY_LM0-sftsht/GEN-SIM-RECO/IDEAL_V11_v1/0009/FE53DD4E-2FF0-DD11-B7FB-001A645D07C8.root
[BDII] exp-bdii.cern.ch:2170: Warning, no GlueVOInfo information found about SE 'moboro.uniandes.edu.co' (with no tag)

"moboro.uniandes.edu.co" is our Storage Element (ignore the warning for the moment). That's all! depending on the size of the file, the system will hang in there until the file has been copied and registered. When it finishes, you can check if the file has been registered:

[aosorio@yali src]$ lcg-lr --vo cms lfn:/grid/cms/SUSY_LM0_sftsht_RECO_V11_1
srm://moboro.uniandes.edu.co/dpm/uniandes.edu.co/home/cms/generated/2009-05-16/file91fe2626-22cc-4bd7-913a-4b3a31942f6b

• You could also check by using CMSSW edmFileUtil:

[aosorio@yali src]$ edmFileUtil  -d /grid/cms/SUSY_LM0_sftsht_RECO_V11_1
rfio:/dpm/uniandes.edu.co/home/cms/grid/cms/SUSY_LM0_sftsht_RECO_V11_1

• Finally, I created the following script (it is sort of case specific) to copy a full data set: replicate.py.txt: replicate files using LCG cmds

CMS Sites Information

A very useful command is lcg-infosites. This command allows to see information on the CE,SE associated to CMS virtual organisation.

• examples of usage:

<lxplus238> lcg-infosites --vo cms ce se

Using lcg-infosites --vo cms closeSE one can have a list of CE and their nearest SE. I attached in here the current Sept 06 2009 result of such command.