Data Management notes

Data transfers

Debugging tools and examples

Some of these commands are old and obsolete (like lcg-cp), but may still be useful.

gfal-copy from GFAL2 utils

Correspondence between lcg-* and gfal-* commands: GFAL to GFAL2 Migration

Description	lcg-util	gfal2-util
Add an alias for a given GUID	lcg-aa	Deprecated
Bring SURLS online	lcg-bringonline	N/A, can be done
Copy files with no catalog	lcg-cp	gfal-copy
Copy file and register	lcg-cr	gfal-copy
Delete a file / directory	lcg-del	gfal-rm
Get checksum	lcg-get-checksum	gfal-checksum
Get TURL/s	lcg-getturls	gfal-xattr
List aliases for a given GUID	lcg-la	Deprecated
Get the GUID for an LFN	lcg-lc	gfal-xattr
List the replicas for an LFN	lcg-lg	gfal-xattr
Directory listing	lcg-ls	gfal-ls
Remove an alias	lcg-ra	Deprecated
Copy between SEs with registration	lcg-rep	gfal-copy
Register a file in the catalog	lcg-rf	gfal-copy
Set a file status to done	lcg-sd	Deprecated
Get space tokens	lcg-stmd	gfal-xattr
Unregister a file	lcg-uf	Deprecated
Create a directory		gfal-mkdir
Dump stdin into a file		gfal-save
Dump a file into stdout		gfal-cat

Examples of usage:

[chudoba@ui9 ~]$ gfal-copy srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps/dump_20160424 file:///tmp/dump_20160424
Copying srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps/dump_20160424   [DONE]  after 3s

The first attempt took 19 s, the second one only 3 s. I do not know why is it so slow for the first attempt.

gfal-copy to register files in LFC

Use simply lfc://lfc-server-name/path/to/files as a destination:

lcg-lr lfn:/grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz
srm://grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz

gfal-copy srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz lfc://lfc1.egee.cesnet.cz/grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz
Copying srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758Copying srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758Copying srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz   [DONE]  after 0s

lcg-lr lfn:/grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz
srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz
srm://grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz

gfal-legacy-replicas lfn:/grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz 
srm://dpm1.egee.cesnet.cz/dpm/cesnet.cz/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz
srm://grid05.lal.in2p3.fr/dpm/lal.in2p3.fr/home/auger//grid/auger/prod/GenDisProtonQGSjetII_gr298/en17.500/th0.65/075841/DAT758419.tar.gz

gfal-ls

[chudoba@ui9 ~]$ gfal-ls srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps/
dump_20151104
dump_20151111
dump_20151123
dump_20151124
dump_20151224
dump_20160124
dump_20160224
dump_20160324
dump_20160424
dump_20160524
dump_20160624
dump_20160724

gfal-xattr

[chudoba@ui9 ~]$ gfal-xattr srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps/dump_20160724
user.replicas = gsiftp://dpmpool12.farm.particle.cz/dpmpool12.farm.particle.cz:/mnt/fs6/atlas/2016-07-25/dump_20160724.422457473.0
user.status = ONLINE
srm.type = DPM

[chudoba@ui9 ~]$ gfal-xattr srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps/
user.replicas FAILED: error on the turl request : [SE][StatusOfGetRequest][SRM_INVALID_PATH] Source is a directory 
user.status = UNKNOW
srm.type = DPM

Further examples are for the VO auger, because ATLAS does not use LFC anymore.

gfal-xattr on a file which was reported as lost:

$ gfal-xattr lfn:/grid/auger/prod/epos_gr03/en20.500/th53.000/DAT002386.long
user.guid = b4a8c3b5-046a-4832-90d7-9a525dab294f
user.replicas = srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/auger//grid/auger/prod/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long
srm://gridka-dCache.fzk.de/pnfs/gridka.de/auger/disk-only///grid/auger/prod/REPLICATED20081014/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long
user.comment = 

$ gfal-xattr srm://gridka-dCache.fzk.de/pnfs/gridka.de/auger/disk-only///grid/auger/prod/REPLICATED20081014/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long
user.replicas = gsiftp://f01-075-136-e.gridka.de:2811//pnfs/gridka.de/auger/disk-only/grid/auger/prod/REPLICATED20081014/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long
user.status = ONLINE
srm.type = dCache

$ gfal-copy srm://gridka-dCache.fzk.de/pnfs/gridka.de/auger/disk-only///grid/auger/prod/REPLICATED20081014/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long file:///tmp/DAT002386.long
Copying srm://gridka-dCache.fzk.de/pnfs/gridka.de/auger/disk-only///grid/auger/prod/REPLICATED20081014/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long   [DONE]  after 2s

Obviously, the file was recovered.

SPACETOKENS

gfal-xattr does not show a spacetoken when no options are given. Although its manual page states: If no attribute is given, all attributes and corresponding values are displayed.

$ gfal-xattr srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/auger//grid/auger/prod/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long
user.replicas = gsiftp://dpmpool14.farm.particle.cz/dpmpool14.farm.particle.cz:/mnt/fs2/auger/2015-08-14/DAT002386.long.378944749.0
user.status = ONLINE
srm.type = DPM

$ gfal-xattr srm://golias100.farm.particle.cz/dpm/farm.particle.cz/home/auger//grid/auger/prod/epos_gr03.CORS.6617.03/en20.500/th53.000/DAT002386.long spacetoken
["0d8a13e9-c053-4373-91b5-6182b198d1f7"]

For dCache site:

$ gfal-xattr srm://dcache.du.cesnet.cz/data/du.cesnet.cz/auger/disk/user/chudoba/jiri-auger-2GB.t630
user.replicas = gsiftp://195.113.231.217:2811//data/du.cesnet.cz/auger/disk/user/chudoba/jiri-auger-2GB.t630
user.status = ONLINE
srm.type = dCache

$ gfal-xattr srm://dcache.du.cesnet.cz/data/du.cesnet.cz/auger/disk/user/chudoba/jiri-auger-2GB.t630 spacetoken
["1338420","1338421","610001","610002"]

How can I get a spacetoken description?

lcg-cp from LCG util

obsoleted 1st November 2014, but still used by many tools

uberftp

We can login directly to many SEs when a proxy is initiated:

[chudoba@ui9 ~]$ uberftp golias100.farm.particle.cz
220 golias100.farm.particle.cz GridFTP Server 10.4 (gcc64, 1463669416-85) [Globus Toolkit 6.0] ready.
230-User :globus-mapping: logged in.
230-Disk Pool Manager (dmlite)
230 End.
UberFTP (2.8)> pwd
/
UberFTP (2.8)> ls
drwxrwxr-x   2     root     root            0 Apr 19 15:27 dpm
drwxrwxr-x   0      282      109            0 Aug 25 11:57 dpmpool10.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep  3 03:12 dpmpool11.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 30 06:40 dpmpool12.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 25 07:33 dpmpool13.farm.particle.cz:
drwxrwxr-x   0      282      109            0 May 18 19:07 dpmpool14.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 25 09:07 dpmpool15.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 12 09:40 dpmpool16.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug  5 03:19 dpmpool17.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 21 18:05 dpmpool18.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 13 12:40 dpmpool19.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug  6 12:40 dpmpool7.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 24 01:38 dpmpool8.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 18 15:02 dpmpool9.farm.particle.cz:
drwxrwxr-x  16      282      109            0 Oct  7 12:51 mnt
UberFTP (2.8)> cd /dpm/farm.particle.cz/home/atlas/atlasscratchdisk/dumps
UberFTP (2.8)> get dump_20160424
dump_20160424: 14065332 bytes in 1.005877 Seconds (13.335 MB/s)

A failure may occur for some SEs:

[chudoba@ui9 ~]$ uberftp -ls gsiftp://se-goegrid.gwdg.de/
530 Login denied

The problem here is in a proxy. I have done only

voms-proxy-init

and I got this proxy:

[chudoba@ui9 ~]$ voms-proxy-info -all
subject   : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba/CN=proxy
issuer    : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba
identity  : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba
type      : full legacy globus proxy
strength  : 1024
path      : /tmp/x509up_u435
timeleft  : 11:59:41
key usage : Digital Signature, Key Encipherment

To be able to get output, I must initialize a proxy with voms extensions with ATLAS

[chudoba@ui9 ~]$ alias voms-proxy-init-atlas
alias voms-proxy-init-atlas='export LCG_GFAL_VO=atlas; export X509_USER_CERT=/home/chudoba/.globus/atlas2016cert.pem; export X509_USER_KEY=/home/chudoba/.globus/atlas2016key.pem; voms-proxy-init -voms atlas -valid 96:0'
[chudoba@ui9 ~]$ voms-proxy-init-atlas
Enter GRID pass phrase for this identity:
Contacting vo.racf.bnl.gov:15003 [/DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov] "atlas"...
Error contacting vo.racf.bnl.gov:15003 for VO atlas: Connection refused
Error contacting vo.racf.bnl.gov:15003 for VO atlas: Connection refused
Error contacting vo.racf.bnl.gov:15003 for VO atlas: REST and legacy VOMS endpoints failed.
Contacting voms2.cern.ch:15001 [/DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch] "atlas"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_u435.

Your proxy is valid until Fri Aug 12 10:45:57 CEST 2016

[chudoba@ui9 ~]$ voms-proxy-info -all
subject   : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba/CN=proxy
issuer    : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba
identity  : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba
type      : full legacy globus proxy
strength  : 1024
path      : /tmp/x509up_u435
timeleft  : 95:59:53
key usage : Digital Signature, Key Encipherment
=== VO atlas extension information ===
VO        : atlas
subject   : /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba
issuer    : /DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch
attribute : /atlas/Role=NULL/Capability=NULL
attribute : /atlas/alarm/Role=NULL/Capability=NULL
attribute : /atlas/cz/Role=NULL/Capability=NULL
attribute : /atlas/lcg1/Role=NULL/Capability=NULL
attribute : /atlas/team/Role=NULL/Capability=NULL
attribute : nickname = chudobaj (atlas)
timeleft  : 95:59:52
uri       : voms2.cern.ch:15001

[chudoba@ui9 ~]$ uberftp -ls gsiftp://se-goegrid.gwdg.de/
dr-x------  1 atlas001   atlas001            512 Feb  5  2009 admin
dr-x------  1 atlas001   atlas001            512 Feb  5  2009 usr
dr-x------  1 atlas001   atlas001            512 Mar  5  2010 pnfs
d--x------  1 atlas001   atlas001            512 Feb  1  2016 upload

[chudoba@ui9 ~]$ uberftp -ls gsiftp://golias100.farm.particle.cz/
drwxrwxr-x   2     root     root            0 Apr 19 15:27 dpm
drwxrwxr-x   0      282      109            0 Aug 25 11:57 dpmpool10.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep  3 03:12 dpmpool11.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 30 06:40 dpmpool12.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 25 07:33 dpmpool13.farm.particle.cz:
drwxrwxr-x   0      282      109            0 May 18 19:07 dpmpool14.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 25 09:07 dpmpool15.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 12 09:40 dpmpool16.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug  5 03:19 dpmpool17.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 21 18:05 dpmpool18.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug 13 12:40 dpmpool19.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Aug  6 12:40 dpmpool7.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 24 01:38 dpmpool8.farm.particle.cz:
drwxrwxr-x   0      282      109            0 Sep 18 15:02 dpmpool9.farm.particle.cz:
drwxrwxr-x  16      282      109            0 Oct  7 12:51 mnt

The access to the gsiftp://golias100.farm.particle.cz/ works also with a simple proxy (obtained using voms-proxy-init)

Why do we have to many filesystems directly in mnt on golias100?

[chudoba@ui9 ~]$ uberftp -ls gsiftp://golias100.farm.particle.cz/mnt/
drwxrwxr-x   1      282      109            0 Aug  5 03:19 fs1
drwxrwxr-x   1      282      109            0 Sep 21 06:14 fs10
drwxrwxr-x   1      282      109            0 Sep 22 12:59 fs11
drwxrwxr-x   1      282      109            0 Sep 27 00:52 fs12
drwxrwxr-x   1      282      109            0 Sep 26 11:16 fs13
drwxrwxr-x   1      282      109            0 Aug 12 09:40 fs14
drwxrwxr-x   1      282      109            0 Oct  7 12:51 fs15
drwxrwxr-x   1      282      109            0 Aug  8 16:21 fs2
drwxrwxr-x   1      282      109            0 Aug  6 12:40 fs3
drwxrwxr-x   1      282      109            0 Sep 18 13:16 fs4
drwxrwxr-x   1      282      109            0 Aug 31 23:57 fs5
drwxrwxr-x   1      282      109            0 Aug 25 11:57 fs6
drwxrwxr-x   1      282      109            0 Aug 26 08:47 fs7
drwxrwxr-x   1      282      109            0 Sep 24 00:57 fs8
drwxrwxr-x   1      282      109            0 Sep 26 19:01 fs9
drwxrwxr-x   1      282      109            0 May 18 19:07 ops_fs1

-- JiriChudoba - 2016-08-08