Data Access

DQ2

Documentation

• DQ2Tutorial
• DQ2ClientsHowTo

• Default disks: CERN-PROD_USERDISK, CERN-PROD_SCRATCHDISK; CERN-PROD_USERDISK is closed to users
  
• Containers have / at end of their name
• Munich dq2 help, short summary of commands
• Nikhef dq2 help
• TraudlHansl-Kozanecki this is my user DN
  user.TraudlHansl-Kozanecki.name.datatype.version     dataset name for my physics data

Commands

DQ2 search tool

Command	Comment
dq2-ls  DATASET or CONTAINER	wildcard character "*"
dq2-ls -f  DATASET or CONTAINER	list files (slower than AMI)
dq2-ls -r  DATASET	list replica sites
dq2-ls -L -f  FILE1, FILE2, ...
dq2-ls -L -P  DATASET	create PoolFileCatalog.xml, is cumulative
dq2-ls -L -fp  DATASET	list physical file names at site, if files are not present the logical names are listed
dq2-ls -f  DATASETNAME | tail -4	to know the size of the dataset
dq2-list-datasets-container CONTAINERNAME	note "/" at end of name of container
dq2-list-dataset-replicas-container CONTAINERNAME
dq2-get DATASET	get data to local disk and pwd
dq2-get -f <file1, file2, ...> DATASET or CONTAINER	get data
dq2-get -F DATASET or FILELIST	get data using a file with a list, one dataset or file per line
dq2-get -s SITE	get data forcing the site
dq2-put -L IN2P3-LPSC_SCRATCHDISK  -s my_input_data   user.TraudlHansl-Kozanecki.my.tutorial.dataset.1
dq2-put -D user.MartinSchmitz.my.tutorial.dataset.1  user.TraudlHansl-Kozanecki.my.tutorial.dataset.1
dq2-sources	is same as dq2-destinations
dq2-erase	use with great caution

echo $DQ2_LOCAL_SITE_ID	local disk setting
export DQ2_LOCAL_SITE_ID = ROAMING	may be needed to be able to dq2-get files
fs lq	on AFS: how much space is available?
python /afs/cern.ch/atlas/offline/external/GRID/ddm/Operations/utils/dn.py

• DQ2 Accounting location of groupdisks

Castor

Documentation

• Atlas/CastorPools (new documentation)
  Each user can request permanent storage at Castor CERN.
• CastorDefaultPoolRestrictions (new Castor setup, only the diagram is now of interest)
• Castor data access (old page on COMPUTINGWEB).

Commands

~/scripts/setcastor.sh sets up $CASTORDIR and $CASTORFILE for a specific file, and Castordest = $CASTORDIR/$CASTORFILE.
ns commands should be used rather than rf commands. rfio commands are strongly deprecated.

Command	Comment
xrdcp root://castoratlas/$Castordest/   localFile	copy file from any file the user has read access to
	copy from pool to pool, if you have write access
nsls [ -lT ]	list directory or files (preferable at Castor)
nsmkdir [ -p ]	make directory
nsrm [ -r ]	removes file for ever from Castor
nsgetacl castordir	get the list of accounts having all permissions (rwx)
rfdir	list directory or files
rfmkdir	make directory
rfrm	remove directory or file

Check access to Castor file

Does running this in your python shell work ? In Python try

   import ROOT
   fname = 'root://castoratlas//castor/cern.ch/...'       do not use  'rfio://castor/...'    which is deprecated
   f = ROOT.TFile.Open(fname)
   assert isinstance(f, ROOT.TFile)
   assert f.IsOpen()

For access to TrkValidation ntuples you need to set export STAGE_SVCCLASS = atlcal before accessing the files, where the files are e.g. on /castor/cern.ch/grid/atlas/caf/atlcal/perm/id/Collisions2009/physics_MinBias/TrkValidation

Using xrdcp for several files

xrdcp works only for one file at a time. To copy several files write a bash script (example: ~/scripts/xrdcpExample.sh, the files no longer exist).

Disk space

For the CAF see AtlasCAF

Name	Comment
atlasscratchdisk	DDM-registered data
atlascernuserdisk	not DDM-registered data, private, "non-published" data
atlasgroupdisk	DDM-registered data, managed by DDM
atlascerngroupdisk	not DDM-registered data, local group data
ATLCAL	CAF
ATLDATA	CAF

Open file on atlasscratchdisk within ROOT

 TFile *file = TFile::Open("root://castoratlas//castor/cern.ch/grid/atlas/atlasscratchdisk/source_filename"); 

Open file on atlascernuserdisk

 TFile *file = TFile::Open("root://castoratlas//castor/cern.ch/user/h/hansl/source_filename"); 

---

-- TraudlKozanecki - 30-Oct-2009