FNAL CRAB Tutorial (JTerm August 2009) These instructions were intended only in preparation for a JTerm. They are now out of date, so use at your own risk. Please follow the official CMS instructions as indicated in these pages.

This tutorial will concentrate on Grid submission of CMSSW jobs. We will use CRAB in server mode to submit jobs to the grid and give an introduction to the most common use cases for CRAB. This tutorial will give those who have never run CRAB before a quick introduction to CRAB. After working through this, you may choose to attend the Advanced Tutorials at JTerm, but for many the best choice will still be the beginner tutorials.

An excellent tutorial was given at the June 2009 CMS week and is in the Workbook. That page is at: https://twiki.cern.ch/twiki/bin/view/CMS/WorkBookRunningGrid and this page is just a few notes on what needs to be changed to run at FNAL instead of CERN.

Login and prepare CMSSW interactive analysis

Note: whenever you see [c]sh you should use "csh" if your shell is csh or tcsh and "sh" if your shell is ash or bash. These instructions are for a simple, prepared analysis. If you have your own analysis, you are welcome to try that, but you may run into problems if the data you would like to run on is not widely distributed.

We'll begin by logging into the LPC cluster:

slogin cmslpc.fnal.gov

Next, make sure that the CMSSW environment is set up:

source /uscmst1/prod/sw/cms/[c]shrc prod

Setup local environment and prepare user analysis code

Follow all the steps in this section except sourcing of the cms_ui_env script. This is done for you on cmslpc.

CRAB setup

To use the latest production release of CRAB at FNAL, set it up like this rather than the instructions for CERN:

source /uscmst1/prod/grid/CRAB/crab.[c]sh

If you need an earlier version of CRAB or a pre-release, look in /uscmst1/prod/grid/ to see what is available, but please use the default version if possible.

Main Tutorial

Follow the rest of the tutorial until you get to the parts on publishing data, then come back here for further instructions

Publishing your data

All of the instructions in the CERN tutorial apply and will work, however, you will have to have space at a Tier2 in order to publish. As a U.S. person, you are assigned to a particular US Tier2 (or FNAL) based on your institution. See the web page http://www.uscms.org/uscms_at_work/software_computing/tier2/store_user.shtml to know where your data should be written and who to contact to have a directory made in /store/user for you.

Changing crab.cfg

We have to change crab.cfg to enable writing data to dCache. Replace the relevant section of crab.cfg with

return_data            = 0
copy_data              = 1
storage_element        = cmssrm.fnal.gov
storage_path           = /srm/managerv2?SFN=/resilient/<username>/tutorial
user_remote_dir    = run1

This instructs CRAB to copy the output files listed in

output_file = 

to dCache instead of making them available when you do crab -getoutput. This allows you to write much bigger output files (2GB vs. 50 MB) and is recommended for CMSSW ROOT files. The rest of the output (stderr, stdout, etc.) is still retrieved with

crab -getoutput

When you submit jobs you should see your output (histogram) files begin to arrive in dCache

ls -l /pnfs/cms/WAX/resilient/<username>/tutorial/run1/ 

but keep in mind that /pnfs/ is not a real file system. So you can use commands like ls, but you cannot use cp or hadd, etc. For more information, take a look at the PNFS instructions at FNAL: http://www.uscms.org/SoftwareComputing/UserComputing/MassStorage.html

Conclusion

That's it. The rest of the CERN tutorial should apply equally to running CRAB at FNAL.

-- EricVaandering - 23 Jul 2009