Procedure to run on the FNAL Batch System.

The main idea is to submit a script using condor tools. that runs a python file with few arguments.

Scripts Needed

The tar file attached with this wikipage has all the scripts needed: fnalBatch.tgz. Once this file is downloded, you can get untar it by issuing the command:

tar -xzvf fnalBatch.tgz

From among the scripts, there is a script called submit_mc.sh that runs taking arguments from a text file called private_mc_files.txt. The aim from this script and text file, is to run on different datasets while just adding lines in the text file, for example:

1 RELVALTTBAR /relval/CMSSW_4_2_5/RelValTTbar/GEN-SIM-RECO/START42_V12-v1/0113

The number 1 in the first column means you want to run on this dataset (if you put 0, the line will be ignored). The second column is only a tag to be used later to differentiate different scripts/root files. The third column points to the dataset.

When you run the scrript submit_mc.sh, another script will be created and it called: sub_RELVALTTBAR, its content is as follows:

universe = vanilla
Executable = run_all.sh 
Requirements = Memory >= 199 &&OpSys == "LINUX"&& (Arch != "DUMMY" )&& Disk > 1000000
Should_Transfer_Files = YES
WhenToTransferOutput = ON_EXIT
Output = sleep_$(Cluster)_$(Process).stdout
Error = sleep_$(Cluster)_$(Process).stderr
Log = sleep_$(Cluster)_$(Process).log
notify_user = chaouki7@FNAL.GOV
Arguments = RELVALTTBAR /store//relval/CMSSW_4_2_5/RelValTTbar/GEN-SIM-RECO/START42_V12-v1/0113
Queue 1

Then submit_mc.sh will run it using the command:

condor_submit sub_RELVALTTBAR

And that is it... Once the job is finished an e-mail is sent to the e-mail you specify through the variable: notify_user.

But in the background, there are few things one needs to know in order to customize the setup. The script sub_RELVALTTBAR contains another script called run_all.sh, which includes the name of the python file to be run under CMSSW. The content of run_all.sh is:

#! /usr/bin/env sh
#

source /uscmst1/prod/sw/cms/bashrc prod 
export SCRAM_ARCH=slc5_amd64_gcc434
cd /uscms/home/chaouki7/work/CMSSW_4_2_5/src
eval `scramv1 runtime -sh`

export MYDIR=/uscms_data/d2/chaouki7/CrabSubmit/testFnal/
export OUTPUT=/uscms_data/d2/chaouki7/CrabSubmit/testFnal/
mkdir -p ${OUTPUT}
cd $MYDIR

export FR=$1
export MSOURCE=$2
export MaxEvt=-1
export MLIST="listToProcess${FR}.list"
export MYFILES=filesToRun${FR}.txt

rm -rf ${MYFILES} ${MLIST}
ls -l /pnfs/cms/WAX/11/$MSOURCE | grep root | awk '{print $9}' >>  $MYFILES

MYFILE=0
for MLINE in `cat $MYFILES`; do
    MYFILE=$((${MYFILE}+1))
    export MFILE="${MSOURCE}/${MLINE}"
    echo $MFILE >> $MLIST
done

export MYNTUP=privateMC_${FR}.root
echo "---> running cmsRun whelicity_sub_cfg.py print files_load=$MLIST output=${MYNTUP} maxEvents=${MaxEvt}"

cmsRun whelicity_sub_cfg.py print files_load=$MLIST output=${MYNTUP} maxEvents=${MaxEvt}

Therefore the command that is run in the background is:

./run_all.sh RELVALTTBAR /store//relval/CMSSW_4_2_5/RelValTTbar/GEN-SIM-RECO/START42_V12-v1/0113

You can/have to change the following:

• Directory where CMSSW is installed /uscms/home/chaouki7/work/CMSSW_4_2_5/src
• Directory where you have copied the files and also where you want the ouput to be saved ( /uscms_data/d2/chaouki7/CrabSubmit/testFnal/)
• Your python file whelicity_sub_cfg.py

The python file needs to have the options enabled for example:

import FWCore.ParameterSet.Config as cms
import sys, os
import FWCore.ParameterSet.VarParsing as VarParsing
##################################################################################################
process = cms.Process("ANALYSISNTUPLE")

options = VarParsing.VarParsing ('standard')

# setup any defaults you want
options.output = 'default.root'
options.maxEvents = -1 # -1 means all events

# get and parse the command line arguments
options.parseArguments()

........

process.source = cms.Source("PoolSource",
   fileNames = cms.untracked.vstring(options.files)
)

.....


process.aodDump = cms.EDAnalyzer('WHelicityMeasurement',
        rootFileName = cms.untracked.string(options.output),
.....

Condor Commands

These are the commands that are useful to run CMSSW jobs using the FNAL batch system.

• To check the status of the jobs use: condor_q -submitter $USER

-- Submitter: chaouki7@fnal.gov : <131.225.191.68:53231> : cmslpc02.fnal.gov
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
265337.0   chaouki7        3/16 11:44   0+06:56:16 R  0   976.6 CMSSW.sh 431 1    
265337.1   chaouki7        3/16 11:44   0+06:55:39 R  0   976.6 CMSSW.sh 432 1    
265337.2   chaouki7        3/16 11:44   0+06:53:12 R  0   976.6 CMSSW.sh 433 1    
265337.3   chaouki7        3/16 11:44   0+06:51:28 R  0   976.6 CMSSW.sh 434 1    
265337.4   chaouki7        3/16 11:44   0+06:49:16 R  0   976.6 CMSSW.sh 435 1    

• To delete a job use: condor_rm 265337.1 (for a particular job) or condor_rm 265337 (for a particular block of jobs)

-- ChaoukiB - 16-Mar-2012