UMass Tier 3 Cluster Info

The UMass cluster consists of a head node named titan.physics.umass.edu along with a series of nodes named titan31, titan32, titan33,... . One must log onto the head node first to gain access or submit jobs to any of those nodes. Batch job management is done with Torque (OpenPBS). See the OpenPBS page for information about using the batch system on titan. Further information about the cluster and the installed Torque/OpenPBS batch system, see the information for the previous cluster still relevant for the batch system at halcluster.pdf.

UMass Tier 3 Center:

The status of the cluster can be monitored with the Ganglia web interface.

The cluster includes about 100TB of storage, most of which in an xrootd partition named /titan_rw/atlas/common. All current ATLAS software releases are available via CVMFS, see the basic setup instructions below.

AFS is available on the head node only, use the command kinit followed by aklog to get a token to CERN (default). This will work only if you have a CERN computer account though.

Account Setup (to be done only once)

Edit the .zshrc file in your home directory to include the following lines:

# cvmfs Tier 3 env setup

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'
export ALRB_localConfigDir=$HOME/localConfig

In your home directory, create a file named .asetup and insert the following contents: .asetup.

Create a general test area directory and a specific one for release 16.6.4:

cd $HOME
mkdir -p testarea/16.6.4

This whole wiki is set up to run with release 16.6.4 but the release can easily be changed in the setup command, see below. At this point, it is easiest to log off and log on again for the above setup to be run (the file .zshrc is automatically executed whenever you start a new zsh session).

Account Setup (every time)

If you only need to run root then the only setting up required in each terminal window is

setupATLAS
localSetupROOT

To have access to the full ATLAS software, you need to set up the release every time you start work in a new terminal window. WIth the above setup file, it is necessary to first change the current directory to your test area before setting up the release:

cd ~/testarea/16.6.4
asetup 16.6.4

To work on the titan30...titan39 machines, one needs to specify the SLC5 architecture (this will no longer be needed when ATLAS software switches to SLC6):

asetup --cmtconfig=i686-slc5-gcc43-opt 16.6.4

Check that the CMTPATH environment variable is correctly set by issuing the command echo $CMTPATH, the output should look something like:

/home/willocq/mytest/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasOffline/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasSimulation/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasAnalysis/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasTrigger/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasReconstruction/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasEvent/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasConditions/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/dqm-common/dqm-common-00-14-02:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/AtlasCore/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/tdaq-common/tdaq-common-01-16-02:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/DetCommon/16.6.4:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/GAUDI/v21r9p3a:/cvmfs/atlas.cern.ch/repo/sw/software/i686-slc5-gcc43-opt/16.6.4/LCGCMT/LCGCMT_59a:/data/share/atlas/ATLASLocalRootBase/Athena/AtlasSetup/V00-02-68/AtlasSetup/share

Running Hello World

Checkout the UserAnalysis software package version associated with this release

cd ~/testarea/16.6.4
pkgco.py PhysicsAnalysis/AnalysisCommon/UserAnalysis

If you are unable to do the above (e.g. no access to CERN or do not have a CERN lxplus account), you can copy the package from a local account

cd ~/testarea/16.6.4
tar xvzf ~willocq/tutorials/16.6.4/UserAnalysis.tgz

Build the UserAnalysis package

cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
gmake

Run the Hello World example

cd ../run
get_files HelloWorldOptions.py
athena.py HelloWorldOptions.py

Output to the screen should contain lines like:

ApplicationMgr       INFO Application Manager Started successfully
AthenaEventLoopMgr   INFO   ===>>>  start of run 0    <<<===
HelloWorld           INFO beginRun()
AthenaEventLoopMgr   INFO   ===>>>  start processing event #1, run #0 0 events processed so far  <<<===
HelloWorld           INFO execute()
HelloWorld           INFO An INFO message
HelloWorld        WARNING A WARNING message
HelloWorld          ERROR An ERROR message
HelloWorld          FATAL A FATAL error message
HelloWorld           INFO Let the tool MyPublicHelloTool say something:
ToolSvc.PublicH...   INFO my message to the world: A Public Message!
HelloWorld           INFO Let the tool MyPrivateHelloTool say something:
HelloWorld.Hell...   INFO my message to the world: A Private Message!
AthenaEventLoopMgr   INFO   ===>>>  done processing event #1, run #0 1 events processed so far  <<<===

SVN

UMass has an SVN repository under

$SVNINST/Institutes/UMass

and everyone should have read/write permissions. Users must request packages to be set up by emailing atlas-svn-managers@cernNOSPAMPLEASE.ch.

General info about SVN can be found in the Software Development Workbook. You can browse the repository via https://svnweb.cern.ch/trac/atlasinst/browser/Institutes/UMass. You will need to login.

To check out the head of the EWAnalysis package, first get an AFS token to CERN via

kinit

svn co $SVNINST/Institutes/UMass/EWAnalysis/trunk EWAnalysis

If the environment variable SVNINST isn't defined you can do so with

export SVNINST=svn+ssh://svn.cern.ch/reps/atlasinst 

Grid and DQ2

To use the local installation of grid tools and DQ2 tools, you must first obtain a valid grid certificate and copy your certificate usercert.pem and key userkey.pem to your ~/.globus directory on titan. Then set up the tools and activate your certificate with

localSetupDQ2Client 
voms-proxy-init -voms atlas

Useful Scripts

Below are a number of very handy scripts that make dealing with ATLAS data and software a bit easier...

• acmd.py: general script interface with sub-commands to perform file merging, skimming, content dumping, etc.

Usage:  acmd.py --help

Usage: acmd.py merge-files file1.ESD.pool.root file2.ESD.pool.root -o mergedfile.ESD.pool.root

• checkFile.py: list the contents of a given file

Usage:  checkFile.py myfile.pool.root

• cmtClients.py: list all clients of a given package

Usage:  cmtClients.py MuonCSC_CnvTools

• dumpRunNumber.py: for a given input file, (guess what?) dump the run number...

Usage:  dumpRunNumber.py myfile.pool.root

• dumpVersionTags.py: for a given input file, get a list of various tags like detector description version, cabling maps, etc.

Usage:  dumpVersionTags.py myfile.pool.root

• setupWorkArea.py: if you check out multiple packages and want to compile all of them in one go, do as follows from your test release top directory

setupWorkArea.py
cd WorkArea/cmt
cmt br make

• AtlCopyBSEvent.exe: to write a single event bytestream file extracted from 1 or more full bytestream files

Usage: AtlCopyBSEvent.exe -e [--event] <eventNumbers> [-r, --run <runnumber>] [-l, --listevents] -o, --out outputfile inputfiles....
eventNumbers is a comma-separated list of events

Further Information

For further information about ATLAS software, see the ATLAS Computing WorkBook and the ATLAS PhysicsAnalysisWorkBook.

For information about installing ATLAS software via release kits, see WorkBookInstallingAtlasSoftware.

For Andrew's guide to using xrootd storage see UMassXrootd.

For Andrew's ntuple analysis framework see UmassNtupleAnalysis wiki.

For Emily's AthenaROOTAccess and event generator instructions, see Emily's wiki.

For Max's guide to setup ElectroweakBosons framework see UmassElectroweakBosons wiki.

For protected information, see: UMassGroupProtectedTWiki

Getting Help

Relevant email lists are

• Distributed analysis help (for analysis jobs on the grid) hn-atlas-dist-analysis-help@cernNOSPAMPLEASE.ch
• Distributed data management operations (for data access issues) hn-atlas-DDMOperations@cernNOSPAMPLEASE.ch
• Offline software help hn-atlas-offlineswhelp@cernNOSPAMPLEASE.ch

-- StephaneWillocq - 04-Sep-2011 -- StephaneWillocq - 07 Nov 2008