RESPONSIBLE
Main.unknown

1.DESY computer resources

DESY hosts

Hosts working with Qsub:

nafhh-atlas01.desy.de ~ nafhh-atlas06.desy.de

Hosts working with HTCondor:

naf-atlas11.desy.de ~ naf-atlas12.desy.de.

submit HTCondor jobs:

In the Dec. 2017, the DESY computer resource is updated to HTcondor https://confluence.desy.de/pages/viewpage.action?pageId=67639562

The following commands is the cheat list from qsub to HTcondor

QSub (SGE)	HTCondor
qsub 123.job	echo "executable = 123.job" > 123.sub condor_submit 123.sub (more details below)
qstat	condor_q
qdel	condor_rm

condor_rm [username]: kill all your jobs

While submitting jobs, it has to be changed from

qsub -q short.q,default.q  -l os=sld6 -l site=hh -l h_vmem=8G -o ${base_dir}/out/${jobname}_"$ijob".out  -e  ${base_dir}/err/{jobname}_"$ijob".err  ${jobfilename}

To

rm -fr  ${submitfilename}
echo "executable     = ${jobfilename}" > ${submitfilename}
echo "should_transfer_files   = Yes" >> ${submitfilename}
echo "when_to_transfer_output = ON_EXIT" >> ${submitfilename}
echo "input          = ${base_dir}/input/${jobname}_"$ijob".txt"  >> ${submitfilename}
echo "output         = ${base_dir}/out/${jobname}_"$ijob".out2" >> ${submitfilename}
echo "error          = ${base_dir}/err/${jobname}_"$ijob".err2" >> ${submitfilename}
echo "log            = ${base_dir}/log/${jobname}_"$ijob".log2"  >> ${submitfilename}
echo "universe       = vanilla"  >> ${submitfilename}
echo "RequestMemory   = 4G" >> ${submitfilename}
echo "queue" >> ${submitfilename}
condor_submit ${submitfilename}

The example scripts are in this path:
/afs/desy.de/user/h/hans/public/condor_example

DESY HTCondor queue :

On 28th Nov. 2017 there are ~1750 cpus, HTCondor can distribute cpus for jobs itself

HTCondor is going to have more and more cpus, please always check http://bird.desy.de/stats/day.html

On 28th Nov. 2017 there are still ~5000 cpus for Qsub (SGE), please always check http://bird.desy.de/status/day.html

DESY disks

Login https://amfora.desy.de to check AFS and NAF quota

AFS is usually 16G, NAF is 1T (default)

DESY Tire2

The quota on DESY-HH_LOCALGROUPDISK will be 20T (default) after applying for atlas/de VO

The transferred files on this disk can be found under /pnfs path

2. IHEP computer resources

http://afsapply.ihep.ac.cn:86/quick/

IHEP hosts:

SLC6 hosts:

lxslc6.ihep.ac.cn
atlasui01.ihep.ac.cn
atlasui04.ihep.ac.cn ~ atlasui06.ihep.ac.cn (IHEP inner hosts)

SLC5 hosts:

atlasui03.ihep.ac.cn
lxslc5.ihep.ac.cn

submit HTCondor jobs:

before doing anything, use the command:

export PATH=/afs/ihep.ac.cn/soft/common/sysgroup/hep_job/bin5:$PATH  # SLC5
export PATH=/afs/ihep.ac.cn/soft/common/sysgroup/hep_job/bin:$PATH  # SLC6

Then you use the following cheat table

Qsub	HTCondor
qsub	hep_sub
qstat	hep_q
qdel	hep_rm

hep_rm -a: kill all the jobs

hep_q -u yourIhepUserName: check your jobs status

normally it is very similar as using Qsub, we just need use the command:

hep_sub job.sh -o out -e err -mem 4800 -g atlas

other optional arguments:

-os OPERATINGSYSTEM, --OperatingSystem OPERATINGSYSTEM
                        # set the system version of resource you want.
-prio PRIORITY, --priority PRIORITY
                        # set the inner job priority of your own jobs.
-np NUMBERPROCESS, --numberprocess NUMBERPROCESS
                        # set the total cores required by your job.

The example scripts are in this path (do not submit in /workfs or /afs):
/publicfs/atlas/atlasnew/higgs/hgg/ShuoHan/ATLAS/job1_data2016

IHEP HTCondor queue :

one universal queue sharing 4300 cpus, the working time is infinite, HTCondor can distribute cpus for jobs itself

IHEP disks :

disk name	disk path	size	backup	Can submit HTCondor jobs
afs	/afs/ihep.ac.cn/users/	500MB	unknown	no
workfs	/workfs/atlas/	10GB	yes	no
scratchfs	/scratchfs/atlas/	500GB	delete every 2 weeks	yes
publicfs atlasnew	/publicfs/atlas/atlasnew/higgs/hgg/	320TB shared	no	yes
publicfs codesbackup	/publicfs/atlas/codesbackup/	320TB shared	yes	yes

IHEP Tier2

The quota on BEIJING-LCG2_LOCALGROUPDISK will be 8T after applying for atlas/cn VO

But those files (on disk /dpm) should be downloaded to Tier3 anyway by command "dpns-ls -l" and "rfcp" , they cannot be directly read

3. ATLAS environment

Setup ATLAS environment:

Setup ATLAS Environment:

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'
setupATLAS

Then, setup Root:

lsetup root

Certificate, ATLAS VO, Rucio, AMI

Certificate: "New Grid User Certificate" in https://ca.cern.ch/ca/
Setup certificate on DESY/IHEP/Lxplus:

openssl pkcs12 -in usercert.p12 -nokeys -clcerts -out ~/.globus/usercert.pem
openssl pkcs12 -in usercert.p12 -nocerts  -out ~/.globus/userkey.pem
chmod 400 ~/.globus/userkey.pem
chmod 444 ~/.globus/usercert.pem
voms-proxy-init voms atlas

ATLAS VO page (request here): https://voms2.cern.ch:8443/voms/atlas/user/home.action

Rucio transfer request page: https://rucio-ui.cern.ch/r2d2/request
Rucio commands:

voms-proxy-init -voms atlas
lsetup rucio
rucio list-dids DATASETNAME
rucio list-file-replicas DATASETNAME
rucio get DATASETNAME
rucio list-rules --file FILETYPE(e.g. data16_13TeV):FILEPATH

AMI page: https://ami.in2p3.fr/
pyAMI commands: https://ami.in2p3.fr/pyAMI/

voms-proxy-init -voms atlas
lsetup pyami
ami show dataset info  DATASETNAME

ATLAS Configs/Tools

Truth Type and Truth origin : https://gitlab.cern.ch/atlas/athena/blob/master/PhysicsAnalysis/MCTruthClassifier/MCTruthClassifier/MCTruthClassifierDefs.h
Electron / Photon ID config: http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/GroupData/ElectronPhotonSelectorTools/offline/
Binary location of e/gamma ID: https://gitlab.cern.ch/atlas/athena/blob/master/PhysicsAnalysis/ElectronPhotonID/ElectronPhotonSelectorTools/ElectronPhotonSelectorTools/egammaPIDdefs.h
e/gamma Author definitions: https://gitlab.cern.ch/atlas/athena/blob/master/Event/xAOD/xAODEgamma/xAODEgamma/EgammaDefs.h
e/gamma Enum (ConversionType, EgammaType, ShowerShapeType): https://gitlab.cern.ch/atlas/athena/blob/master/Event/xAOD/xAODEgamma/xAODEgamma/EgammaEnums.h

Lowest un-prescaled trigger: https://twiki.cern.ch/twiki/bin/view/Atlas/LowestUnprescaled
Cross-Section and Branching-Ratio page: https://twiki.cern.ch/twiki/bin/view/LHCPhysics/LHCHXSWG
Current Luminosity: https://twiki.cern.ch/twiki/bin/view/AtlasPublic/LuminosityPublicResultsRun2
GRL page: http://atlasdqm.web.cern.ch/atlasdqm/grlgen/All_Good/
Lumi-calculator: https://atlas-lumicalc.cern.ch/, select "Create output plots and ntuples (--plots)" to create PRW input
Athena nightly builds: http://atlas-nightlies-browser.cern.ch/~platinum/nightlies/globalpage
HGamma twiki: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/HggDerivationAnalysisFramework
MxAOD twiki: https://twiki.cern.ch/twiki/bin/view/AtlasProtected/MxAODs

ATLAS Tutorials

PRW config page: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/ExtendedPileupReweighting
Creating Event-Loop package: https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/SoftwareTutorialxAODAnalysisInROOT
Analysis Base tutorial: https://atlassoftwaredocs.web.cern.ch/ABtutorial/
ATLAS tutorial meetings: https://indico.cern.ch/category/397/
ATLAS software tutorial twiki: https://twiki.cern.ch/twiki/bin/view/AtlasComputing/SoftwareTutorialSoftwareBasics
PubCome Latex twiki: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/PubComLaTeX

4. Softwares

GitLab

Get a ssh Key here : https://gitlab.cern.ch/help/ssh/README
Add the ssh Key here : https://gitlab.cern.ch/profile/keys
Start a new project here : https://gitlab.cern.ch/dashboard/projects
Basic commands here : https://gitlab.cern.ch/help/gitlab-basics/README.md
Some basic commands:

git clone SSH_PATH
git checkout BRANCH_OR_TAG
git pull
git add --all
git commit -am "something"
git push origin BRANCH

Cmake

CmakeLists.txt of Athena based package : https://gitlab.cern.ch/atlas-hgam-sw/HGamCore/blob/master/HGamTools/CMakeLists.txt
CmakeLists.txt of local code : https://gitlab.cern.ch/shhan/Purity_2D/blob/master/CMakeLists.txt

Use Virtual machine (or Docker) to run Athena on your computer

Docker: https://docs.docker.com/docker-for-mac/
Docker with Athena: https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/AthenaMacDockerSetup

docker pull lukasheinrich/atlas_base_fromhepsw
docker run --privileged -it -v $HOME:$HOME -e DISPLAY=${ip}:0 lukasheinrich/atlas_base_fromhepsw /bin/bash

CERNVM: https://atlas-vp1.web.cern.ch/atlas-vp1/blog/tutorial/running-vp1-on-any-system-in-a-virtual-machine/
Follow this page until “SetupATLAS”, so you have a linux virtual machine in ATLAS environment
Administrator (in case you want to use sodo): account : atlas, password: boson

ATLAS Event Display

Latest event: http://atlas-live.cern.ch/
Public plots: https://twiki.cern.ch/twiki/bin/view/AtlasPublic/EventDisplayPublicResults
Event Pickup (JML and ESD): https://twiki.cern.ch/twiki/bin/view/AtlasComputing/Atlantis
Atlantis: download latest version here (http://atlantis.web.cern.ch/atlantis/), run in a system where Java is available
VP1:

asetup 20.7.9.2,AtlasProduction, here
vp1 DATASET.ESD.root

EUTelescrope

Home Page: http://eutelescope.web.cern.ch
Setup on DESY NAF disk:

setupATLAS
lsetup "root 6.10.04-x86_64-slc6-gcc62-opt"
export ILCSOFT=/nfs/dust/atlas/user/hans/EUTelescrope_desy
cd $ILCSOFT
git clone -b dev-desynaf https://github.com/eutelescope/ilcinstall $ILCSOFT/ilcinstall
cd $ILCSOFT/ilcinstall
./ilcsoft-install -i examples/eutelescope/release-standalone.cfg

5. ADCoS Shift links

• ADCoS links summary: https://adcos.web.cern.ch/adcos/For_ADCoS_Shifters/

• ADCVCR control room: https://twiki.cern.ch/twiki/bin/view/AtlasComputing/ADCVirtualControlRoom
  • Jabber services: https://www.jabberes.org/servers/
  • Room: adcvcr
  • Server: conference.chat.uio.no
  • Password: point1

6. Internal Code/Packages

• Template fit and ABCD method: https://gitlab.cern.ch/shhan/Purity_2D

• E->gamma fake rate measurement: https://gitlab.cern.ch/shhan/TagAndProbeFrame

7. Weekly Discussion

• JET/MissingET Reconstruction: see twiki https://twiki.cern.ch/twiki/bin/view/AtlasProtected/IntroductionToHadronicCalibration

• HTcondor tutorial: HTCondor_Manual-ATLAS_English.pdf