Notes on running CMS ML Studies
The following page contains some notes/hot tips on doing CMS ML studies.
Logging onto the auton cluster
You first have to ssh through a gateway, either
bash.autonlab.org
(or
lop1.autonlab.org
, depending how Predrag set you up)
e.g:
ssh USERNAME@bashNOSPAMPLEASE.autonlab.org
(You can add your public key to allow passwordless access)
From the gateway you can then access the various gpu servers through
ssh
.
e.g:
ssh gpu10
.
(This can also be setup to allow passwordless access)
Configuring local environment.
The instructions
here explain how the
pytorch
environment is configured for the first istall.
In subsequent sessions you can activate the environment with:
conda activate pytorch
Git on the auton cluster
Note that git only seems to work on the gateway on the worker servers (e.g:
gpu10
).
However
emacs
is not installed on the
Local data exploration
Setup a python virtual environment with python3 and mathplotlib numpy and h5py
Activate via something like:
source python-virtual-environments/env/bin/activate
You can then use some of the (very much) work in progress scripts (barely) documented here:
https://github.com/johnalison/QCD_Glu_Quark/tree/BJetML/scripts
to make some basic plots.
Running a remote jupyter session
To launch a remote jupyter session in the auton cluster from your local machine, you need to chain 2 tunnels: first into
bash.autonlab.org
(or
lop1.autonlab.org
, depending how Predrag set you up) then again into your target machine, e.g.
gpu10
:
local$ ssh -i ~/.ssh/auton -L 8010:localhost:8010 mbandrews@bashNOSPAMPLEASE.autonlab.org -t ssh -i /zfsauton/home/mbandrews/.ssh/auton -L 8010:localhost:8010 gpu10
where the first
~/.ssh/auton
is the private key in my local machine for accessing
bash.autonlab.org
, and the second
/zfsauton/home/mbandrews/.ssh/auton
is the private key in my auton home folder for accessing the actual machines from the
bash.autonlab.org
gateway. The port number is arbitrary but should not be in use (to check:
ps aux | grep ssh
). Then activate whatever environment you'll need to do your jupyter stuff:
gpu10$ conda activate my_env
Launch jupyter through whatever port you specified in your tunnel:
(my_env) gpu10$ jupyter notebook --no-browser --port=8010
In addition to notebooks, the jupyter browser also doubles as a neat file/image browser allowing you to remotely look at whatever validation plots your training scripts might be producing in the background. They don't necessarily need to be produced from your notebook --- the browser gives you access to the complete local file system starting from the directory you launched jupyter from.
Also, if you want to install
ROOT in your conda environment:
conda install -c conda-forge root
--
JohnAlison - 2019-06-12