CernVM trouble-shooting

I have been using CernVM to run LHCb software on my Mac OSX 10.6.5 laptop and hope to use it on Linux desktops in the near future. It is an extremely useful and elegant way of dealing with the OS and software installation problems. In a nutshell, it runs the appropriate version of CERN scientific Linux in a virtual machine and takes care of loading necessary software on-demand, as long as the software has been made available in the CernVM servers. I tried one of the early versions a few years ago, and have recently installed the latest version from scratch, and I must say the installation has become much easier and solid. There are a currently few little things that have to be taken care of, particularly if you want to run interactive sessions, and you want grid access. This is a purely personal page, and some of the problems I encountered are likely to be due to my own incompetence rather than a problem with CernVM. And some problems or features are specific to the LHCb software distribution.

Hardware

I have a Macbook Pro running Mac OSX 10.6.5. It claims to have a 2.4GHz Intel Core 2 duo processor, and 4GB or RAM.

Virtual machine

There are a few options here but VirtualBox, formerly from Sun Microsystems, now Oracle, is very good and easy to use. The insatallation is just a matter of downloading it from here. I am using version 3.2.12 r68302. And I am keeping my fingers crossed that VirtualBox will not start going the same way as OpenOffice.org or Java seem to be going lately...

CernVM installation

My suggestion here is to follow the steps in the CernVM page to the letter. They will be the most up to date instructions and if something goes wrong you know where to ask for support. There are already quite a few obsolete wiki pages lying around on the interwebs. So, go to the CernVM page, follow User support -> Documentation -> Running in VirtualBox on Mac. Follow all those steps!

Step 1. As a CernVM image, I got the 64 bit desktop version. I suggest putting the image in a place you will remember and you won't delete by accident. I have something like ~/VM/CernVM/. After unzipping and untarring this leaves me with an image at /Users/my_local_username/VM/CernVM/cernvm-2.2.0-x86_64/cernvm-2.2.0-x86_64.vmdk.

Step 2 tells you how to configure the VirtualBox. It is worth mentioning that you can have more than one virtual machine on VirtualBox. I currently have a CernVM and an Ubuntu installation, but you could have more than one version of each, so you don't have to delete one CernVM if you want to upgrade to a newer version. I set the memory to 2048MB. I also followed the instructions to set up the shared folder, which points to my /Users/my_local_username/VM/CernVM/Shared, and is available in the VM as /mnt/shared/jpalac. I can see the folder, I can create files on ib both from the VM and from the host, but there are pboblems (see below).

Step 3 is where you would have to take experiment-specific actions. I took no action other than selecting LHCb as my group.

Specific configuration issues

CernVM fixes and improvements

It seems that (t)csh users (I am not one of them) have problems with the LHCb environment being set. There are also some available optimisations that cure part of a problem whereby accessing many catalogs (for instance upon issuing SetupProject DaVinci) is exceedingly slow. These problems are not fully fixed but the cernvmfs updates should help. Issue the commands

sudo conary update cvmfs2=cernvm.cern.ch@cvm:cernvm-2-devel
sudo conary update cernvm-config=cernvm.cern.ch@cvm:cernvm-2-devel

Then, reboot, and check the changes:

conary q cvmfs2
> cvmfs2=0.2.52.5-1-1
conary q cernvm-config
> cernvm-config=0.18.13-1-1

Setting up AFS

Following the CernVM 2.20 release notes, I do

sudo conary update group-openafs-client openafs openafs-client

sudo mkdir -p /afs

echo cern.ch > /tmp/ThisCell

sudo cp /tmp/ThisCell /usr/vice/etc/ThisCell

To log in and get AFS tokens I do

kinit 
aklog

Setup for passwordless access to SVN, CVS

Follow the instructions http://svn.web.cern.ch/svn/howto.php#accessing-ssh, doing the same in .ssh/config for isscvs.cern.ch as is done for svn.cern.ch. My .ssh/config now looks like this:

Host lxplus.cern.ch lxplus
Protocol 2
PubkeyAuthentication no
PasswordAuthentication yes

Host svn.cern.ch svn
Protocol 2
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
ForwardX11 no

Host isscvs.cern.ch isscvs
Protocol 2
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
ForwardX11 no

This works because my username is the same as my AFS username. If not, add an User your_afs_username entry to each Host. I need to have valid tokens.

Outstanding problems

No automatic LHCb group login

The LHCb script that sets up useful aliases does not run automatically. The script is /opt/lhcb/group_login.sh or /opt/lhcb/group_login.csh depending on your shell, and need to be run by hand. There is a fix for (t)csh in the updates above, whereas bash is an LHCb problem that is being addressed.

Group login script, SetupProject, getpack hang

Running . /opt/lhcb/group_login.sh, SetupProject and getpack take about 1 minute or more. I don't know if this is normal but it seems too slow to me. In fact, I would not like the "no automatic LHCb group login" problem to be solved if it is going to slow down xterm startup like this!

Cannot build software on shared folder

I have set up a build area in /mnt/shared/jpalac. I can checkout code from the repository via SVN, I can write files there, but when I build, I get the following:

#CMT---> installing file libDaVinciTools.so into /mnt/shared/jpalac/cmtuser/DaVinci_v26r4/InstallArea/x86_64-slc5-gcc43-opt/lib
eval ${install_command} ${full_source_name} ${dest_file_path}
python /opt/lhcb/lib/lhcb/GAUDI/GAUDI_v21r11/GaudiPolicy/scripts/install.py -xCVS -x*~ -x*.stamp -x*.bak -x.* -x*.pyc -x*.pyo -s --log=./install.x86_64-slc5-gcc43-opt.history /mnt/shared/jpalac/cmtuser/DaVinci_v26r4/Phys/DaVinciTools/x86_64-slc5-gcc43-opt/libDaVinciTools.so /mnt/shared/jpalac/cmtuser/DaVinci_v26r4/InstallArea/x86_64-slc5-gcc43-opt/lib/libDaVinciTools.so
Create Link to '../../../Phys/DaVinciTools/x86_64-slc5-gcc43-opt/libDaVinciTools.so' in '/mnt/shared/jpalac/cmtuser/DaVinci_v26r4/InstallArea/x86_64-slc5-gcc43-opt/lib'
Traceback (most recent call last):
  File "/opt/lhcb/lib/lhcb/GAUDI/GAUDI_v21r11/GaudiPolicy/scripts/install.py", line 309, in <module>
    main()
  File "/opt/lhcb/lib/lhcb/GAUDI/GAUDI_v21r11/GaudiPolicy/scripts/install.py", line 84, in main
    opts.symlink, realpath(dirname(opts.logfile)))
  File "/opt/lhcb/lib/lhcb/GAUDI/GAUDI_v21r11/GaudiPolicy/scripts/install.py", line 266, in install
    update(src,dest,old_dest,syml,logdir)
  File "/opt/lhcb/lib/lhcb/GAUDI/GAUDI_v21r11/GaudiPolicy/scripts/install.py", line 231, in update
    os.symlink(src,realdest)
OSError: [Errno 1] Operation not permitted
echo ${full_source_name} >${ref_file}
if test "${makecmd}"; then set +v; fi
#CMT---> installation done
#CMT---> (constituents.make) DaVinciTools done
#CMT---> (constituents.make) Building DaVinciToolsRootMap.make
#CMT---> Info: Document DaVinciToolsRootMap
#CMT---> (constituents.make) Creating DaVinciToolsRootMap.lock and Starting DaVinciToolsRootMap
lockfile: Try praying, giving up on "DaVinciToolsRootMap.lock"
gmake[3]: *** [DaVinciToolsRootMap] Error 69
gmake[2]: *** [common_target] Error 2
gmake[1]: *** [check_config] Error 2
gmake[1]: Leaving directory `/mnt/shared/jpalac/cmtuser/DaVinci_v26r4/Phys/DaVinciTools/cmt'
#CMT---> Error: execution_failed : make
make: *** [Phys/DaVinciTools_build] Error 2

This is because VirtualBox doesn't allow links on the shared area. I don't think this can be easily solved.

Cannot SSH to CernVM.

In the past, I used to ssh to the VM via port 2222:

alias cernVM='ssh -Y -p 2222 jpalac@localhost'

This now gives

ssh: connect to host localhost port 2222: Connection refused

The same problem happens when I ssh to the default port, 22.

Since I am running X, this is no longer a serious issue, but I'd like to know what ssh magic has to be done to get this to work again.

Copy and paste

I have not been able to figure out how to copy-and-paste in the X session to between emacs or xterm and anything else. Using a terminal emulator I can cut and paste easily between guest and host.

-- JuanPalacios - 07-Jan-2011