Implementing an ATLAS Tier3 Virtual Machine

Motivation

Having a drop-in SLC4 machine, configured as necessary for Athena analysis with the latest useful releases, and equipped with appropriate DDM software, can be a great advantage for a physics end user who wants full Athena/ARA analysis on a laptop but does not have the time or expertise necessary to set this up.

The machine offered (for now) at http://www-hep.uta.edu/~stradlin/Tier3VM is an initial stab at such a system. Implemented in a standard VMware container is a SLC4.6 virtual machine, with separate VM disks containing ROOT 5.19 and 5.18 (built for the installation), Python 2.5, gLite 3.1 for grid and SRM support, DDM client tools, automatic SSH key generation scripts to ease linking with an lxplus account for authenticated CVS/cmt checkouts.

In preparation for this install, please be sure you have the following:

1. Your machine should probably be running a Core Duo processor or better for good results. The VX processor extensions help a lot with virtual machine performance. The more RAM you have, the happier you will be with your VM.
2. Either VMware Fusion, VMware Player for Windows, or VMware Player for Linux must be installed on the host computer you intend to use.
3. Verify that you have at least 30 GB of disk free for this.
4. More RAM is always better for virtualization, and 2 GB is more or less the minimum. RAM is cheap - one can find 4 GB for ~ $70, both in the US (Newegg) or in Switzerland (Prodimex). Invest.
5. Create a folder (like /DataShare on Linux or OS X, or c:\DataShare on Windows) intended for sharing with the VM. For performance reasons, this would ideally be on a different physical volume than your boot volume, though this is uncommon with laptops.

Configuration

This VM was created using the VMware Fusion product. While this should in principle work with any VMware product, including the free Player software, I have made no tests to be sure that this is true. The Mac VMware Fusion version you should use with this particular VM is the VMware Fusion 2.0, which is fast, stable and easy to use, and allows up to 4 cores per VM.

This VM is build to be as CERN-independent as possible (though there will surely be changes and improvements to this as time goes on). The idea is to be able to install it on any site, in any country, or on your laptop, without CERN services dependencies, as bandwidth is sometimes a problem.

This installation contains the following:

• SLC 4.6, kernel 2.6.9-67.EL.cern, rebuilt with correct parameters for good VM performance. These kernel mods DO make some timeouts a LOT longer -- sudo and login, for example.
• Rebuilt AFS for kernel compatibility
• Athena 14.2.20 (Athena 13.0.40, with recent EV Group Area, and Athena 14.2.0 are also available)
• ROOT 5.18 and 5.19, built to order and preinstalled
• DQ 2 Client Tools (the new dq2-ls and etc.)
• gLite 3.1
• Rebuilt Python 2.5 for grid tools compatibility
• VMware Tools installed for ease of use in GUI mode
• vmnet network drivers for enhanced performance

The present VM version has been touched up a bit (version 3.1.1), but is functionally the same as previous releases. The corrected VM images were posted the evening of Sept. 19, 2008.

Network Configuration and Data Storage

• The VM will be operating in NAT mode. This means that the host machine runs a process that emulates a router with a NAT enabled. Most NATs run in the address space of 192.168.xxx.xxx. Your VMware installation will choose the 192.168.xxx part of this, and the DHCP server inside that NAT will assign the last three digits at will to individual VMs. Since this is a NAT, these IP addresses are not accessible from the outside without special configuration. The NAT emulator is presently somewhat processor-hungry. We recommend using it in spite of this defect because shared filesystems with the host are much faster in NAT mode than in bridged mode.

• The VM is NOT operated in Bridged network mode because of the severe constraints on file transfer bandwidth we observed in VM benchmarking. It is also worth considering that the VM will be exposed to the breadth of the Internet in bridged mode, while in NAT mode is it fully isolated from incoming attacks.

• This interest in shared filesystems (such as hgfs, NFS and xrootd) arises from a desire to be able to store both datasets and ATLAS software installations once and give access to both the host and several VMs as necessary, with the most performant access possible.

Installation

1. Download the file located at http://www-hep.uta.edu/~stradlin/Tier3VM/Tier3.1.1.tbz ( Tier3.1.1.tbz). You can also download Atlas14.2.20.tbz Atlas14.2.20.tbz if you intend to install Athena (see below).
2. Untar the virtual machine (tar -xjf Tier3.1.1.tbz) and place it somewhere on your boot volume.
3. In the VM Settings (varies for different platforms), set a Shared Folder to point to your /DataShare directory, if it has not happened automatically.

Logging in and first run

1. Start up the VM. If it asks whether this VM has been copied or moved, state that it has been copied. This will change the MAC address for the VM, preventing future conflicts.
2. As the VM starts up, it may have to reconfigure its virtual network card to account for the new IP address. Delete the old hardware configuration as prompted, and configure the new one to work via DHCP. Continue the boot process. If this does not happen during boot you will need to configure the network card after login.
3. Once booted, log in to the VM as tier3, password T3ChangeMe.
4. Launch the GUI using the command startx.
5. As mentioned earlier, change the password as soon as possible (using Applications->Preferences->Password). The same password allows root access - change that as well using Applications->System Settings->Root Password.
6. Run /sbin/ifconfig to find out the IP address your VM has been assigned by the VMware NAT daemon.
   • If the address you see is inet addr:127.0.0.1, open Applications->System Settings->Network and enter the root password when prompted. In the Devices tab, you will see device eth0. Select it, then click the Edit button (the wrench icon) in the toolbar. Select the Hardware Device tab in the window that comes up, and you'll see an entry called Bind to MAC address, with a button beside it labeled Probe. Click that button, and select the OK button at the bottom of the window. Click on eth0 again, and click the Activate button in the toolbar (the green checkmark). Click to accept any dialogs you see. Run /sbin/ifconfig to find out the IP address your VM has been assigned by the VMware NAT daemon.
7. Edit the /etc/hosts file, changing the IP address (192.168.150.135) to match the IP address you got from the previous step
8. If you created a /DataShare directory as mentioned previously, go to the VMware Settings (the lightswitch in the VM toolbar), select Shared Folders, and click the Enabled checkbox.
9. Install the Athena release as follows (also found in the README file)

Working Account Setup

1. Once in, change the passwords.
2. Run the scripts in the Scripts directory for keyless SSH setup with lxplus
3. Run the Athena setup scripts (assuming you have already installed Athena).
4. Change things about the VM to suit your needs. Some of these things might be:
   • Turn on KDE activation when the machine starts by editing /etc/inittab and changing id:3:initdefault: to id:5:initdefault:.
   • Shell and other user prefs (using Applications->System Settings->Users and Groups)
   • sudo permissions - On the command line, do the following:

     su

     export EDITOR=/usr/bin/vim;/usr/sbin/visudo 

   • Working by example from the "root" line, add username "tier3" (and any other users you might need to have sudo privileges) - save

     exit

User account creation and configuration (if desired)

1. Using Applications->System Setttings->Users and Groups, create a new username for yourself. It is best to match the CERN username for convenience in working with AFS. Likewise, the user ID and group ID are nice things to get right the first time. First, create a zp group (for Atlas) with group ID 1307. Then create your user with the appropriate user ID. If you don't know your CERN UID, log on to lxplus and do a ls -n in your user directory. You'll see the UID in the third column.
2. Copy the files (including all the .* files) to the user directory from the ~tier3 directory. This is best done as your new user.
3. Add a sudoer entry if necessary (see previous)
4. Copy in any grid certificates necessary (see below)
5. Make any necessary changes to the .bashrc
6. If you have already configured Athena/CMT with the scripts, do so again.

Athena Installation

Two methods can be used - both take substantial download time.

• For Athena 14.2.20, download Atlas14.2.20.tbz, and untar it - tar xjf Atlas.tbz. When untarred, copy it into your Tier3.1 VM to replace the existing Atlas.vmdk file. You can also download the AtlasOld.tgz package, giving you Athena 13.0.40 and 14.2.0.

OR

• Go to /Atlas/pacman-3.25. Type source setup.sh, then go to the /Atlas/Athena directory. Run the command pacman -allow tar-overwrite -get am-BNL:14.2.20, and follow the directions. You can do the same thing in the same place for different versions of Athena, and they will all install correctly. You can also choose a mirror closer than IU (options are BNL, RAL, CERN, IHEP, BU ad UM).

Once this is done, Athena is in place. Run the scripts in

~/Scripts

to set it up and make it active.

Updating Athena is simple, and does not require a full separate copy of all redundant files. * Go to /Atlas/pacman-3.25. Type source setup.sh, then go to the /Atlas/Athena directory. Run the command pacman -allow tar-overwrite -get am-BNL: where would be the Athena version you want.

Grid Setup

Your Grid client is already in place, but may need to update its CRLs. Run:

sudo /opt/glite/libexec/fetch-crl.sh

After that, all that is necessary to gain access is to copy your usercred.p12 file to your ~/.globus directory, and do either a

grid-proxy-init

or (preferably) a

voms-proxy-init -voms atlas

NFS Configuration

On OS X Leopard, it's terribly simple to set up a host NFS server. Add the line:

/DataShare -network 192.168.0.0 -mask 255.255.0.0

to

/etc/exports

then,

sudo nfsd restart

On the guest side, add the NFS shared drive to your VM's /etc/fstab: sudo mkdir /DataShare

sudo cat "192.168.190.1:/DataShare   /DataShare   nfs   rw,rsize=32768,wsize=32768" >> /etc/fstab
sudo mount -a

The 192.168.xxx.1 will change based on what you see when you type

/sbin/ifconfig

The IP address you assign to the NFS server in this network should be the same as the guest VM's IP address, with a "1" as the last of the four IP address fields.

And all is done.

xrootd Setup

Also simple. I assume that you have installed ROOT on the host machine. On OS X Leopard, ROOT versions before 5.19 are not recommended. Once ROOT is set up on the host, create the following /Installations/root/xrootd.cf file:

all.role server
xrootd.fslib $ROOTSYS/lib/root/libXrdOfs.so
xrootd.export /DataShare
xrd.port 1094
xrd.adminpath /tmp

Run the command:

xrootd -b -l /tmp/xrootd.log  -c /Installations/root/xrootd.cf

And on the guest, all xrootd calls can be made as

root://192.168.150.1//DataShare/...

Recommendations

The VM you download here has been starved down, minimizing the disk sizes for bandwidth reasons. They will expand as necessary to take your datasets. Use the Shared Folders as your dataset containers, but do not install software in them, or trust their reported filesizes.

Installation of Athena on a VM disk is recommended. If you want to share the Athena install across multiple VMs, and are in a place where AFS is too slow for Athena use on a routine basis, it is simple (on linux or OS X Leopard) to establish NFS disks that can easily share data across a set of machines, both real and virtual. See instructions above. These VMs also have ROOT (and therefore xrootd) available for data sharing.

If you want to shrink or expand the VM disks, do so by creating new disks and copying the contents of the old disks over to them. VMware offers a slow, inefficient solution for disk size management that I have abandoned - it is far faster to do the whole thing by hand. Detailed instructions can be found here. Until that page is actually working, you can email me if you need instructions for this.

Single-Processor Machines (for previous VM versions)

VMs created in April 2008 had assumed multicore host machines. If you have such a computer and the VM will not boot, please select (at the very beginning of the boot process) a different kernel - the CERN non-SMP kernel. Once logged in as root, download

http://www-hep.uta.edu/~stradlin/Tier3VM/kernels/kernel-2.6.9-67.0.7.EL.cernvm.i686/kernel-2.6.9-67.0.7.EL.cernvm.i686.rpm

and install it via

rpm -ivh kernel-2.6.9-67.0.7.EL.cernvm.i686.rpm

Then select the non-SMP cernvm kernel next time you boot.

I made a couple of errors in the setup scripts included with the April VM version. The most important is in the ~tier3/Scripts/KeyGeneration.sh script, there is a malformed line:

Host *cern.ch lx*

which needs to be

Host lxplus.cern.ch

If you have already run this script, the ~tier3/.ssh/config file needs to have that modification made to it.

(Older) Athena Installation and Configuration

cd /Atlas/Athena
bash /mnt/hgfs/DataShare/13.0.40.time-2008-04-11-21-06-08.md5-80de7efd6acfef389bac87283a46d7ab.sh
cd /Atlas/EVGroupArea/AtlasOffline-13.0.40
bash /mnt/hgfs/DataShare/EventView-13.0.40.323.pacman.time-2008-04-18-17-42-18.md5-abda37fa98d033b6a540db86af3461b0.sh

Once this is done, Athena is in place. Run the scripts in

~/Scripts

to set it up and make it active.

-- AldenStradling - 17 Apr 2008