Distcc as a service

Introduction

distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network.

distcc should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile.

This quickstart will show you how to use the it-puppet-module-distcccern module.

Please note it supports kerberos auth only and no pump mode.

It need access to AFS for using wlcg compilers.

Why self service ?

As IT ran a pilot service for many years, it became clear that users have different needs. A cluster tuned with some settings may conflict with other users expectations.

So we decided to provide a puppet module to be able to tune/configure to your needs.

Available clusters

Openstack IT Validation Cluster

(This is a temporary cluster for testing our puppet changes if you want a production class service you will have to create your own.)

At this time we have an Openstack cluster run by IT.

You can get a list of available machines by running the following on aiadm :

aiadm$ ai-pdb hostgroup lxdistcc --plain 

Monitoring : https://meter.cern.ch/public/_plugin/kibana/#dashboard/temp/Zzfqq1xWRx2P0n-zap3Y6Q

Atlas (Self Service Openstack cluster)

aiadm$ ai-pdb hostgroup voatlasgcc/gcc --plain 
aiadm$ ai-pdb hostgroup voatlasgcc/gcc47 --plain
aiadm$ ai-pdb hostgroup voatlasgcc/gcc49 --plain 
 

Machine are configured per hostgroup. The idea is to have only one compiler enable (WIP).

Documentation

This is not an introduction to Openstack or Puppet at CERN, you can read them first:

• Please read http://configdocs.web.cern.ch/configdocs/
• Please read http://cern.ch/clouddocs

Quickstart

Enable koji repository

Latest Distcc (https://code.google.com/p/distcc/source/browse/) has been recompiled in koji and is available through koji repos.

• Please check koji doc : https://twiki.cern.ch/twiki/bin/view/LinuxSupport/BuildingRPMswithKoji

Please note that an old version of distcc is available in slc{5,6}-extras and is still supported during the removal of the old physical cluster.

Install the new repositories for distcc6-stable on client machine (distcccern puppet module will do that for you on the server side.)

  [distcc6-stable]
  name=distcc cern version (krb enabled) [stable] (OS)
  baseurl=http://linuxsoft.cern.ch/internal/repos/distcc6-stable/x86_64/os
  enabled=1
  gpgcheck=0
  priority=5

Server

• You need, first, to include in your hostgroup definition the distcccern module:

      include ('distcccern')
     

• Configure modules options in your hostgroup yaml file :

     lcg_compiler_dir: "/afs/cern.ch/project/linux/dev/distcc"
     afs_cachesize: 8256000
     distccd_timeout: 1200
     

• lcg_compiler_dir can be setup to your own directory. Just copy the file available in the default location. this location need to be available to both clients and servers.
• afs_cachesize can be set to AUTOMATIC please refer to afs puppet modules for details.
• distccd_timeout in second adjust to your need. Default 1200.

Client

  # yum install distcc 

Set client options:

  $ export DISTCC_IO_TIMEOUT="600" (Default to 300)
  $ export DISTCC_HOSTS="--localslots=8 --localslots_cpp=16 builder1/$numcore,auth"

A oneliner to generate DISTCC_HOSTS variable from an hostgroup name:

  aiadm$ LIST="--localslots=8 --localslots_cpp=16"; for i in `ai-pdb hostgroup voatlasgcc/gcc --plain` ; do LIST="$LIST $i/8,auth"; done; echo "export DISTCC_HOSTS=\"$LIST\"";
  export DISTCC_HOSTS="--localslots=8 --localslots_cpp=16 aidistcc001.cern.ch/8,auth aidistcc002.cern.ch/8,auth aidistcc004.cern.ch/8,auth aidistcc003.cern.ch/8,auth"