What's new

This is the first release of glite-CLUSTER. It is a new node type that can publish information about clusters and subclusters in a site, which be used by any number of compute elements.

Overview of interoperation with CEs

While glite-CLUSTER can publish information about clusters and subclusters it should not publish UniqueIDs which are already published by another node such as lcg-CE or creamCE. For instance since version 3.1.46 the lcg-CE can be configured in 'cluster mode' which will prevent it from publishing its own cluster/subcluster IDs and allow the glite-CLUSTER node to be used to do that.

Configuration

Currently the creamCE yaim configuration has no option to disable the publication of its own cluster and subcluster IDs and so can not be directly configured to work with glite-CLUSTER. However the manual steps shown below may be used to use creamCE with glite-CLUSTER:

Steps to modify an already configured creamCE to work with glite-CLUSTER on a separate machine:

1. Consider how many new clusters are wanted. One can use the same cluster for one or more queues. The cluster associated with each queue may be changed individually leaving some with the built in cluster, to be changed later. Currently each cluster needs exactly one subcluster. For each new cluster decide on a globally unique ID - the associated subcluster ID can be the same as the cluster ID. Configure the glite-CLUSTER node to publish the new cluster/subcluster records. See https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables#CLUSTER for information on how to do that with YAIM. Note that the unique IDs must be globally unique and may not contain uppercase letters, e.g. they could be based on the site DNS domain name to ensure they are unique across the grid. To avoid possible confusion with the built in cluster ID used by the creamCE do not use the hostname of the creamCE as the unique ID.
2. The directory creamCE: $EDG_LOCATION/var/info on the creamCE should be shared with the cluster node cluster: $EDG_LOCATION/var/info
3. Edit ${GLITE_LOCATION}/etc/gip/ldif/static-file-CE.ldif. For each GlueCEUniqueID block (queue) you want change the 'GlueClusterUniqueID=<ce hostname>' to 'GlueClusterUniqueID=<cluster id>' where the "cluster id" is the Unique ID of a cluster chosen in step (1). If all of the queues are changed go straight to step (5).
4. Edit ${GLITE_LOCATION}/etc/gip/ldif/static-file-Cluster.ldif and remove all the lines with 'GlueCEUniqueID' corresponding to the queues for which new clusters have been defined.
5. Once all of the clusters of all queues are changed remove the following files from the creamCE:

   rm -f ${GLITE_LOCATION}/etc/gip/provider/glite-info-provider-software-wrapper

   rm -f ${GLITE_LOCATION}/etc/gip/plugin/glite-info-dynamic-software-wrapper

   rm -f ${GLITE_LOCATION}/etc/gip/ldif/static-file-Cluster.ldif

   rm -f ${GLITE_LOCATION}/var/tmp/gip/glite-info-static-cluster.conf

6. OPTIONAL - Configure WNs with their subcluster Unique IDs: In order to allow VO managers to set subcluster specific application tags they need to use the --sc option with lcg-tags/lcg-ManageVOTag commands, giving the subcluster name (Unique ID). To find the subcluster ID on a given worker node the glite-wn-info command can be used, if the configuration file ${GLITE_LOCATION}/etc/glite-wn-info.conf contains the information. Therefore the WNs should have their subcluster ID set in the configuration file. YAIM will do this if given the information in the WN_LIST file at the time the worker nodes are configured.

Steps (3), (4) and (5) will need to be reapplied if yaim is subsequently run on the creamCE.

Known issues

1. There is a known issue when reconfiguring a glite-CLUSTER so that a subcluster ID is removed from the configuration. (e.g. a subcluster is completely removed or the subcluster's UniqueID is changed): Directories with the name of the old subcluster Unique IDs are not removed and need to be removed manually from the glite-CLUSTER node or they will continue to be published:

   cd $GLITE_LOCATION/var/info

   rm -rf old_subcluster_name

   where <old_subcluster_name> is the UniqueID of any subcluster which is no longer defined.

2. The cluster and subcluster Unique IDs, set with:

   CLUSTER_<cluster-identifier>_CLUSTER_UniqueID

   SUBCLUSTER_<subcluster-identifier>_SUBCLUSTER_UniqueID

   must not contain upper case letters, i.e. they may contain only lower case alpha numeric characters and any of the three characters '.', '_' and '-'. The unique IDs must be globally unique, for instance base it on the DNS domain to ensure it will not collide with an ID at another site. However, the cluster Unique ID do not have to be unique with respect to subcluster Unique IDs. e.g. currently a cluster will only have one subcluster and the subcluster UniqueID may be the same as the cluster UniqueID.

3. There should only be one subcluster defined per cluster. This is currently a restriction needed for correct operation of the workload management system.

4. The variable CE_HOST_<ce name>_QUEUES should be set in the yaim cluster configuration file, but if it is missing yaim does not give a warning or error.