Resource brokering in PROOF

---

Contents:

---

Introduction

With ROOT 5.08.00 we introduced a class - TProofResources - modelling the resources needed on a PROOF cluster, i.e. the worker nodes. These is an abstract class whose only available implementation is so far TProofResourcesStatic , in charge of parsing and decoding the standard static configuration file.

The integration of this model in the code has been done by hard-coding the instantiation of TProofResourcesStatic in TProof::StartSlaves . The purpose here was to prove that we can actually work in this was and to clean-up the code.

But in general we plan to have several implementations of these resource finder - already prototyped are implementations for LSF, PLB, ... The hard-coded way is therefore non optimal.

Moreover, a study of possible ways to implement multi-user scheduling showed that resource brokering should done elsewhere (in the coordinator or in the scheduler) and should be available in the master via its coordinator.

What remains is a method

EQueryAction TProofServ::GetWorkers(TList &toBeStarted, TList &toBeStopped, Int_t &changePriority)

which should be called by TProof::StartSlaves or TProof::ConfigWorkers() .

The coordinator should be in charge of parsing the config file (or query any other resource) to produce a list of object usable to start the session.

Here is what we need to implement all this:

1. TProofServ::GetWorkers
   This is a virtual method that will get implementation for TProofServ and TXProofServ; the first just parses locally the config file creating the list of TProofNodeInfo . The second asks its coordinator for the information, it receives a string with a list of '&' separated tokens used then to build TProofNodeInfo .

A first action must be to go back to ROOT convention for abstract classes by renaming TProofResources as TVirtualProofResources

New API

The main purpose of the new API is to decouple the actual implementation of the resource broker for the usage of the result.

Requirements

Re-scheduling and re-configuration of the resources are crucial issues when dealing with many users.

The system must be able to automatically find the best configuration, i.e. the one that minimize the latency in getting the result.

The user should be as less as possible exposed to determining this; typically it does not have all the elements take a decision. Even processing a few events can result in a relatively long query.

The only real way out is to "measure" the time needed using the first events.

The possible scenario could be something like this:

• the system start processing a query with all the resources it can give to the user assuming the query would be very short
• this should be enough to estimate the integral of the resources needed by the terminate processing the query
• the system should be able to decide if it continue with the current session configuration or if it has to re-configure giving up some resources.

New Ideas

(to be written)

---

• The user must be able to indicate how long a query is going to be; the reserved number of machine may depend on that.
  • This must also be possible between one query and the other

Typical use case

Sample of session start-up and query processing changing query lengths:

root[0] proof1 = TProof::Open("master.domain.one", EQueryLength::kShort);
root[1] chainShort->Process("MySelector.C")
...
root[n] proof1->SetQueryLength(EQueryLength::kLong)
root[n+1] chainLong->Process("MySelector.C")

The information

New TProof::Open signature

A possible new signature for the session start-up is
TProof::Open("master.url", EQueryLength length = kShort)
where
enum EQueryLength { kXShort, kShort, kMedium, kLong, kXLong }
(this enumeration is just for illustration and reflects what is usually found in batch systems - e.g. LSF at CERN: 8m, 1h, 8h, 1d, 1w ).
At this level, the user will not be able any longer to choose her/his configuration: what happens behind the scenes must be completely under control of the cluster administrator.

New TProof::ConfigWorkers(...) method

This method should initially what TProof::StartSlaves does now; however, it should be callable at any time to reconfigure the cluster accordingly to instant needs. It should perform the following actions:

• • Get the list of available resources via TVirtualProofResources::GetWorkers(...)
  • Check it against the one of currently active slaves, if any:
    • the action on workers to be given back depends on the status of the session:
      • idle : those workers need just to be terminated
      • not idle : those workers need to be stopped and their partial results (and statistics) collected and added to the output list in the master
    • new workers must be started; this is the only case at session start-up .

The behaviour of this method may depend on additional parameters which indicated by the ellipses. These need to be defined.

Internal interface to the resource facility

The information about the query length will be used internally by TProof to query the resources. The minimal internal interface to the resource facility should have a signature as this
static TList *TVirtualProofResources::GetWorkers(EQueryLength length, ...)
The ellipses denote additional arguments which might be needed by resource facility; they will be discussed later on.

Action in case of time expiration

The concept of query length introduces two issues:

• the definition of the maximum time associated to a query category
  • this is probably the task of the resource facility and the interface must provide access to this number.
• a policy defining the action to be undertaken when the time is expired.
  • this may depend on the query length, i.e. short or very short queries should be just stopped, saving the snapshot of the output list available, while for medium or long queries the action could be to lower the priority and/or reduce the resources.
  • Again, this may be defined at the configuration level of the resource facility by the cluster administrator, and the interface must provide access to this information.

Definition of the ellipses

The definition of the ellipses requires decisions to be taken about some relevant issues.

• A global metric for the processing time used by the query (user)
  • e.g. the sum of the processing time in the worker nodes
• Who should provide this number
  • the master may be the best placed
• Should the decision depend on the history of the user or on the fact she/he may be running other sessions?
  • The information may be stored in the coordinator, so the possibility to trasmit / receive this kind of information must be envisaged

---

• Use the plug-in manager to load the correct class accordingly to administrator specifications, defaulting to static when no specification is found
  • this requires a standardisation of the list of arguments accepted by constructors; given that the number and type of parameters can be very different, the best way is probably to pass a string to be decoded internally by each specific class. The standard constructor signature would then be
    TProofResources...::TProofResources...(const char *)
    The string should contain '|' separated tokens of information in the form key>:value . For example, for the static case, we may be interested in: the name of the config file (default $HOME/.proof.conf); if private config files are allowed; the number of workers to be used (default: all those specified); if a subset of workers should be chosen randomly; etc etc
    Some keywords, like the number of workers, could be common to more implementations (it may be the only parameter for the time being for LSF, Condor or PLB).
• The API for the client of this class should not care about the machinery behind; at most it will specify a length index , which in the first version could be a batch-like queue index. The client will call a static method:
  static TList *TVirtualProofResources::GetWorkers(EQueryLength length, const char *opt = 0)
  where
  enum EQueryLength { kXShort, kShort, kMedium, kLong, kXLong }
  The second option is a generic string to be used as container for additional parameters that could be needed for the future. Inside TVirtualProofResources::GetWorkers() the plug-in manager should be used to load the appropriate resource class implementation according to an environment variable PROOFRESOURCES to be setup by the launcher xproofd following the choice of the administrator.
  The format of the variable PROOFRESOURCES should be:
  resource-type:key1=value1|key2=value2|...
  Here resource-type is the resource type identifier ("static", "lsf", "condor", "plb", ...) to be used to identify the right plug-in. The set of key=value pairs are parameters decoded by the implementation loaded.

• Currently the method to start the workers is TProof::StartSlaves : this is called at the beginning and its name reflects this aspect. Perhaps we should rename it TProof::ConfigWorkers , and modify it in order to achieve the following: %BR
  • Get the list of available resources via TVirtualProofResources::GetWorkers()
  • Check it against the one of currently active slaves:
    • if a slave needs to stopped the action depends on whether a query is being processed or not:
      • the slave should be stopped here if we are intra two queries
      • the slave should be added to a list of slaves to be stopped if we are processing a query; the slave will be stopped by the packetizer after receiving and one with the slaves to started.
  • If the method is called during a query, the slaves to be stopped will just will be actually stopped by the packetizer, after

A new generic method to setup the slaves should be introduced, to be called at any time a reconfiguration is needed. The name of the method should be TProof::ConfigWorkers

( TProof::StartSlaves and potentially another method to be called between queries; perhaps we could call it TProof::SetupSlaves and use it everywhere we need it, both at session start-up and later on when rescheduling)

-- GerardoGanis - 06 Mar 2006