The PROOF protocols

Introduction

The PROOF-kernel protocol (simply referred to as PROOF protocol) is the set of rules governing how the processes of a PROOF cluster (root.exe on the client, proorfserv.exe on master and workers) exchange information. The vector used for that are TMessage objects. The message types specific to PROOF are defined in $ROOTSYS/core/base/inc/MessageTypes.h . The PROOF protocol is defined by the order in which these messages are exchanged.

The range of messages reserved to PROOF is 1000-1999 . Currently only the first 52 are used.

An addition protocol, referred to as XPROOF protocol, describes the low level xrootd-based communication layer; the message types for the XPROOF protocol are defined in $ROOTSYS/proof/proofd/inc/XProofProtocol.h .

The PROOF protocol

As PROOF implements a multi-tier client-server architecture, the requests are generally initiating by the component having role of client (the client vis-a-vis of a master; the master vis-a-vis of a worker or a sub-master). When the request is fully processed, the component acting as server (the master, sub-masters or workers) sends the incremental log file and a 4-byte status code.

Technically, clients issue the request to the active nodes (the master, the workers) by calling TProof::Broadcast(...), and then wait for the reply in TProof::Collect(...). When a node is done, it sends back the log file by sending a kPROOF_LOGFILE message withe the length of the log file, followed by a set of raw buffers with the actual log file content; a kPROOF_LOGDONE message, including also the status code, ends the log file transmission and the request. Upon receipt of KPROOF_LOGDONE the node is removed from the active list; TProof::Collect returns when all the active nodes have sent their log file and status back.

Session setup

The session setup is the phase where a the processes are created and put in condition to talk to each other via the PROOF protocol. This basically comprises the opening of the connections (structured sockets), the minimal exchanges defining the session and (on master) the startup of the workers.

After opening the connection, the servers run TProofSetup::Setup which sends the following messages:

1. A welcome string (type kMESS_STRING)
2. The top master sends the session unique tag (type kPROOF_SESSIONTAG)
3. Information about architecture, compiler and ROOT version run by the server (type kPROOF_VERSARCHCOMP)

After this, on masters a TProof object is created which in turn creates the workers; the creation of a worker is made of to separated phases: the connection opening (including proofserv startup) and the server final setup; the two steps are initiated serially but kept separated: in this way the heaviest parts (the proofserv startup) are parallelized and the whole startup time considerably reduced. The end of each startup phase is signaled by the master to the client with a kPROOF_SERVERSTARTED message type; the master should therefore receive a number of such messages twice as the number of workers.

Clients will also ask regularly for statistical information (bytes read, real and CPU time) by issuing a kPROOF_GETSTATS; the reply is a message of the same type with the information, not followed by the log file.

The master sends information about the cluster size and worker ID to the workers (type kPROOF_GROUPVIEW) and ask the workers to reset their environment (type kPROOF_RESET).

The client ask the master to work in parallel mode with all workers (type kPROOF_PARALLEL) and broadcasts the log level to be used for the session (type kPROOF_LOGLEVEL).

Typical request

The XPROOF protocol

Opening a connection

Login

Authentication

Sending a message

Urgent actions

-- GerardoGanis - 05 Jun 2008