Socket monitoring in XPD

Introduction

We discuss here the socket monitoring system for XPD.

Types of connections

Basic XPD connection are implemented by the TXSocket class. This class inherits from TSocket and from XrdClientAbsUnsolMsgHandler . Inheritance from TSocket simplified its integration in the existing PROOF structure. Inheritance from XrdClientAbsUnsolMsgHandler allows this class to exploit all the asynchronous functionality of XrdClient connections.

TXSocket contains an instance of XrdProofConn , the XPD version of the connection module; XrdProofConn plays the same role played by XrdClientConn for XrdClient , i.e. it describes a logical connection aware of the XProofProtocol ; a PROOF-dedicated static instance of the connection manager XrdClientConnManager is owned by XrdProofConn ; this is needed to allow working concurrently with the data serving and PROOF serving protocols (the two cannot share the same physical connection).

A variant of TXSocket, TXUnixSocket , is used for the internal connection between proofserv and XPD; as the name suggest, this class uses UNIX sockets.

Reader thread

XrdClient low level connections ( XrdClientPhyConnection ) use a separate reader thread to read incoming information. Messages coming from the server can be of two kinds: solicited , i.e. following an explicit request of the counterpart, or unsolicited . The latters are just propagated up until the appropriate handler is found. Solicited messages are added to the input buffer ( XrdClientInputBuffer ) and picked up from there by the waiting process.

Types of messages

XPD makes an heavy use of unsolicited (also referred to as asynchronous ) messages. All operation messages are forwarded as unsolicited. Acknowledgement messages are send back as solicited, as they are requested by the sender. A crucial role is therefore played by the unsolicited message handler ( TXSocket::ProcessUnsolicitedMsg ).

Asynchronous collection

Asynchronous collection is done via TProof::CollectInputFrom() . In classical PROOF this procedure is associated to an input handler and called inside its Notify() method.

Synchronous collection

This is done via Collect(TMonitor *mon) . The monitor contains as active sockets those from which something is expected. The sockets are registered in TMonitor via instances of TSocketHandler . When something is available on the socket, TSocketHandler::Notify() sets the socket as ready , and the TMonitor::Select() returns a pointer to the fired socket. This approach cannot be directly applied on XPD connections, because the socket is monitored by the reader thread and its activity cannot be seen not in the main event loop.

Synchronous / asynchronous collection in XPD

To keep unchanged the PROOF internals a workaround had to be found to allow monitoring the connection activity in the main event loop. The workaround is based on a pipe and a new type of socket handler, TXSocketHandler associated to it. The pipe is needed because message production and procesing are done in different threads, which therefore need to communicate. The pipe is implemented as a static instance in TXSocket . An event is posted on the pipe (a simple character is written to it) by TXSocket::ProcessUnsolicitedMsg whenever a unsolicited message is arrived (these are the only messages that need to be monitored in the main event loop). Depending on the type socket or context, TXSocketHandler::Notify performs the appropriate action, which can be:

1. if the socket is the input socket in proofserv invoke TXProofServ::HandleServInputHandler
2. else set the socket ready in the active TMonitor instance
3. if no monitor instance is active, invoke TProof::CollectInputFrom(TSocket *s) for the fired socket

Worries about this approach

The use of a pipe is a workaround for the impossibility to monitor directly the physical sockets. The problem with pipes is that they are resource-consuming and not directly portable to Windows (BSD pipes on Windows are not selectable ). An alternative approach based on a timer to watch on possible readiness at a given frequency is under study.

-- GerardoGanis - 22 Mar 2006