How to write conditions data - PoolDBOutputService

%COMPLETE3%

Overview

Parameters and public interfaces described in this page are compatible wit CMSSW_12_1_X and higher. For using older release versions, read older version of the page.

In CondCore system, edm service module PoolDBOutputService is responsible for persistify conditions data as well as managing the interval of validity, it takes care of all the interactions with the database. A documentation about how to use an edm service in general can be found here. In software version higher than CMSSW_1_8_0, logging feature is added. One can choose to log transaction status into a log DB which may or may not be in the same account as the real data. This feature is the building block of the CMS.PopCon application.

Writing conditions data into database consists of two parts: writing the bulk payload data and assigning the interval of validity to the payload. The payload data must be defined as objects by the user. User is required to provide the dictionary information for each object. The user does not need and should not write any database specific code in his/her analysis module nor in the object definition. Guidelines and procedures for creating new persistent-capable objects and the dictionary are described in details here.

A detailed description of the concept of interval of validity(IOV), IOV metadata and IOV tagging policy can be found here. In the current implementation, an IOV index is always closed with the end-of-time.

To resolve the PopConSourceHandler leaks memory issue, new policy is advice to keep the ownership of the payload object at the client side. The new reference based method, writeOneIOV, is introduced and encouraged to migrate.

Service callback functions

The following public functions of PoolDBOutputService can be called by the client module :

writeOneIOV : write one object, either create or append. The template function stores also the given payload object and takes the reference of the payload object as the first parameter.

writeOne : Obsolete use writeOneIOV write one object, either create or append.

createOneIOV : this is the new reference based method to replace the Obsolete createNewIOV.

createNewIOV : Obsolete use writeOneIOV create a new IOV sequence. There are two forms of the function. The template function stores also the given payload object. Note: the function takes the ownership of the payload object pointer. The non-templated function attaches IOV to existing payload objects. Till-time of the first payload is required. If since-time is not specified, global begin-of-time is assumed.

appendOneIOV : this is the new reference based method to replace the Obsolete appendSinceTime and appendTillTime.

appendSinceTime : Obsolete use writeOneIOV append since time. There are templated and non-templated forms of the function.

isNewTagRequest : returns true if the tag name specified in the configuration is new to the given EventSetup record name

currentTime : returns the current time of the event according to the "timetype" parameter specified in the configuration

endOfTime : returns the end-of-time value according to the "timetype" parameter specified in the configuration

tag : returns the tag name attached to the given EventSetup record name

setLogHeaderForRecord : set log entry header for a given record

tagInfo : retrieve the tag information associated with a record

queryLog : get the DB log interface

Configuration Parameters for PoolDBOutputService

name	type	description	default	allowed
connect	string	database connection string	None	POOLORA connection string format
timetype	untracked string	the type of time variable	"runnumber"	"runnumber", "timestamp"
BlobStreamerName	untracked string	blobStreaming service to use	"DefaultBlobStreamingService"	"TBufferBlobStreamingService", "DefaultBlobStreamingService"
toPut(see below)	VPSet	records to put into the database	None	comma separated parameter pairs
DBParameters(see below)	named Parameter Set	common database connection parameters	default values provided by the common .cfi file

Parameters expected in toPut VPSet

name	type	description	default	allowed
record	string	EventSetupRecord the data belong(required)	None	existing records
tag	string	the tag to the iov(required)	None	non-empty strings

Named Parameter Set DBParameters sets the connection parameters. The default values are provided by the framework in the following files CondCore/DBCommon/data/CondDBCommon.cfi or CondCore/DBCommon/data/CondDBSetup.cfi.

The common practice is to include one of the above default .cfi in job configuration and replace the parameters you want to customise. Parameters in named parameter set DBParameters

name	type	description	default	allowed
authenticationPath	untracked string	directory path to the authentication file	"." current dir	empty string or a valid path. If empty string, environment variables CORAL_AUTH_USER, CORAL_AUTH_PASSWORD should be set by the user
messageLevel	untracked int32	verbosity level	0	0(Error),1(Warning), 2(Info), 3(Debug)
enableConnectionSharing	untracked bool	connection sharing switch	true	true, false
connectionTimeOut	untracked int32	connection timeout value	600(sec)
connectionRetrialPeriod	untracked int32	connection retry time interval	30(sec)
connectionRetrialTimeOut	untracked int32	connection retry timeout	180(sec)

Hint: If in one job you use more than one Cond DB modules and connect to more than one database instance, include CondDBSetup.cfi instead of CondDBCommon.cfi and define the connect and catalog parameters for each DB instance.

Note: parameter DBParameters.authenticationPath should point to the location of your authentication.xml. For example, at CERN one of the locations on AFS is "/afs/cern.ch/cms/DB/conddb".

How to create a new IOV sequence

In your algorithm module EDAnalyzer, get hold an instance of cond::service::PoolDBOutputService:

edm::Service<cond::service::PoolDBOutputService> poolDbService;
if( poolDbService.isAvailable() ){ //use it to write new data}

Then assign the interval of validity to the new payload object using the public callback function writeOneIOV. A snippet of a EDAnalyzer module MyCalibAlgo is shown below as example:

    edm::Service<cond::service::PoolDBOutputService> mydbservice;
    if( !mydbservice.isAvailable() )  return;
    Pedestals myped; // Create a normal instance of Pedestals object
    //fillup myped
    //create new infinite IOV
    cond::Time_t firstSinceTime=mydbservice->beginOfTime();
    mydbservice->writeOneIOV(myped, firstSinceTime, "PedestalsRcd");

Note: the payload object, e.g. Pedestals, should not be created with the new operator. A normal object instance or a smart pointer is recommended. The user takes the ownership of the object and should delete it when no more needed.

Hint: in the simple case of writing a payload with infinite IOV, you can put the DB service related calls in the function EDAnalyzer::endJob since you do not rely on the time information. If you need event time information for assigning IOV, put Cond DB calls in EDAnalyzer::analyze method and ask for currentTime from the PoolDBOutputService.

Each new IOV sequence must associate to a tag. The tag should be unique for your task, it should not already exist in the database you are writing into. The tag should be specified in the configuration parameter toPut of PoolDBOutputService. Shown below is an example job configuration:

Example:

   process MYTEST={
        include "CondCore/DBCommon/data/CondDBCommon.cfi"
        replace CondDBCommon.connect = "sqlite_file:test.db"
        replace CondDBCommon.DBParameters.messageLevel = 3
        replace CondDBCommon.DBParameters.authenticationPath = "/afs/cern.ch/cms/DB/conddb"
        source = EmptyIOVSource {
           string timetype = "runnumber"
           untracked uint32 firstRun = 1
           untracked uint32 lastRun = 1
           uint32 interval = 1    
        }
        service = PoolDBOutputService{
           using CondDBCommon
           untracked string timetype="runnumber"
            VPSet toPut = {
              {string record = "PedestalsRcd" string tag = "mytest" }
           }
        }
        path p = {mytest}
        module mytest = MyCalibAlgo{
              string record = "PedestalsRcd"
        }
}

How to append IOV to an existing tag

There is no difference in job configuration for appending new data to existing tag and for creating new IOV. However, the PoolDBOutputService call logic inside your EDAnalyzer may become slightly more complicated if you insist that the first entry shall start from beginOfTime. We modify MyCalibAlgo example shown above to include the append usecase:

    cond::Time_t sinceTime =  mydbservice->isNewTagRequest("PedestalsRcd") ? 
            mydbservice->beginOfTime() : mydbservice->currentTime();
    mydbservice->writeOneIOV(myped, sinceTime,"PedestalsRcd");
 

More examples can be found in CMSSW/CondCore/DBOutputService/test/stubs and data directories.

How to log status into log db

This is the machinary behind the CMS.PopCon applications. Though PoolDBOutput users can use the logging feature as it is, it is mainly expected to be used by the CMS.PopCon package. In the service configuration file, set the untracked string parameter "logconnect" to the log db connection string, the logging will be switched on.

untracked string logconnect = "sqlite_file:log.db"

If you want to set appropriate metadata to identify your log entry, in your source code, call the service method setLogHeaderForRecord.

Review Status

Reviewer/Editor and date	Comments

-- YuanChao - 2021-10-20