HCal offline DB instructions for the CBPF Group

How to change or make new tags in the offline DB

The instructions to follow are those from HCalOperationsDatabaseInstructions, with a couple of changes, listed below:

• Using CMSSW_6_1_0_pre4 (or above), which is necessary for the new DropBox uploading script (see next point). With this change, the steps 1,2,3, and 6 of the above instructions are still working fine.

            cmsrel CMSSW_6_1_0_pre4
            cd CMSSW_6_1_0_pre4/src/
            cmsenv
       

  • If using CMSSW_7_0_X (for global runs in 2014, for example) the writing script works but not the dump script. One should create the sqlite files in CMSSW_7_0_X and then for the validation dumps use the previous release CMSSW_6_1_0_pre4. According to Salavat it is a problem with our 'dumper' case, where HcalTopology is not really taken from Geometry record, which should be fixed at some point by Yana Osborne. It also needs new versions of the scripts for the IdealGeometry module.

• Omitting steps 4 and 5 and instead follow the new DropBox guidelines for submitting changes. This new upload script requires CMSSW release 6 or higher, in order to have Python2.6 and pycurl available (so also, one needs to log into a machine with SLC6).

• The connection strings have also changed:
  • For the conditions of HcalRecoParams and HcalLongRecoParams is frontier://FrontierProd/CMS_COND_44X_HCAL. This is also the account to use for tags aimed at CMSSW releases above 53X (And the one I used to submit those for upgrade simulation) unless instructed otherwise.
  • All others (HcalPedestals, HcalGains, HcalGainWidths, HcalPedestalWidths, HcalQIEShape, HcalQIEData, HcalChannelQuality, HcalElectronicsMap, and HcalDcsMap) use frontier://FrontierProd/CMS_COND_31X_HCAL. This account contains the conditions for the 2012 run, so tags relevant to CMSSW<=53X should go here.
  • The development database connection string is frontier://FrontierPrep/CMS_COND_HCAL, which can be used for uploading tests, for example.
  • Notice that for the upload (i.e. in the destinationDatabase field of the metadata file used by the upload script or in the wizard) the connection string should start with oracle://cms_orcon_prod (or oracle://cms_orcoff_prep for the development DB) instead of frontier://FrontierProd (or frontier://FrontierPrep, respectively), since frontier is read-only.

• Once the new IOV payloads are uploaded, one needs to fill the documentation tables for each object, like the ones in the links above. (The complete list is here).

Especial instructions to create a new tag from an older one

If the request if for a new tag, where a new IOV is included with new conditions, but older IOVs stay with the same information, there is a way to "clone" the older tag and then create a new one based on that.

• The following command exports the tag HcalChannelQuality_v5.00_offline, present in CMS_COND_31X_HCAL, to a local sqlite called newtag1.db with a modified tag name myHcalChannelQuality_v5.00_test

      cmscond_export_iov -s frontier://FrontierProd/CMS_COND_31X_HCAL  -d sqlite:newtag1.db -i HcalChannelQuality_v5.00_offline -t myHcalChannelQuality_v5.00_test 

• The new conditions in txt form can be written to a test.db sqlite file as instructed above, using the write.csh script to write into the same tag myHcalChannelQuality_v5.00_test

• The two sqlites can be then merged with the following command. Sometimes it might be necessary to add the options -i (input tag option, if the name of the final tag should be different) or -b (since or beginning IOV).

      cmscond_export_iov -d sqlite:newtag1.db -s sqlite:test.db -t myHcalChannelQuality_v5.00_test 

• In order to check the result n the new sqlite, you can do one or all of the following:
  • cmscond_list_iov -c sqlite:newtagtest.db -t myHcalChannelQuality_v5.00_test lists the IOVs (and should have one more IOV than the original HcalChannelQuality_v5.00_offline, coming from the test.db addition)
  • ./makedump.csh ChannelQuality myHcalChannelQuality_v5.00_test after modifying the connection string inside the makedump.csh script to sqlite:newtag1.db will dump the txt file that can be compared to the input of test.db (choose a value that makes sense, i.e. RN>= start of new IOV) to compare through a diff to the input txt.

• Then the file to submit to dropBox will be newtag1.db, following the DropBox twiki. There's a new dropbox logs website.

       ./upload.py newtag1.db

• Other useful commands can be found in WriteToHcalDB

Other things to consider

• In addition to subscribing to the cms-cond-dropbox egroup one should request the cms-offlinedb-exp @ SPAMNOT cern.ch to allow one privileges to use the upload script. Also, if the data type uploaded does not yet exist in the account (as was the case with Jordan's new conditions for the HOSiPMs), these experts (Miguel Ojeda and Giacomo Govi) have to create the tables (schema) for all the new types that one needs to store before one uploads them.
• If for some reason the previous upload failed and one has to re-submit the exact same .db file one should change a bit the userText (like add a space or something of the sort), in order to avoid the following error message:

    ERROR: The uploaded file with hash NNN already exists in the Acknowledged files (i.e. files that were already pulled by online not too long ago -- we do not keep all of them forever). This probably means that you sent the same request twice after some time.

Useful Tips and Tricks

• To check a recently uploaded tag in oracle, since it takes some time to be replicated to Frontier, you can do, for example:

       cmscond_list_iov -c oracle://cms_orcon_adg/CMS_COND_31X_HCAL -P /afs/cern.ch/cms/DB/conddb -a

• New script makedump_test.csh (based makedump.ch) doesn't need to be modified to change connection script, but it is taken as the last argument, for example:

        ./makedump_test.csh ElectronicsMap HcalElectronicsMap_v7.04_hlt 100004 frontier://FrontierProd/CMS_COND_31X_HCAL
        ./makedump_test.csh ElectronicsMap HcalElectronicsMap_v7.04_hlt 100003 sqlite:mylocalfile.db

• New script write_modified.csh (based on write.ch ) takes as the last argument the name of the sqlite file to create (if 5th argument not specified, test.db is used by default). Example:

       ./write_modified.csh Gains NewGains.txt HcalGains_v2.08_express 216308 newgains.db

Implementation of object HcalOptFilter

The request was to take the 50 coefficients hard-coded in HcalSimpleRecAlgo.cc and read them from a database object. Salavat suggested we based this class on the existing HcalFlagHFDigiTimeParams and its dependencies as just a list of parameters and not a "per HcalDetId" table.

Zhen's instructions for the new class:

• Write the new class files that will be placed in CondFormats/HcalObjects
• Implement the use of the class and the hardcoded parameters in :
  • CalibCalorimetry/HcalAlgos
  • CalibCalorimetry/HcalPlugins
  • CondTools/Hcal
• Send the new class files to Giacomo Govi and the CondDB experts for review, they should provide changes in:
  • CondFormats/HcalObjects
  • CondFormats/DataRecord
  • CondCore/HcalPlugins
  • classes.xml
  • when everybody is happy, they build and send back all other CondDB necessities code

Jake Anderson suggested we move to a proper "per HcalDetId" table to avoid confusion in the implementation, when the base class HcalContainer was used.