Datasets and Data Preparation Exercise

Note

This exercise page is an updated version of https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCMSDataAnalysisSchoolNTU2016PPDExercise The original exercises are created by Giovanni Franzoni (giovanni.franzoni@cernNOSPAMPLEASE.ch), big thanks to him!!

Overview

In this set of exercises you will learn how to look for datasets and find out their key properties relevant for analysis: how to navigate their parent-child relationship and determine the software release, production configuration and alignment-calibration conditions they've been produced with. You'll be exposed to using the principal services providing status and details of datasets: das, brilcalc, McM, pMp, cmsDBbrowser. You'll also find out how to compute the integrated luminosity for your analysis.

Introduction to datasets

Summarise there the content of the slides, add a pointer to them:

• For an introduction, please see the slides of the data preparation talk at the DAS@NTU school introduction
• Take note of these two documentation twiki: PdmVDataReprocessing: data reprocessing campaigns * PdmVMcCampaigns: Monte Carlo campaigns

Getting ready for these exercises

A bit of preparation will ease following this tutorial. Below a few concrete actions you can take before starting the hands-on session.

Prerequisites

In order to carry out the exercises of this session you need:

• a CMS account at CERN to access web services (you can access this twiki you have it) and log into lxplus

Color coding conventions

There's color scheme followed in this page exercises:

• Commands to be typed into a shell will be embedded in a grey box:

  ls -ltr

• Other code, scripts and queries will be displayed in a pink box:

   const TString var = "ST";

• Output and screen printouts will be embedded in a green box:

   Number of events with ST > 2000 and N>= 3 is 10099. Acceptance = 0.842075

It is expected that you can cut-and-paste from the command box into the command line. Similarly, you should be able to cut-and-paste from the configuration fragments directly into a text editor, when necessary.

Setup a CMSSW area

Most of the exercises will be carried out using your web browser.

A CMSSW work area will also be needed; please create a working directory on your lxplus home

ssh -XY your_login@lxplus.cern.ch
mkdir data-preparation-exercises/
cd data-preparation-exercises/
cmsrel CMSSW_8_0_21
cd CMSSW_8_0_21/src
cmsenv 

Now you are ready to proceed to the exercises.

Note: CMSSW_8_0_21 is the release which contain changes for L1 software, desired for Moriond17 Monte Carlo (DIgi-Reco) production.

Exercise 1: find an accessible file with single electron events in miniaod format from the latest data reprocessing of 2016D

Reminder about the general structure of a dataset name:

 dataset = /PrimaryDataset/ProcessingVersion/DataTier

Examples:

dataset = /SingleElectron/Run2016D-23Sep2016-v1/MINIAOD

To start with, you need to establish the full dataset name "with single electron". Either you know it from a reference, or you need to construct a key elements of it, and put them together in a search. With the key elements at hand, you'll be able to use the das web interface and queries with wildcard in order to establish or confirm the complete dataset name.

1. what is the primary dataset name ? 1. what is the primary dataset name ?

2. what is the Processing Version ? 2. what is the Processing Version ?

3. what is the Data Tier ? 3. what is the Data Tier?

You can now place a query to das to find the dataset; using the wildcard increases the chances of finding at the first try what you're looking for with no need to remember the details of the naming conventions - of course, you ought to use the wildcard with a pinch of salt, not to be flooded with too many results matching your query.

dataset dataset=/SingleElectron*/*Run2016D*23Sep2016*/MINIAOD 

4. Is the dataset preset at a site which is accessible to cmsRun ? 4. Is the dataset preset at a site which is accessible to cmsRun?

Place your final query to das looking for the file you want at a site where the dataset has a presence. (note: the site where the dataset is present might change with time, thus the site chosen in detailed query which follows might need to be updated):

file dataset=/SingleElectron/Run2016D-23Sep2016-v1/MINIAOD site=T1_US_FNAL_Buffer
Does the query return a file ? If not, why ?

You can now run on one of the files and find out its basic properties, exploiting the fact that xrootd will serve the file you've chosen from the CMS site where it's available on disk to your cmsRun process:

 edmFileUtil   --eventsInLumis  -P   root://xrootd.unl.edu//store/data/Run2016D/SingleElectron/MINIAOD/23Sep2016-v1/70000/04E8F72C-AF89-E611-9D2F-FA163E1D7951.root 

 edmFileUtil   --eventsInLumis  -P   /store/data/Run2016D/SingleElectron/MINIAOD/23Sep2016-v1/70000/04E8F72C-AF89-E611-9D2F-FA163E1D7951.root (*update with the actual file you've found*) 

Exercise 2: compute the integrated luminosity collected by CMS in run 276775

The data collected by CMS are certified on a luminosity-section basis to determine which data is of good quality to be included in physics analyses. The data certification is carried out taking into account both the health in operation of the sub-detectors at and the scrutiny of the reconstructed physics objects by DPG and POG experts. The outcome of the certification process as more data gets collected and for each new version of the data processing is regularly updated by the DQM-DataCertification with reports at the PPD General Meeting and by means of json files, also available in this certification repository:

ls -ltrFh /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions16/13TeV

The json files from the certification are used to restrict the events to be included in analysis, typically setting the lumiMask in the crab configuration.

You can see the run and luminosity section structure by opening one of the files:

cat  /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions16/13TeV/Cert_271036-280385_13TeV_PromptReco_Collisions16_JSON.txt

Only successfully processed luminosity sections should be used to compute the integrated luminosity of your analysis: that's typically achieved by asking for the crab report, which is also in json format, and provides a summary file of the runs and luminosity sections processed by completed jobs. Here, for semplicity, we'll use directly the certification exercise for luminosity calculation, assuming all processing jobs for run 276775 have been successful.

The luminosity information can be accessed via the BRIL Work Suite , which needs a simple installation procedure:

*bash* : export PATH=$HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.0.3/bin:$PATH
*tcsh* : setenv PATH $HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.0.3/bin:$PATH

pip install --install-option="--prefix=$HOME/.local" brilws

(Do it again after installation)
*bash* : export PATH=$HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.0.3/bin:$PATH
*tcsh* : setenv PATH $HOME/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.0.3/bin:$PATH

The integrated luminosity as measured during the data taking ( Norm tag: onlineresult), delivered and recored, is provided for the luminosity sections specified in the json, limiting to the run 276775:

brilcalc lumi --help
brilcalc lumi -b 'STABLE BEAMS' -r 276775 -i /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions16/13TeV/Cert_271036-280385_13TeV_PromptReco_Collisions16_JSON.txt [--byls]

outcome for: Norm tag: onlineresult outcome for: Norm tag onlineresult

 
#Data tag : v1 , Norm tag: onlineresult
+-------------+-------------------+------+------+----------------+---------------+
| run:fill    | time              | nls  | ncms | delivered(/ub) | recorded(/ub) |
+-------------+-------------------+------+------+----------------+---------------+
| 276775:5093 | 07/12/16 21:26:20 | 1165 | 1165 | 222078295.493  | 210692911.427 |
+-------------+-------------------+------+------+----------------+---------------+
#Summary: 
+-------+------+------+------+-------------------+------------------+
| nfill | nrun | nls  | ncms | totdelivered(/ub) | totrecorded(/ub) |
+-------+------+------+------+-------------------+------------------+
| 1     | 1    | 1165 | 1165 | 222078295.493     | 210692911.427    |
+-------+------+------+------+-------------------+------------------+

You can verify that you get the same output of you constrict yourself a json file limited to run 256843 and process it without run restrictions:

cd CMSSW_8_0_21/src 
cmsenv
filterJSON.py --min=276775 --max=276775 /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions16/13TeV/Cert_271036-280385_13TeV_PromptReco_Collisions16_JSON.txt  | tee 276775.txt
cat 276775.txt
brilcalc lumi -b 'STABLE BEAMS'    -i    276775.txt

Exercise 3: for a given Monte Carlo AODSIM sample, find: the global tag, the digi-reco configuration, the production history/advancement and all the pile up scenarios available for it

The sample we start from is the neutrino gun overlaid with the pile up which matches the profile of instantaneous luminosity of the 2015 data taking:

dataset dataset=/SingleNeutrino/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/AODSIM 

1. what is the global tag used for its digi-reco ? 1. what is the global tag used for its digi-reco ?

2. Search the dataset in das and click the Configs: you'll find the .py files for all the steps of the sample production 2. Search the dataset in das and click the Configs: you'll find the .py files for all the steps of the sample production

Any Monte Carlo sample is associated to a prepID, a unique identifier of the production request which has produced it. prepID 's are strings like HCA-RunIIFall15DR76-00002, formed by the physics group which has placed the production request, the production campaign and an integer number.

prepID 's are used by the Monte Carlo Management Meeting, where production requests are notified and prioritized, and by the computing operation teams, and are the identifier used in its two key web based platforms: Monte Carlo Management (McM) and production Monitoring platform (pMp).

3. Find the prepID of the neutrino gun sample using das 3. Find the prepID of the neutrino gun sample using das

4. Find in McM the request which has produced of the neutrino gun sample 4. Find in McM the request which has produced of the neutrino gun sample

5. Get the McM test command sequence with the full digi-reco configuration 5. Get the McM test command sequence with the full digi-reco configuration

The production Monitoring platform (pMp) is a service available to CMS members to monitor the status of progress of single Monte Carlo production requests, full campaigns, and group of requests (defined by physics working group, processing configuration, priority etc). It can be accessed directly or linked from Monte Carlo Management (McM).

6. Get the production history using pMp 6. Get the production history using pMp

Most datasets in any campaign is produced with a single pile up scenario: the pile up scenario typical of the campaign. In 2015, because of the transition from 50 to 25 ns data taking, the main production campaign had 2 pile up scenarios for a large set of requests. Some datasets, however, get processed with multiple pile up configuration to support specific studies. Different version of the pile up in the DIGI-RECO step all process the same parent GEN-SIM: this is the simplest way of finding out if there's more than one pile up version for a physics dataset.

7. Find with das the GEN-SIM parent of the AODSIM dataset 7. Find with das the GEN-SIM parent of the AODSIM dataset

8. Find with das the children of the GEN-SIM parent 8. Find with das the children of the GEN-SIM parent

Exercise 4: Simulate your private sample

-- PhatSrimanobhas - 2016-11-11