• Final answers in green
• In-progress answers in blue

MET+X Conveners New Questions

Question: - BDT vs PTmiss: It’s not clear to us which binning in the BDT you’re using when you perform your fit. If not done already, can you show Fig. 29 when you use a binning in the BDT such that your background yields per bin are identical to the four bins of PTmiss from Fig. 25?

For MET: I used 14 bins when perform the fitting [0, 10., 14., 20., 28., 41. , 58. , 83. , 119. , 170., 242. , 345. , 492. , 701., 1000.] For BDT: I used 20 bins from -1 to +1

Question: - Table 19: Can uncertainties in lepton ID and reco really go up to 15%? What kind of muons/electrons are those? Is this the effect per lepton or of all 4 leptons combined? Please clarify. In case it’s the combined effect, please clarify if you’re using uncorrelated uncertainties for electrons and muons and how you can get a combined 1% uncertainty because even the smallest individual lepton uncertainties are bigger than 0.25%. So please, just clarify overall this huge 1-15% range

The uncertainty of lepton Identification and reconstruction shown in table 19 for all 4 leptons together. The uncertainty for muons and electrons are uncorrected. The lower value of uncertainty related to muons and the higher values for electrons. Efficiency uncertainty comes from the scale factors and the TnP method, the main reason for electron high uncertainty is that we don’t have low mass resonances to measure electron efficiencies while we do have them for muons.

Question: - Table 20: Why is the ttW PDF uncertainty so huge (30% and more)? Please check

The uncertainty of TTV and VVV was taken from HH analysis [*] that have been approved by HZZ4l group.... I can check this again. * https://cms.cern.ch/iCMS/analysisadmin/cadilines?line=HIG-20-004&tp=an&id=2315&ancode=HIG-20-004

Question: - Table 22: Why is VVV the only background not growing over the years?

Actually there was a type in 2018 yield, the correct value is 0.163 I'm showing here the yield for each background used in VVV including the statistical uncertainty

Question: - Fig 19: Why does the 2e2mu training give you the best ROC curves compared to 4mu or 4e? Is the background cleaner? How would a ROC curve in 4mu look like if you apply the 2e2mu training?

If you look at table 18 you can find the area under the ROC curves for different channel and per year which shows comparable values, moreover the following plots shows the ROC curves for the three channels together in 2016 as example for Low A mass region (left) and High A man region (right) which are almost the same.

Question: - Section 10.1: I don’t understand what you mean by BDT discriminator shape uncertainty. Just as you propagate shifted inputs to your PTmiss, you would feed it to your BDT discriminator and get one set of up/down BDT templates for each uncertainty that affects the BDT shape. It’s not a source of experimental uncertainty itself. I guess that’s what you mean, but please clarify.

Yes that what we mean, we propagate the shifted inputs of MET and feed to the BDT discriminator and get the a set of up/down BDT templates.

Question: - In general: I see you’re trying to figure out why you have lopsided uncertainties. That’s good, since it needs to be understood. You can also just symmetrize them by mirroring the larger of the up/down variations to the other side.

It has been fixed and the new plots are in the AN.

Followup: MET+X Conveners Questions

###General:

Question: - Thank you for providing many additional plots and corroborating information. Please make sure to propagate clarifications and plotting changes (e.g. stacking order of backgrounds) also to the AN.

Done

Question: - For the MET plots: Please start showing them with the y axis event yields divided by the bin width rather than absolute (i.e. events/GeV) That will make it much easier for the viewer to understand the distribution shape.

###HEM:

Question: - You compare plots with and without a HEM veto, but I am not sure what veto definition is used. What HEM veto is applied here?

The HEM veto is excluding the events with Jet pT > 30 GeV in the region of eta -3.0 to -1.3 and phi -1.57 to -0.87 .

- Question: -You show ratio plots of the MC before and after the HEM veto. Given that the HEM effect is only present in data, but not simulation, I am not sure what I learn from these plots. If anything, please show the ratio of data for the CR.

Yes that is true, we should show the ratio for data events in the CR before and after applying the HEM veto as show in the following plots for PFMET (left) and MET phi (right) distributions. The overall effect around 2% in some bins.

Question: - Please make the met phi plots for a few different slices of PFMET. Your current plots are inclusive in PFMET, which means that low-MET dominates, and you might miss a localized high-MET contribution that is drowned out in phi.

That is true, if I plot the PFMET phi in different PFMET slices I see difference in phi distribution dominated in low met values. The plots from let to right shows the PFMET if PFMET<=25, 25<PFMET<=40, 40<PFMET<=200, PFMET>200 GeV.

###Prefiring:

Question: - The plots you show here indicate prefiring effects of up to 5% on the signal sample you tested here, growing with MET in 2016. This seems like a significant effect, but your reply says there is "no noticeable effect", which I do not understand. What exactly do you mean? Regarding the uncertainty here: How is it concluded that no uncertainty is necessary? You seem to only show the central value of the reweighting, so one cannot make any judgment about uncertainty based on the plot here. Suitable ways of estimating the uncertainty would be to use the up/down variations of the prefiring weights, and to vary between the way the prefiring weights are applied (compare the scheme where the full jet pt is used to the scheme where only the electromagnetic part is used).

In the previous plot the prefiring effects go up to 5% in few bins not overall effect. In the following plots I show the PFMET distribution with prefiring weight along with the up/down variations for total bkgs (left) and one signal sample (right) in the SR. We can see the prefiring effect goes to 1 % in case of bkg and goes to 8 % in one bin in case of signal sample while all the other bins the effect around 1%. Anyway I already stored the up/down variations of the prefiring weight that can be used as a shape uncertainty. I used the recipe in [1] which include the full jet pT but I don't have the information while using the EM jet pT.

[1]https://twiki.cern.ch/twiki/bin/viewauth/CMS/L1ECALPrefiringWeightRecipe#Recipe_details_10_2_X_X_10_or_9

### Delta phi(jet, met):

Question: - Please clarify what jet is used here. Is this the leading jet or all jets in the event or something else? You should see the most sensitivity for rejecting fake met by using min(dphi(j1, met), dphi(j2, met), ...) for jets above 30 gev or so. If that is not what is done here, please show a plot of this minimum dphi. - Please show the plot in the mll side band CR and include data.

the plot was shown for all jets in the event with pT> 30 GeV. Here is the plots using min(dphi(j1, met), dphi(j2, met), ...) for jets with pT> 30 GeV under the Higgs peak for signal and bkg events (left). The same plot is shown in the CR (side band of M4l) in right.

###Z+X:

Question: - Thank you for adding the m4l shape plots, which illustrate the fake rate estimation procedure. The target of my question was different, though: I would like to know how you arrive at the shape of the Z+X background for your limit setting in the SR. How is this done differentially, and how is the BDT evaluated for this contribution? This should be documented and shown in the Z+X section

We build the 2P2F & 3P1F CRs after applying the 2 additional cuts to define the SR (m4l cut & Number of leptons) using the BDT discrimination function, we feed the function with the input variables in the 2P2F and in the 3p1F CR then we apply equation 7 to get the final estimation. We have BDT discrimination function for each channel (4mu, 4e, 2e2mu) and for each year of data taking which allow us to get the Z+X bkg per channel/ year.

Question: - Related to the above: The BDT plots in Fig 22 show significant Z+X contribution at high BDT score in 2017, but not the other years. Why is that?

The y-axis scale in 2017 HM region (right plot) is different from 2016 and 2018.

Question: - In general: You say that Z+X is included in the BDT training only via the MC samples. However, we already know that MC cannot be trusted to model this contribution well, hence the need for a data-driven estimate. How are we ensuring that the BDT response to this process is modeled accurately?

I used the Z+X estimated from data as in the BDT training in 2016 as example and I didn't get change in the BDT training. I'm showing the ROC curves for BDT training in 4mu (left), 4e(middle), 2e2mu (right) with/without Z+X in the training.

Question: - Your reply seems to say that there are no shape uncertainties on Z+X. Is that true? Given that you already have to bin the fake rate in MET, it seems hard to believe that there would be no shape to the resulting uncertainty of this method.

###MVA:

Question: - Fig 16: Please clarify which of these signals is shown as the histogram labelled "signal" in this figure.

The plots show the low mass (LM) signal samples (all signal samples with mass of heavy pesudoscalar A of 200, 300, 400 whatever is the value of mass of light pseudoscalar or tan beta or sin theta ) and all bkg process for 4mu channel in 2016 as example.

Question: - Validation: thank you for providing the inputs plots in the side band. Inclusively the agreement seems quite good. Two comments: - In the highest MET bins in both 2017 and 2018 you observe sizable excesses in data (hard to read exactly from the plot, but it seem like ~9 observed in either case with ~2 expected in 2017, and maybe 4 or 5 expected in 2018). Please pick these events in data, and check whether there is anything anomalous about them.

-In 2018: There are 6 data events in the last bin of MET distribution in the CR listed in the following file

* List_data_Events_CR_lastbin_2018.docx: List_data_Events_CR_lastbin_2018.docx

-In 2017: There are 7 data events in the last bin of MET distribution in the CR listed in the following file

* List_data_Events_CR_lastbin_2017.docx: List_data_Events_CR_lastbin_2017.docx

Question: - Please also provide the other distribution plots with a minimum requirement of MET>100 or so. The current plots are inclusive in MET, which means that low met dominates and high-met mismodeling might be obscured.

The plots with MET>50 GeV for each year of data taking:

-2018

-2017

-2016

###Uncertainties & Yields:

Question: - It seems that you are currently using normalization uncertainties for everything except MET-related uncs (object scales etc). Since you perform a shape-based analysis to distinguish the signal and background, this might not be appropriate for everything. This is particularly clear for the theory uncertainties, where the total normalization uncs used in SM HZZ will be heavily biased towards low-pt, while the true uncertainties at higher pt might be significantly larger. Please evaluate the shapes of these uncertainties in the phase space that is relevant to your analysis.

We are reconstructing the 4l in the same way as the HZZ4l analysis where the uncertainties well evaluated we don't need to evaluate again.

MET+X Conveners Questions

#### General

Question: - You do not discuss the impact of the HEM effect on your analysis of the 2018 data set. Please document how you study / estimate / control this effect.

The following plots show the MET and MET phi distributions before (left) and after (right) applying the HEM veto in CR (side band of the m4l). To check how much the effect, the ratio of before (red) and after (black) applying HEM veto for 2018 MC samples is shown. We find no noticeable change in the above-mentioned distributions before and after applying the veto. We have also checked them for SR and no noticeable deviations have been observed. Since we find no noticeable change in the above-mentioned distributions before and after applying the HEM veto, we didn't add any uncertainties related to the HEM issue.

• MET distribution in CR before (left) and after (right) applying HEM veto:

* MET phi distribution in CR before (left) and after (right) applying HEM veto:

* ratio distribution in CR before (red) and after (black) applying HEM veto:

* ratio distribution in SR before (red) and after (black) applying HEM veto:

#### Introduction
Question: - L181+: You refer to EFT models being used for interpretation. I assume this is outdated?

Yes it is out of date ... We removed.

#### Signals

Question: - There is currently no mention of the Dark Higgs model. What is the plan/status here?

Still in progress

#### Datasets

Question: - L232+: Please clarify a bit more the generation settings for the signal. Is it LO or NLO? Are there additional jets in the matrix element / is jet multiplicity merging used?

The signal samples are NLO and No jet multiplicity have been used. The information has been added to the note.

Question: - L268: You say that all samples use NNPDF 3.0 NLO. Is that also true for 2017/2018? There, the CMS default is 3.1 NNLO (in line with tune CP5).

Indeed it is NNPDF3.1... fixed in the note.

Question: - Tab 8: IIUC, this table suggests that gg and qq initial states are produced separately. Based on the XS, it seems that qq is suppressed by 2-3 order of magnitude inclusively for tan beta =1. Does qq matter here or only for higher tan beta?

That is true, it doesn't matter the qq samples in the case of tan beta = 1 but it contribute in the case of the samples with high tan beta values.

#### Objects

Question: - sec 4.4.41: please compare your filters to the latest JME twiki [1]. It seems there is at least one filter missing from your list (Flag_eeBadScFilter) [1]https://mmm.cern.ch/owa/redir.aspx?C=IuNGg-mha1tMX52vcYoQ2Rpy52250ILG8V88myVlrJ5DkypayPLYCA..&URL=https%3a%2f%2ftwiki.cern.ch%2ftwiki%2fbin%2fviewauth%2fCMS%2fMissingETOptionalFiltersRun2

The filter is already included in the analysis for data. Fixed in the note.

Question: - sec 4.5: How big is the effect of the prefiring weight and how are the related ucnertainties treated?

The following plots show the MET distribution with (red) and without (black) the prefireing weight on signal sample in 2016 (left) and 2017 (right) signal sample (ratio would be the best). We find no noticeable change in the above-mentioned distributions before and after applying the preferring weights. We have also checked them for MC backgrounds and no large deviations have been observed. Since the effect is small, we didn't add uncertainties related to the prefireing weights.

#### Selection

Question: - L445+: You say that events must pass a four-lepton selection and refer to [26]. It's not clear to me whether you mean that the selection described in the following (e.g. 5.3) is this selection or whether there is something else that is not described here?

All the 4l event selection is described in in the note in section 5 "Event Selection" . I just mentioned the refenece of the HZZ4l note.

Question: -Your selection does not mention any requirement on separation of jets and the met vector in the transverse plane. Do you apply such a requirement? This is usually a good way of rejecting fake met contanimation.

We didn't consider any requirement on separation of jets and the met vector, since we think that the fake MET have been cut with the applied MET filter. But for completeness we checked the delta phi (met, jet) for 2016 signal and different bkg under the Higgs peak but we found that the distribution is flat as shown in the following plot.

#### CUt-based selection

Question: You say that you have attempted optimizing cuts on various variables (whcih?), but only the cuts on m(4l) and ptmiss were useful. What quantitative improvement did the third-best cut give you?

On top of the cut on (m4l) we tried putting cuts on different variables such as (Delta phi (4l, MET) - Transverse mass (4l, MET) - DeltaR (Z1, Z2) - b tagged jet multiplicity - |MET- pT(4l)/ pT(4l)| ) and check the effect to of the cuts on the analysis sensitivity measured from the upper limit on the signal strength, moreover we check the effect of the cut on the different signal sample. But instead of doing rectangular cuts on those variable we injected the variables to the BDT.

#### BG esitmation

Question: - L551+: In multiple places, you mention "signal" and "background". What exactly does this refer to? Please clarify.

L 551 mention gg ZZ background modeling ... The sentence has been rephrased since we mean the Higgs sample and ggZZ sample .

Question: : - Eq 7: I do not quite understand the logic of having 3P1F and 2P2F regions in the same estimation method. You have previously explained, and write in the text, that the goal here is to include different types of prodction processes. However, I am fundamentally confused as to why that is necessary. Naively, I would have thought that the 3P1F region already includes a contibution that could be written as f/(1-f)*N2P2F. In that case, the formula in eq 7 is just double counting the 2P2F contribution tothe signal region. What am I missing?

The 2P2F control region contain the contribution coming from processes have 2 fake objects like Z+jets, tt while the 3P1F control region contain the contribution coming from processes have 1 fake obiect like WZ background, so need to include the contribution from both regions. As you mentioned that the 3P1F region already includes a contibution from 2P2F region that we subtract to not double count this contribution.

Question: -Fake rate: What fake rate value is used for met>last bin?

For the MET value > last bin we use the fake rate of the last bin.

Question: - Z+X BG in general: You quote the resulting expected yields of this process, but you do not show any shape information. How is the differential shape of this process estimated? Same question for uncertainty.

The estimation of Z+X background depend on the method described in the HZZ4l analysis, we are not using shapes uncertainties we use normalization. In the following plots we show an example of 2p2f and 3p1f CR in 2016.

Question: -Fig 15: Please rearrange the y-ordering of the ackgrounds so that small BGs (e.g. Z+x) are easier to see

#### Observables

Question: - General: You show signals with constant mA,ma in your plots and vary tan beta and sin theta. As you explained earlier, tan beta and sin theta do not alter the kinetmaics. Please show variations of the masses instead.

The signal with different MA values are shown in the previous question.

Question: - Fig 13: Do these plots already include the cut on m4l? If not, what do they look like after?

The plots on Figure 13 don't include the cut on m4l. Here are the MZ1 (left) and MZ2 (right) with the m4l cut.

#### MVA

Question: - L734: is Z+X included?

Z+X background estimated from data not included in the MVA study but the MC background samples contributing to this background have been used in the study.

Question: - Fig 16: What sample is this? LM or HM?

I divided the MonoHiggs signal samples to two category depending on the mass of the heavy pesudoscalar A.

Low Mass region (LM) contain all the signal samples with mass of heavy pesudoscalar A of 200, 300, 400 whatever is the value of mass of light pseudoscalar or tan beta or sin theta.

High Mass region (HM): contain all the signal samples with mass of heavy pesudoscalar A of 500, 600, 700, 800 whatever is the value of mass of light pseudoscalar or tan beta or sin theta.

Question: - How does the shape of MZ1, MZ2 compare between the mono-H signals and the SM H samples?

The distribution of mZ1 and mZ2 are shown in Figure 13 in the AN. The shape of MZ1, MZ2 for the mono-H signals and the SM H samples should have the same distribution, where we reconstruct Z1 as the Z with mass closest to 90 GeV "on-shell" and the second Z as Z2 "off shell"

Question: - Generally: If we use the MVA based method, we should take care to build confidence that the MVA does not introduce significant data/mc differences that might affect the analysis. As far as I can tell, you are currently doing this by a) showing some of the data/mc comparisons of the input distributions and b) showing a part of the MVA output distribution in the SR. In both cases, we can only take limited conclusions, given that the phase space in neither case really represents your most sensitive signal bins. Additionally, the MVA relies on the correlations between variables, whcih is not tested when looking at individual distributions. I would therefore like to encourage you to think about a more systematic approach to building confidence in the MVA model. Could we for example look at a validation region that is relatively similar to the SR? Off the top off my head, I would think that a mass sideband in m4l would at least allow to check the modeling in ZZ events, if not in SM H events directly, but that should already be instructive given that the sources of fake MET should be similar between the two.

The following plots shows the MVA input variables distributions in CR (side band of m4l) in different years if data taken:

1) 2016:

2) 2017:

3) 2018:

#### Uncertainties

Question: Tab 20: Are the theory uncertainties shape or just normalization? If shape, please plot the variations. If not, please explain why shape not

All the theory uncertainties are normalization as provided by LHCHXSWG in https://twiki.cern.ch/twiki/bin/view/LHCPhysics/CERNHLHE2019 and HZZ4l note.

needed. Question: - Please show all variations for the ptmiss distirbution as well as for the BDT dist.

Al the variations for the ptmiss distribution are shown in Figure 23.

Question: - How is the uncertainty associated to the qqZZ electroweak correction evaluated? In EXO-19-003 (mono-Z(ll)), which also has ZZ as leading background, there is a significant amount uncertainty stemming from higher order corrections on ZZ (including also a rough estimate of mixed ewk-qcd corrections). There, the situation is different in the sense that EXO-19-003 directly probes the Z boson pt spectrum, while you are likely more sensitive to the pt of the di-Z system. Please clarify what is done and how it relates to what is done in EXO-19-003.

This analysis relay on HZZ4l analysis where the qqZZ background has been deeply studied since it is the main background of the Higgs analysis. We kept the same uncertainty values as SM HZZ4l analysis.

Question: - Fig 24: Same question that was asked in a previous presentation: The up and down variations for a number of sources go in the same directoin (e.g. bottom left), which seems pathological. Is there a physical reason for this behavior?

It has been fixed and the new plots are in the AN.

#### Statistiscal analysis

Question: - L804: You say you fit on both the BDT and ptmiss distribution. Judging from the rest of the text, I assume that you mean that these are two separate approached (you do not fit both at the same time). Suggest to write this more clearly

I mean that we used separate approaches, one approach use the MET distributions as input for the shape analysis and the second approach uses the BDT distributions as input for the shape analysis. rephrased in the note.

Question: - Fig 28: It seems that at high mA values, the BDT is worse than the ptmiss analysis. That seems pathological. What is the reason?

In the high MA region the loss in sensetivty is small as shown in the following values, this loss could be introduced since the statistics of signal sample is low compared to low MA region but since the analysis is not sensitive at high MA region this loss will not affect the final results.

MA = 600

Expected 2.5% Expected 16.0% Expected 50.0% Expected 84.0% Expected 97.5%

MET: r < 1.4032 r < 2.0580 r < 3.2656 r < 5.3223 r < 8.2383

BDT: r < 1.5917 r < 2.2606 r < 3.4531 r < 5.4077 r < 8.1530

MA = 800

Expected 2.5% Expected 16.0% Expected 50.0% Expected 84.0% Expected 97.5%

MET: r < 2.9758 r < 4.5203 r < 7.4688 r < 12.5298 r < 19.9273

BDT: r < 4.1052 r < 5.8306 r < 8.9062 r < 13.9828 r < 21.2491

Question: - Fig 29: Would be good to explain the y axis in the caption. Gain > 1 means BDT wins?

Yes If the gain is greater than 1 means that BDT wins. The definition of the Gain has been added to the caption.