TWiki> Main Web>TWikiUsers>CarlGwilliam>ExoticsApprovalDraft (2020-04-14, CarlGwilliam) EditAttachPDF

%CERTIFY%

ExoticsApprovalDraft

Introduction

This page describes the process of having your analysis results approved for showing at conferences & seminars, or for journal submission. Here is the official document describing the procedure. The ATLAS Publication Committee also has a PaperCheckList (the checklist for any public result is found here) and generally useful information at PubComHome. Note, below a "day" is a non-CERN holiday working day.

Before you start ... how long will this take?

Warning, important Below is some very useful information. Click the drop-down and read!
READ ME!! I am ready!
By using the analysis progress for papers using 2015+2016 data the following observations have been made. Please consider them when planning your analysis.
  1. EB Request: From request to formation the median (average) time was 1 (1.5) weeks. You should be ready for the subgroup approval process to start approximately 4 weeks after obtaining an EB (longer only if you have a very complex analysis) which is enough time for ~2 EB meetings.

  1. Subgroup Approval: After the customary 1 week circulation, from the subgroup approval meeting to being subgroup-approved, the median (average) time was 7.5 (7.5) weeks. Some analyses took a lot of time here because:
    • the analysis strategy was not sound - to be mitigated by the newly introduced EB request meeting (see below),
    • poor documentation quality - this can confuse reviewers which prevented an efficient approval (thus require at least one round of review with EB beforehand),
    • poor planning - for example not running things in parallel such as updating with latest recommendations.
  2. After subgroup approval to understand the SR and finalize all the systematics, the median (average) time is 6.5 (8.5) weeks. This is an exceedingly long period of time because
    • there was not a good understanding of the profile likelihood i.e. not enough investigation of pulls, constraints, basic checks of the shape and quality of systematic uncertainties. (See Profilelikelihood to avoid the same problems)
      • See TWikis (functional forms, signal morphing) on ExoticsWorkingGroup TWiki that are meant to help analyzers.
    • on a few occasions the SR data had features
    • new productions of NTuples took very long (and were not run in parallel to the approvals).
  3. Group Approval: After the customary 1 week circulation, from the group approval meeting to being group approved, the median (average) time was 4.5 (6) weeks. The physics and the paper draft are both approved by the group in this one step. Since paper drafts came very late (written by people not running the code usually) we require the paper draft to have been sent to the EB and a round of comments provided before Exotics circulation.
  4. First Circulation usually comes a few days after the group sign-off as the EB is reading the paper draft in parallel.
  5. First Circulation to Second Circulation to arXiv: The average time is 12 weeks (not counting those which have done a conf-conversion). This can be done much faster but often the team starts to vanish after the first circulation. If momentum is lost, the problem only gets worse.
  6. Referee comments and journal acceptance: Don't forget about these steps. Time varries according to the journal.

Selecting your topic and starting

If you have new ideas for physics analyses, please contact to group and subgroup conveners. A presentation at a subgroup meeting will be set to discuss your new ideas. In particular, physics impact, similarity to other analyses (for collaboration), and duplications with other existing analysis will be discussed. If a similar analysis does not exist (inside or outside of Exotics) then the conveners will start an entry in the Analysis Glance system.

Starting in Glance

This will create a Phase0 entry in Glance. Phase0 concludes with the group approval then either a Paper or CONF or PUB note glance entry is created for the approval steps outside of the group. The Physics Office has a Phase0 tutorial AnalysisPhase0Tutorial for details of technical information.
Action on Group Convener after ensuring the effort does not duplicate others existing in ATLAS.

When looking for more information... Hide
When looking for more information click edit, then click the black circle with the white "i" symbol. This gives some details about what that particular section is for. If that does not help, and this TWiki does not clarify, write a mail to the Exotics conveners with your subgroup conveners in cc immediately. Don't worry, just send the mail.

Instructions Hide

  • Fill in the form to start the analysis glance entry with the minimal information: short title, collision, dataset, leading group, subgroup, reference code (use the next available)
  • Set "Phase 0 start date", click "Proceed"
    • for PUB notes click "Skip to PUB note"
  • The INT1 repository will be created automatically (see Gitlab section below).

Gitlab

Two gitlab areas are needed per analysis (for now).

Tell me about gitlab repos Hide

  1. For the INT and PAPER/CONF notes, the physics office hosts the gitlab repository in https://gitlab.cern.ch/atlas-physics-office/EXOT. A repo is created using the ANA-ID (ANA-EXOT-20XX-XX) when the Group convener creates the entry Glance. Each new repo comes with one INT note area (INT1 - see 1st bullet below). For more INT or PAPER repos, the Group convener needs to click a button in Glance. Developer permissions are given to all in the analysis e-group. Two areas are created:
    • ANA-EXOT-20XX-XX/ANA-EXOT-20XX-XX-INT1 is for the internal note and created as soon as the glance entry is created. (Additional created upon request)
    • ANA-EXOT-20XX-XX/ANA-EXOT-20XX-XX-CONF/PAPER for the paper/conf/pub note and created. Created upon request.
  2. For now, analysis code is organized in this area https://gitlab.cern.ch/atlas-phys/exot/. PC and ASG are in discussions as to where/how to host an automatically created area. Please request a new repo using the ANA-ID (ANA-EXOT-20XX-XX) in the appropriate subgroup area from the Exotics Librarian (see ExoticsWorkingGroup).

Action on Analysis Coordinator to ask the group convener to create any repo needed for documentation.

e-groups for communication

The analysis e-group is the list which contains all the analysis members. It is in Glance (left-hand-side of entry page). It can be filled by hand to start, but usually, the names (and contributions) are imported from the CDS link of the support note. After a CDS entry exists, you should make all changes there and import to glance: it's impossible to propagate glance changes back to CDS.

The occupants of the egroups are listed in the central glance documentation. Please include the *-editors egroup when discussing with the EB so that all parties can follow.

Kick-off / Expression of Interest

Once it is clear the analysis is viable, it is polite to announce the project to the community and allow an easy point of entry for interested parties. This is done in the Expression of interest (EOI) meeting.

*NB: Often the number of people who express interest is much higher than the number that will actually participate in the analysis.* This is a feature of the job as we are all excited by, and interested in, many topics. However, one gains nothing by expressing interest and doing nothing. It is not advisable to be devoting a small fraction of time on many projects. There should be one project where you dedicate at least 50% of your time. Working on 2 projects is possible, 3 is getting tough. Anything beyond that cannot (usually) be done without being, at most, a bystander.

Do I need to wait for the EOI meeting to start working? Can they be delayed? Hide
For brand new ideas, there needs to be some work done before the kick-off to make sure the idea is viable. For the next iteration of a given analysis, the kick-off meeting could be delayed in order to have coordination with other overlapping activities such as a previous round of paper publication of the same analysis. In such a case, the team that shows their new idea in the subgroup meeting may start the analysis activities in advance to the kick-off meeting in discussion with the subgroup conveners.

Action on Subgroup Convener to announce and organize the meeting:

Instructions Hide

  • Announce an EOI meeting to the subgroup at least one week ahead of time (also to be included in the Exotics digest). Each team should upload 2 slides (max) to state their interest, their expertise/experience, and names the people who will participate and how much FTE each will contribute to the project. This should not take more than a few minutes per set of slides. (See SGC TWiki for example call)
  • Enter EOI meeting indico link into "Expression of interest (EOI) meeting data" section of Glance and fill in the fields in the following way:
    • Title - “Kick-off meeting in XXX”
    • Date - day of meeting
    • Label - blank
    • URL - meeting agenda
    • Comments
      • Institutes (Number): List
      • Total FTE committed (from estimates in slides)
      • New things being introduced or major themes mentioned
  • Fill in "Analysis definition after EOI meeting":
    • Main physics aim: Be brief O(5) sentences. Also, add any non-physics aims i.e. validate release 21
    • Dataset: years of data collection and release to be used

Analysis Contact & High-Level Overview

One of the primary roles of the group and subgroup conveners, as well as the analysis coordinator (analysis contact), is one of communication. Information needs to flow freely between these parties. The following sections define the contact point of the analysis team and give the high-level summary of the analysis.

Analysis Coordinator Selection (Analysis Contact)

To be invited by the Group Convener in discussion with Subgroup Convener who has a closer connection to the team members: the team members are welcome to suggest names to the Subgroup Convener and the latter may also decide to issue a call for nominations. The mandate can be found in ExoticsAnalysisContactMandate. Most analyses have two with a varying degree of experience (we prefer to have at least one young post-doc), that are committed to the project as their main task (>50% time commitment especially for the more junior colleagues who take on the role) and are not a coordinator for another team.

Action on Group Convener to edit glance.

Instructions Hide
  • Start Date: Date email invitation from conveners was replied to by coordinator (mandate is given here)
  • End Date: Leave blank

Setting Analysis Overview Information in Glance

The next two sections of the Glance entry are to give a bit more information about the analysis, but cannot be very detailed.
Action on Analysis Coordinator to enter information in discussion with the Subgroup Conveners.

Instructions Hide

Section: Analysis metadata

  • Physics Signature: The cuts here are not set in stone but should be indicative and include the major cuts (i.e. it is not necessary to state that 1 PV is required)
  • Model Tested: Brief description of the model with a sentence to the motivation. Link to arXiv if not widely used.
    • Some details useful to mention: production mode (qq, gg, 3rd generation, VBF, etc), single (t/s-channel) vs pair-production, spin, polarization (V's and tau's),
  • Previous analysis: Put in Glance entry at least most recent public result (see "internal" link on public results page)
  • Analysis methods: A high-level summary of the method.
    • CP techniques: Critical or new ones
    • List of major background processes and method of estimations: MC, data-driven (ABCD, functional form, etc)
    • Signature: resonance (peak reconstructed i.e. dijet vs peak not reconstructed i.e. VLQ) vs non-resonant search
    • Final variables
    • CRs, VRs, SRs
    • Model testing: cut-and-count vs 1 distribution vs multi-region, profile likelihood
    • Which analysis blinding strategy is proposed? Have a look at the ExoticsBlindingStrategy wiki. This should be discussed and agreed with the subgroup conveners before looking at data.
    • Other critical items

Section: Analysis coordinators’ target date

  • The target date is the date when the analysis aims for ATLAS circulation.. See "data-driven timelines" below for help with the approximations.

Section: Summary (left):

  • Statistical Tools: please add the statistical packages you use for setting limits. There are drop down menus for common packages.
  • MVA/ML Tools: if you use any MVA of ML techniques please add the info on which packages you use here
  • Triggers: Please add the list of triggers you use here both for the main analysis and any CRs etc. There are drop down menus to make it easier.

The Analysis Coordinator should organize a request for the necessary Monte Carlo samples if they do not exist already (see ExoticsMCRequestsHowTo).

Analysis Tracking in Glance

The "Auxiliary metadata" section includes technical information and the progress of the analysis as presented in subgroup meetings starting from any after the EOI meeting. Action on Subgroup Conveners for information on presentations in subgroup meetings.
Action on Analysis Coordinators and key analysis links.

Instructions for Subgroup Conveners Hide

  • Subgroup meetings: for each report, add an entry. These can also be added for round-tables. Continue to track there reports until the analysis is released.
    • Add the meeting and *save*
      • Title - Something meaningful/useful. Use "round-table" if appropriate
      • Date - day of meeting
      • Comments
        • Speaker
        • Mention in a short sentence the main items discussed. (more details to be given in the minutes of the meeting, on the indico page)
        • Note if timelines shifted and why.
    • Add the content and *save*
      • Meeting - Select what you added above
      • Type - Indico
      • Date - the day of the meeting
      • Label - blank
      • URL - meeting agenda (not link to contribution as can change)

Instructions for Analysis Coordinators Hide
  • TWiki - just put the link
  • Data derivation/MC - Add the derivation names here. If you use an EXOT derivation also check that your analysis is listed on the on DerivationframeworkExotics TWiki and you have one member of your team listed as a contact for the derivation you are using. Please contact the derivation contact (see ExoticsWorkingGroup).
  • Analysis ntuples (Rucio)

Preparation of Internal Note and CDS Entry

Action on Subgroup Convener Once the Analysis Coordinators have entered the requested information above, and the analysis is progressing, click "Proceed" so that Internal Note editors can be formalized.
It is in the ExoticsAnalysisContactMandate of the Exotics Group Analysis Coordinators to prepare the internal note. As the analysis develops and presents in the subgroup meetings, the Analysis Coordinators (Internal note editors) will provide the CDS link to group conveners and subgroup conveners.
The internal note should include some standardized sections (executive summary, object selection tables and event selection items) based on the .tex files automatically provided in the git repository created for your INT note template/EXOT/ directory. These can be seen in the physics office git and you can look at the resulting PDF from the latest version here.

Action on Analysis Coordinators/Subgroup Convener/Group Convener Fill in the Analysis Coordinators in the "Internal note editors appointment"
Action on Analysis Coordinators to prepare CDS entry with full analysis team (will be list who will have gitlab developer rights as stated above).
Action on Analysis Coordinators/Subgroup Convener/Group Convener to add CDS entry to glance and import team members.

As the analysis develops, the unblinding strategy should also be discussed and agreed upon between the Analysis Coordinators, Subgroup Convener and if necessary Group Conveners. Action on Analysis Coordinators and Subgroup Convener Discuss and agree on a blinding strategy (see ExoticsBlindingStrategy)

Editorial Board

The formal responsibilities of the Editorial Board (EB) are given here.

EB Request

The formation of the EB is a point when additional ATLAS resources (EB member's time) is requested. Before doing this, we would like to make sure a solid team is in place, the analysis strategy is sound, and all areas are covered. The baseline strategy should be in place in the supporting note with only (a short list of small) perturbation are left to check. This, along with the full analysis status, is to be presented by the team in a subgroup or an Exotics plenary meeting. At this point the information entered in glance should be checked to insure it is complete and up-to-date. If the team has a critical mass, and the analysis strategy sound and well advanced a request will be made.
Action on Analysis Coordinators to request an EB request presentation through Glance 5-7 weeks before the subgroup approval meeting (time explained in the following drop-down) but a bit longer for complex analyses.

Instructions Hide
Click "Edit" then "Proceed" in Editorial Board request meeting and formation data section. An automatic email will be sent to the Subgroup and Group Conveners.

Why do I need to do this? Hide
In the 2015-2016 (rel20.7) campaign several features were noted and are addressed by this additional, formal step.
  1. Many analysis efforts had a lot of interest at the kick-off and R&D stages, but when it came time to actually do the analysis, they left. The remaining people were then tasked with doing more than is possible in many instances. The '15-'16 papers lost scientific value if published too far into 2018 when a much larger dataset is available and needed to be scrutinized. Slow papers were therefore in an uncomfortable situation. To avoid this, the presentation and INT note, as noted below, should include a task list noting who covers what and what else they are committed to.
  2. An ATLAS author can be on no more than two EBs. Therefore, the lifetime of an EB should be kept to a minimum. Also, when agreeing to an EB, one knows what they are already committed to for the next few months. If the approval process does not start in a timely fashion, the EB might no longer be as available. In the 2015-2016 (rel20.7) campaign the median (average) time between EB formation and subgroup approval meeting was 15 (19) weeks (nearly 4 months). The median (average) time to form an EB is 1 (1.5) weeks.

The EB request presentation should cover:

  • Physics overview
    • Motivation
    • Signals - physics description and request status
    • Background model (i.e. MC samples used, ABCD plane)
    • Event selection & categorization (CRs, VRs, SRs)
    • List of systematics - include demonstration of "custom systematics" (i.e. not a recommendation from a CP group)
    • Model testing - final variable, discussion of binning, systematic correlation model and treatment (pruning, smoothing, etc.)
    • Expected limits with custom systematics included
    • Missing pieces
    • Will the analysis be blinded? Discuss the unblinding strategy: have a look at the ExoticsBlindingStrategy wiki - this should be discussed with the subgroup conveners before looking at data!
    • Would any control regions be good to unfold?
  • Technical overview
    • Analysis framework & derivation
    • Statistical and other packages
  • Team overview task list
    • List of all critical tasks, who is responsible for each, and what else they are working on outside of this analysis.
    • Please note which for analyzers this is their main PA effort.
    • It is not advisable to be devoting a small fraction of time on many projects. There should be one project where you dedicate at least 50% of your time. Working on 2 projects is possible, 3 is getting tough. Anything beyond that cannot (usually) be done without being, at most, a bystander.

The support note should follow the exotics template (see above). It should include sufficient detail for the EB to be able to thoroughly review the analysis in a timely manner. It should generally include the following section and items before the request is made, where minor pieces of one or two sections, but not all, may still be in progress. These requirements are also listed directly in the above template. Please note these are guidelines only and there is some discretion to be applied.

  • (Explicit) Executive Summary: context and motivation; target i.e. physics goal and general signal characteristics, overall analysis strategy including what is new, outline of background estimation method including general characteristics of CRs/VRs/SRs; milestones i.e. list of outstanding tasks with a rough timeline.
  • List of contributions: this should also be uploaded into CDS and synced to glance
  • Data and MC: Dataset used with blinding strategy, full list of background samples and details of the signal samples 

  • Object selection tables (following template) and detailed event selection: there may, of course, still be some minor open items, as long as they don’t significantly affect the analysis strategy, but these should be well defined and clearly indicated (e.g. coloured/bold) in the text in this section and in the list of outstanding tasks within the executive summary. Both should be updated as the analysis progresses.
  • Clear and well defined background modelling strategy: this should include CR/VR plots fo the main backgrounds with the full data (full run-2 analyses) or at least a representative majority of the data (analyses during data-taking); for minor backgrounds that do not need to be determined with high accuracy this may still be in progress but an outline of the planned method should be present.
  • Systematics: Several systematics may still be missing but the note should include a proposed plan listing the CP systematics you will need to consider in this analysis (+ timescale on which they will be available if not already) and an outline of how the systematics on the backgrounds are proposed to be determined. If not statistics-limited, the most dominant systematic(s) should be present.
  • Results: An overview of the final fit setup including the final discriminating variables(s), the (SR/CR) regions to be included in the fit and the floating normalisation parameters. Some rough first expected limits/discovery sensitivity plots are useful if you have them but not necessary. 
In this case the binning of the final variable(s) and the systematics smoothing/pruning should be indicated.

Action on Analysis Coordinators to contact the EB members as soon as they are assigned and notify them of the status of the analysis and arrange the first meeting.

Action on Group Conveners to enter EB request meeting information in Glance and fill out EB request form.

Instructions Hide

  • Section: Internal note editors appointment: Nominally these are the analysis coordinators but others can be appointed. To be decided case-by-case.
  • Section: Editorial Board request meeting data: A presentation in either an Exotics plenary or subgroup meeting to show the full status of the analysis. See above for details.
    • Title/Date/Label: see subgroup meeting reports
    • Comments: Impression of status, critical items to settle on
  • Add CDS entry of internal note to "Summary" (LHS section) and import analysis team by clicking "import from CDS". Delete full list of analysis team, then save, then import as import does OR of existing team and list in CDS.
  • At least 4, preferably 6, suggestions for EB members (including noting possible chairs)
    • A list of EB members can be found on EditorialBoardMembership. No one can be on more than 2 EBs so consider this when you are suggesting people.
    • The list should contain at least one young postdoc
    • Group and subgroup conveners are usually not appointed to EBs

EB Meetings

The EB meetings should be attended by the analysis team, and if possible the subgroup and group conveners.
Action on Analysis Coordinator to enter each EB meeting information in Glance with a short summary in discussion with the Editorial Board Chair.

Instructions Hide

Section: Editorial Board meeting data

  • Editorial Board meeting - to be added for each meeting with the EB
    • Title/Date/Label: see subgroup meeting reports
    • Minutes: Topics presented. The outcome of the discussion. Any action items. To be filled in by the Analysis Coordinator and confirmed by the EB Chair.
  • Editorial Board timeline: this is the timeline discussed between the analysis team and the editorial board and should be set at the first EB meeting.

Exotics Subgroup Preapproval

The blinding strategy of the analysis should be discussed before looking at any data with the subgroup conveners. The blinding strategy can be of various types, explained in the ExoticsBlindingStrategy wiki. If blinded, the decision to unblind will be made following the preapproval meeting. If unblinded as per previous agreement with the subgroup conveners, the preapproval meeting will serve to scrutinize the analysis with subgroup experts. A full status report is given to the subgroup after the detailed support note has been made available to the full Exotics group at least 5 days prior to the approval meeting but traditionally 7 days is recommended to allow for a proper review. The circulated supporting note must by this point detail the list of author contributions. The review is open to the full community but the Group and Subgroup conveners will jointly explicitly invite three people to take part to ensure a solid review; in addition, the subgroup conveners may appoint additional explicit technical reviewers (See “Expert Review” below). . At least one meeting with the EB, after they have reviewed the support note, is required before the subgroup approval meeting. The EB is invited to attend the meeting but attendance is not required.

For RECAST: Must show can create docker image for analysis selection code and stat analysis. See documentation ExoticsRECAST for further information.

The subgroup decides if the analysis is mature and complete enough to seek group approval. For major open points, follow-up presentations are scheduled as needed and minor open points are closed with CDS comments. Once the subgroup conveners agree there are no more open issues, the analysis can proceed to unblinding (if blinded). Once unblinded, the unblinded results should be presented in the subgroup for them to see. Once the supporting note is update, with all CDS comments settled, and the subgroup is happy with the unblinded results the analysis can proceed to group circulation.

When starting subgroup approval, it is good to request a gitlab area for the Paper/Conf note.

Checklist Summary Hide
Please see the CheckListINTNote TWiki.

Action on Analysis Coordinators to

  • prepare the INT note (see checklist) and post it on CDS, including a separate file for the list of author contributions created on CDS and which can then be uploaded to glance by the Subgroup Conveners. Send it to the Subgroup Conveners requesting an approval meeting,
  • request a Paper/Conf gitlab area and discuss potential editors with the conveners
Action on Subgroup Conveners to
  • announce the approval meeting to the full exotics group (not just subgroup) if the documentation is sufficient and analysis looks solid (see SGC TWiki for circulation mail template),
  • include the meeting information in Glance in "Pre-approval meeting data",
  • arrange for "Expert Review" in conjunction with Group Conveners (see below for details)
  • sign-off in Glance when all questions relating to unblinding (when it applies) and the analysis strategy have been answered in "PGC or SGC pre approval sign-off" form.

Expert Review Hide

In the pre-approval section, the Group and Subgroup Conveners invite 3 members of the exotics community with good knowledge on relevant topics for a given analysis to read and comment the notes during the subgroup circulation; in addition, they may invite additional technical reviewers. These can be:

  • experts on the analysis technique
  • experts on the model interpretation
  • experts on the CP objects (especially useful if using something new or different from the recommendations)
  • experts on the background models
  • anything you think is relevant
Consider asking reasonable people. Please make sure the person is familiar with what is important in a search - we do not need overly pedantic comments to satisfy academic curiosity, we need to get things correct within the precision that our final statements quote. From CP groups or PMG, subgroup or group conveners can help direct you to a knowledgeable person if they don't have the time themselves. Also consider the Exotics contacts to the CP groups and PMG

Action on Subgroup and Group Conveners: To enter a reviewer:

  • Role: Choose from the drop-down menu OR fill in something (if you fill it, it will go blank when you hit "save" but will appear again after you click "read mode" Ticket created to fix this feature)
  • Member: Put the person's name
  • Date: This is how they sign-off - they put the date in when they are satisfied. (You can also fill in the date if they sign-off in CDS, in a meeting, etc. but please inform them that you have done it).

When inviting the experts, you can point them to the ExoticsApprovalExpertSignoff TWiki and give them the link to the glance entry.

Draft preparation

Once the analysis is subgroup approved, writing a draft can start in earnest. Please make sure that the paper includes the requisite auxiliary/HepData material (such as efficiencies x acceptance, cut-flow table(s) and digitised plots) as detailed below
The EB must receive a draft of the paper and provide comments before Exotics circulation will be considered. PubCom has information to help paper editors. The paper draft must have commented on by the EB before circulation.
Action on Group Conveners to name two paper editors in Glance after consulting the Subgroup conveners and the Analysis Coordinators. The two should have a mix of seniority to balance experience to write a good paper and enthusiasm to get it done on time (and get educated how to do it!). See the gitlab information above.
Action on Analysis Coordinators to send a CDS link of the paper draft to the EB and conveners once the structure and main points are in place.

Exotics Group Approval

A full status report is given to the group after the detailed support note has been made available to the full Exotics group at least 5 days prior to the approval meeting but traditionally 7 days is recommended to allow for a proper review. The review is again open to the full community but the Group conveners will explicitly ask the Expert reviewers from the pre-approval stage to attend the approval meeting and check their comments from pre-approval have been addressed (see "Expert Review" above). The support note must contain the full, unblinded results. The paper draft must have commented on by the EB before circulation. An Exotics group approval talk is scheduled (usually: 30 min talk + 45 min discussion). The approval is similar to a subgroup approval - outright, follow-ups in CDS, or a closure meeting.

For RECAST: Must show have captured commands before circulation and demonstrated signal workflow before approval is granted. See documentation ExoticsRECAST for further information.

It is mandatory to meet with the EB in between the subgroup and group approval meetings and that they have read and commented the support note again.

Checklist Summary Hide
For Exotic group circulation/approval:
  • if the analysis was blinded prior to subgroup pre-approval, the unblinded material must be shown at a subgroup meeting
  • all open point from pre-approval meeting and from the unblinded results have been addressed
  • the appointed expert reviewers must have signed off
  • all CDS comments from the pre-approval stage have been replied
  • The Debug stream must have been checked
  • at least one iteration of paper draft with EB team on CDS
  • for RECAST - capture commands before circulation (and full workflow for signal before sign-off)
  • group conveners agree to start in discussion with subgroup conveners
  • no EB approval is required (although EB will be informed of the Exotic approval)

Action on Analysis Coordinators to prepare the INT note and pape draft (see checklist). Send INT to the Group Conveners requesting an approval meeting.

Action on Group Conveners to

  • announce the approval meeting if the documentation is sufficient and analysis looks solid,
  • include the meeting information in Glance in "Approval meeting data",
  • invite Expert Reviewers from pre-apprvoal stage to attend the meeting.
  • read the paper draft,
  • sign-off in Glance the physics comments are all addressed and the paper has all the necessary physics content.

Editorial Board Sign Off

The EB reads and reviews the draft, posting comments in CDS. Respond to their comments and modify the draft until the EB and signs-off on the support note and paper/CONF note draft. This may take a few days or several weeks, depending on the complexity of the analysis. It is worth your time to make sure the paper/CONF note draft is of high quality at this point, to ease the next steps. When the EB and conveners sign off, the EB sends the draft to the Publication Committee for ATLAS-wide review and approval.

For papers, you may request that the plots be given preliminary approval during the Paper Closure Meeting (see next section). For this you must inform the Publication Committee when the draft is sent to them, i.e. before the commenting period.

For preliminary approval of plots during the open discussion, since Jan 2012 it is also required that the plots be documented in a CONF note (the paper draft can be used as a CONF note) in order to avoid having preliminary plots without documentation for a long period of time. This is explained further below in the section called "CONF Conversions" For preliminary approval of plots during the final reading there is no need to document the plots in a CONF note since presumably at this stage the paper will be submitted shortly.

Auxiliary Material and HEPData

The auxiliary material of a paper usually contains additional information to be made public that is useful for readers but not necessarily needed in the main body. It is also where additional details are included to facilitate reinterpretation or recasting. The JDMAuxMaterial TWiki gives a summary of useful information to include. One should also consider what should be put into HEPData. Information can be found on the ExoticsHEPData TWiki. All of this information should be discussed with the group and the EB. The auxiliary material should be relatively complete for the first draft but can be amended in the second draft. The entries in HEPData should be completed between the start of the second circulation and the completion of the final sign-offs. Also to this end, the RECAST framework is being used (see steps in above) to capture analyses as they are run within ATLAS.

NEW The aux data should include a detailed cut-flow for weighted (taking into account lumi, cross-section and BR) signal MC events for the minimum number of benchmark signals needed to cover each signal region. The cut flow(s) should start from the total number of weighted events before any selection (except any generator filters). The next line should report the number of events after the combined derivation (and perhaps preselection if similar) cuts. Each subsequent row should detail a single analysis cut. This should also be uploaded to HepData. An example from SUSY can be found here

ATLAS-wide review:

The remainder of the process is described on the PubCom TWiki. See here for the key links.

A brief summary is here Hide
Summary of steps for approval after Exotics Conveners and EB sign off on 1st Paper Draft. See here for the full set of information and links. A summary flow chart of this process can be found in Figure 1 of ATL-COM-GEN-2015-001.
  • Writing and Reviewing the Paper Draft tells what the EB should review and gives links for useful information, text snippets, etc.
  • Draft 1 Circulation in ATLAS is 10-14 days (depending on paper length) and ~3 days after the comments are due, the Paper Approval Meeting (PAM) is held (see A.2.2.d for details on PAM, further instructions are sent via Glance/PubCom)
    • A conf conversion must be requested before this circulation
    • Exceptional author requests are done during this time
  • The EB signs off on the changes to the draft once all of the comments from the 1st circulation are included in the paper
  • Language Editors (LE) (See A2.2.g) are assigned. Instead of a tarball, one can give the LE access to GitLab directly.
  • The LE sign's off once all changes are made / misunderstandings discussed. This step is usually complete within two weeks of the LE being contacted.
  • Draft 2 Circulation in ATLAS is 7 days
    • Public plot approval (showing the plots before the paper is on the arXiv) must be requested before this circulation
  • A Paper Closure Meeting (PCM) is held only if any of the points in A.2.2.h apply.
  • After the second circulation, the EB signs off on the third draft which has all the comments addressed
  • Two reviewers are assigned and review the draft in parallel
    • PubComm chair or delegate read for sign-off in Glance (typically following this guide)
    • Spokesperson reviewer considering sign-off in Glance (typically following this guide)
  • The Spokesperson checks the final draft, and signs off in Glance (rubber stamp approval)
  • Submission begins (see guidelines for paper submission and follow the PO-GitLab papers procedure)

Conf Conversions

A conf conversion is a release of a paper in between 1st and 2nd circulation. This is usually done to show the result at a conference. In order to request one
  • talk to the conveners early on to make sure PC is on board
  • tell PubCom before the 1st circulation to ATLAS that a conf conversion will be requested (it must be included in their announcement mail)
  • after the PAM (approval meeting after 1st circulation), the major of comments should be implemented and the conf released (after further sign-offs). The 2nd circulation for the paper should start straight away - there is an explicit requirement not to let analysis sit in between draft 1 and draft 2 circulation when a conf conversion is released. If allowed to slow down after a conf conversion, the time taken to complete the submission grows quickly.

Post submission activities

  • The support note needs to be turned into an INT note (and before that, updated in CDS with all latest results, plots, and studies, if this is not the case already, in particular, if additional studies or changes were made in the analysis after the first distribution to ATLAS). If the EdBoard is satisfied with the final version of the support note, the authors should request the INT approval in CDS (which will not trigger any additional review, don't worry).

  • The analysis code and the plots in ROOT format (or the scripts required to produce the plots) should be stored in SVN/Git with proper documentation. Please contact the Exotics Librarian for more information.

  • The most important results/plots should be stored in HEPDATA so that people do not have to get a ruler to read numbers off our plots. Instructions about how to submit results to HEPDATA are available here. This can be done as soon as the paper is on arXiv. Please let us know if you run into any trouble with this.

Major updates:
-- GabrielFacini - March-2018 -- GabrielFacini - 26-Jun-2017

%RESPONSIBLE% GabrielFacini MarieHeleneGenest
%REVIEW% Never reviewed

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2020-04-14 - CarlGwilliam
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback