ReleaseProcedure

Introduction

Goal

People involved:

Nightlies:

• dev & dev-val : ex. 13.X.0, 13.X.0-val
• bugfix & val: ex. 13.0.X, 13.0.x-val
• "final" bugfix: ex. 13.0.40
• pcache : ex. 13.0.30.Y

Before the Release is created

Day -14:

1. Announce Date of release to Developers HyperNews (hn-atlas-SWDevelopersAnnounce@cernNOSPAMPLEASE.ch)
2. Remind users of MIGn nightlies that tags have to be merged into the release branch by Day -7

Day -7:

1. Close and stop MIGn nightlies
2. Remind DBRelease coordinators about need for new release
3. Put default new DBRelease name into nightlies (External/AtlasDBRelease)

Day -3:

1. Build BDRelease
2. Use explicitly the DBRelease tar file in the FCT transforms to check the consistency of the software and DBRelease

Day -2:

1. Disable tag submission by everyone apart from release coordinator

Day -1:

1. Evaluate nightly as release candidate and make kit available for validation.
2. Timeline assumes unsuccessful: Release coordinator inserts essential patches into nightly

• Add comments here for the pre-release phase -- Main.ManuelGallas - 09 Oct 2007

Creation of the Release

Day 0: Creation of the Release

1. Evaluate nightly as release candidate and make kit available for validation. Timeline assumes successful [if unsuccessful, delay release by 1 Day]
2. Autotag, close and terminate release in tag collector
3. Install release on AFS and make unvalidated kit/release available making sure the right DBRelease version is included
4. Close release nightlies, and open pcache/point1 nightlies (announce deadline for tag approval in the pcache)

Day 1: Unvalidated Release

1. Deploy of the unvalidated release and KV tests in at least one outside center
2. At this point the KV will be using the DBRelease shipped with the unvalidated release
3. Announce the available un-validated release installed at AFS at CERN ??

Dealing with problems

1. Problems with the pacman kits (not with the software contains): kits have to be re-done using the same release number and KV re-started
2. Problems with DBRelease: the DBRelease is fixed with the same number and tests with the KV should be re-done, in addition FCT will pickup automatically the new version
3. ?? Should we freeze the tag approval in the next release up to Day2 in order to:
   1. Force developers to look once more to the release
   2. Have a backup solution just in case we can not cover problems with the pcache and point1 patching mechanism

Day 2: Validated Release

1. Declare release validated and announce it (where ?? )
2. Announce deadline for the first production/point1 cache
3. Replica of DBRelease in DDM

Dealing with problems

1. Problems with the software contains: use the AtlasProduction and AtlasPoint1 patch mechanism (will not cover fixes in the header files)
2. Problems with DBRelease (or needed update by Physics or Data preparation coordinators):
   • Create and test a new DBRelease with a new number
   • For GRID production replicate the new DBRelease in DDM (send mails if needed to accelerate the process)
   • Due to the size of the DBRelease (>150MB) it can not be included in the production caches
   • For individual users, pacman should allow to install as well an specific version of the DBRelease:

      
                pacman install release (or prod-cache) + KV + DBRelease
               

This is a test.

-- Main.AndresPacheco - 18 Oct 2007

• Add comments here for the release creation phase -- Main.ManuelGallas - 09 Oct 2007

After the Release is created

Day 6: First Production Cache

1. The production transforms were running into the nightlies previous to the Day0 in the AtlasOffline project. The test results in ATN, RTT and FCT can tell more about the readiness to build the release
2. The "pcache" nightlies have been started at Day0:
   • the AtlasProduction project is created in TC as a empty project ready to host patches for the release
   • NICOS is informed that the pcache nightlies should run using the AtlasProduction cache
   • the production cache kit is created (automatically by NICOS) and tested nightly (by KV at LanCaster center without AFS)
   • the FCT is running using cache created nightly or the pcache nightlies on AFS if the cache is not created in a nightly base
   • the FCT is using a specific version of the DBRelease (an unforeseen change on the DBRelease version implies at least one day of stable tests with the FCT and the DBRelease)
   • the RTT tests are using the pcache nightlies
3. Day 5 is the deadline for tag approval in the pcache nightlies (deadline for the pcache is always 7pm CERN time)
   • Data samples for the Physics Validation Sample A have to be defined
4. Day6 Build (manual or automatically) the production-cache
   • If the cache is created automatically, it is only needed to decide which cache from the previous nightlies is good in terms of tests and functionality
   • If the cache is created manually this implies ~1Day delay per platform
   • Test the unvalidated-cache at CERN with the FCT (if the cache is created manually, this implies 1 Day delay)
   • Test the unvalidated-cache on the GRID with the KV (if the cache is created manually, implies delay which is included in the previous step)
   • ProdSys should be aware about the process
   • Monitor the replication of the DBRelease in DDM
5. Day7 announce first production cache
   • Install the production cache at CERN
   • announce the cache in the lists: hn-atlas-physics-software-validation@cernNOSPAMPLEASE.ch, hn-atlas-job-transformations@cernNOSPAMPLEASE.ch, atlas-sw-prodcaches@cernNOSPAMPLEASE.ch.
   • Wait few hours in order to have the cache distributed in the GRID centers
   • Release the jobs defined for the SampleA

Iteration with more production caches

1. After the first production cache for a given release is announced we enter in a two-week software cycle for the production of new caches driven by the Physics Validation of the SampleA. This two-week cycle is established in order to obtain fast feedback from the GRID production but it can be shortened/enlarged under Physics Coordination requirement (announced in advance)
2. Day8 switch the "pcache" to the next production cache candidate
   • Announce the deadline for tag submission (~2 weeks, taking in account that typically the cache has to be done ~Tuesday)
   • First nightly should not get any new tag in order to check the nightly itself and be ready for an emergency cache in case of a major problem on the GRID
3. Day9: start collecting tags for the next cache (strict tag approval described here)
4. Follow the same process as with the first production cache

Integration of tags into the base release

1. Bug-fix releases: in this case the tag has to be added in the val and bugfix nightlies for the next bug-fix release
2. Final Release: the tags in the cache will go in a dedicated nightlies in which a selected set of point1 tags will be collected after solving possible merging problems.

Dealing with problems

1. If the kit for the production cache is created and tested by the FCT (and KV?) nightly we can always select the previous "good" cache (buffer of 7 days) if we need to stick to a given deadline
2. If we find a problem with the production cache the day of the deadline we can try to solve it for the next day

• Add comments here for the production cache phase -- Main.ManuelGallas - 09 Oct 2007

Day ?: First Point1 Cache

• Add comments here for the point1 cache phase -- Main.ManuelGallas - 09 Oct 2007

Links:

• Installing ATLAS software with pacman
• DBRelease: correspondence release-DBrelease number, internal tests
• Status of the DBRelease replica in DDM
• [[][KV]]
• Full Chain Tests for GRID production

General Comments

• Add general comments here -- Main.ManuelGallas - 09 Oct 2007

•  Intial Mail from David Quarrie
  
  Day -14 - Announce Date of release to Developers HyperNews 
  (hn-atlas-SWDevelopersAnnounce@cernSPAMNOT.ch). 
  Remind users of MIGn nightlies that that tags have to be merged into
   the release branch by Day -7
  
  Day -7 - Close and stop MIGn nightlies - Remind DBRelease coordinators 
  about need for new release - Put default new DBRelease name into nightlies
  
  Day -3 - Build BDRelease and register with DDM
  
  Day -2 - Disable tag submission by everyone apart from release coordinator
  
  Day -1 - Evaluate nightly as release candidate and make kit available for validation. 
  Timeline assumes unsuccessful. - Release coordinator inserts essential patches into nightly
  
  Day 0 - Evaluate nightly as release candidate and make kit available for validation. 
  Timeline assumes successful [if unsuccessful, delay release by 1 Day]. 
  - Autotag, close and terminate release in tag collector 
  - Install release on AFS and make unvalidated kit/release available
  - Close release nightlies, and open pcache/point1 nightlies
  
  Day 1 - Rebuild unvalidated kit in the case of build/install problems
  
  Day 6 - Build and announce first production cache.
  
  

  -- Main.ManuelGallas - 15 Oct 2007

-- ManuelGallas - 09 Oct 2007