Preserving DELPHI data access
Overview
There are four main areas to be respected when it comes to data preservation.
Bit preservation
DELPHI data is stored on EOS at CERN, a disk based mass storage system. Two additional copies are stored on tape on CTA.
Software preservation
The general strategy for preserving DELPHI data access is to keep the software stack alive as long as possible, as only this option really ensures the full usability of the data. This does not exclude other options, in particular the use of containers and virtualisations, as these allow to rerun using original executables without recompiling, which can be very useful for validation.
The main areas in this category are
- Data analysis framework
- Simulation
- Reconstruction
- Event display
Documentation
The bulk of DELPHI related documentations is availabe in CDS at CERN. Only a small subset requires authentication, the bulk of it is open.
Analysis preservation
Preservation of analyses is a weak point as this was not on the radar of the collaboration while it was still running. Some bits have been preserved though. If DELPHI members still have their code and ideally their results (e.g. ntuples), please get in touch so that this can be preserved.
Software preservation status in 2023
Sources are in CERN gitlab, and binaries are on CVMFS. The run time environment has no CERN specific dependencies. It can be run on any desktop running a supported Linux distribution, which has CVMFS and EOS installed. Access to sources and data requires Kerberos authentication though. Both 32bit and 64bit binaries are provided, for those distributions where this is still possible. The default is 64bit now.
Data analysis frameworks
- The Fortran framework Skelana is available and basic tests work fine.
- The C++ framework Idea is available but entirely untested. This tool requires attention.
Simulation
- delsim is available for all years 1992-2000. In depth evaluation is needed though
- None of the fast simulation attempts have been preserved (so far).
Reconstruction
- delana is available for all years 1992-2000 for MonteCarlo processing. In depth evaluation is needed though
- The DELPHI event server (des) which allows to reconstruct events from raw data is available. Tested on individual events. Tests with larger statistics would be nice
Event display
- delgra is fully available.
- Alternative displays (e.g. for the VFT) have not been preserved. If you know where they are, get in touch.
- wired has not been preserved. The code is missing.
Changes since 2002
Many years after the end of the experiment IT services are changing, and the software stack has to adapt to that to ensure the continued accessibility of the DELPHI data. Changes in services include:
- With CentOS7 the Fortran compiler g77 will finally disappear and is replaced by gfortran.
- Support of LSF is stopped at the end of 2018.
- RFIO has been abandoned. It won't work any longer. Data has to be read in from EOS
- CASTOR is gone and has been replaced by CTA. CTA is a pure tape system. Access to it is restricted.
- AFS is still around for home directories but project areas have been phased out. Backups exists so if you are missing something contact us.
- AFS scratch areas are gone.
- Old binaries no longer work due to change in glibc (or only in VMs supporting the old OS). Re-compilation and linking is needed for the affected libraries and executables.
- Fortran and C compilers are evolving, and getting more picky on things. In particular, g77 is gone.
- 32bit libraries start to disappear upstream. While still mostly available on RHEL9, there are no 32bit development packages on Ubuntu 20.04 and newer.
At the same time new services have arrived which replace the functionality of these disappearing services.
- EOS is taking over functionality from CASTOR and AFS
- HTCondor replaces LSF
- CVMFS is widely used for world-wide read-only sharing of software. It is available at all GRID sides, and works as well on PCs outside CERN.
- CTA replaced CASTOR
- gitlab and gitlab-ci for audit, version control and testing
- Multiple pre-alpha PHIGS source code libraries appeared with free licenses.
- In autumn 2021 a new initiative was launched to revive CERNLIB. The community CERNLIB 2022 supports 64bit builds, and thus prepares the way for a 64bit build of the DELPHI software stack.
Revised DELPHI software stack
Based on the software CD created in 2002 a new software environment has been created. It is the default since the switch of lxplus to CentOS7.
- All DELPHI data has been copied from CASTOR to EOS. On lxplus and lxbatch it can be directly accessed from /eos/experiment/delphi. The CTA tape copy (2 copies) remains intact for archiving and disaster recovery.
- Access to DELPHI data requires valid kerberos credentials for an account which is registered in the CERN unix group XX.
- The default compiler is gfortran. All libraries have been re-compiled using this version.
- The sources of the main components of the software stack have been transfered to https://gitlab.cern.ch/delphi . Any changes applied to the libraries for data preservation are tracked in the git history.
- All DELPHI libraries are available on /cvmfs/delphi.
- The sources are available at https://gitlab.cern.ch/delphi
- The event display has been migrated to OpenPHIGS which is free of charge. This new version of delgra is the default now. It works, albeit there is some known missing functionality which may or may not be followed up, as time allows.
The new software stack has no remaining dependencies on AFS, LSF or CASTOR. It works fine outside CERN as well. Currently, the following operating systems are supported:
- CentOS/CentOSStream-7,8,9 and compatible, namely RHEL and ALMA
- Ubuntu-18, 20 and 22
Setting up the software locally:
- To run the stack locally in 32bit, you need to have the 32bit X11 and gfortran libraries installed, e.g. you need gfortran-multilib for Ubuntu.
- To initialise this software stack, it is enough to source /cvmfs/delphi.cern.ch/setup.sh or /cvmfs/delphi.cern.ch/setup.csh (depending on the shell).
- The correct flavor is automatically selected. If you need support for other version (Linux only), please get in touch with us.
Note that the AFS area are gone for good, thus the cvmfs installation is the only remaining stack. Since 20/05/2023 the default is 64bit,
for all versions, which was already the case for newer vesions as of July 2022.
Using and testing
Please make use of pre-defined environment variables, specifically for compiler options. See here for more details.
Things to be done
Tasks still to be performed are tracked in the DPHEP JIRA task tracker. Input to any of the tickets is very welcome. Some remarks:
- AFS areas have been backed up on EOS. DELPHI notes are available there, as well as from the CERN document server CDS . If you detect something which is missing, please contact us.
- DELPHI web pages have been moved to EOS.
- It would be nice to add some more complex sample analyses to the new stack. Input on this is very welcome!
Disclaimer
It is essential to perform more meaningful in depth tests of the stack. Any help on this is highly appreciated. In any case, if you use DELPHI software please do cross checks with old archived data, and report any problems to delphi-core@cern.ch
List of recent changes:
- 28/4/2022 (US)
- Added/updated 32bit and 64bit versions for CentOS Stream 8 and 9
- Added 64bit version for Ubuntu 22
- 64bit versions in /cvmfs/delphi.cern.ch/64bit are ready for validation
- 14/6/2022 (US)
- Switch to new community cernlib
- Add openphigs and add back delgra
- Wrap rungra script in 32bit CentOS versions to use 64bit delgra. The 32bit version crashes in either Motif or in GL initialisation due to missing 32bit support on lxplus at CERN.
- Explicitely set/unset LEGACY environment variable in the 32bit and 64bit trees. Otherwise the paths are wrongly setup.
- Explicitly set LEGACY flags in 32bit/centos*/profiles/delphi.[c]sh
- 20/7/2022 (US)
- Event server: use recreated delana binaries for real data. Many thanks to John Wickens for helping out!
- Updated stacks for CentOS and CentOS Stream in /cvmfs/delphi.cern.ch/32bit and /cvmfs/delphi.cern.ch/64bit
- Switch to 64bit stack for Centos-Stream 8 and newer.
- Update dphep web page (this page)
- 24/11/2022 (US)
- Updated 32bit and 64bit versions for CentOS Stream 8 and 9 and Ubuntu
- Bump DELGRA version to 2.01
- IDEA is now included in these builds
- 20/05/2023 (US)
- Switch to CERNLIB tag 2023.02.16.0
- Updated 32bit and 64bit versions for CentOS, Alma and Ubuntu
- Updated Openphigs and re-add string dialog in DELGRA
- Archive obsolete versions on CVMFS in the attic folder
- Synchronise CC7 with the other supported versions. 64bit is now the default.
Links and references
Last modified: Wed Jul 20 15:33:00 CET 2022