Models of Networked
Analysis at Regional
Centres Project (MONARC)
January 14, 2000
CERN
Dr. Hans Hoffmann
DG Division
Professor Manuel Delfino
IT Division
CERN
Dear Hans and Manuel,
On behalf of MONARC, we are
pleased to inform you of our intention to continue our study and development
of the LHC Computing Models, by
extending the Project into a third
phase. We hope you will agree to the valuable nature of this additional
phase, in support of planning for LHC
Computing at CERN and at the worldwide-distributed Regional Centres.
The deliverables of MONARC Phase 3 are realistic technical options for the site and network architectures, and estimates of the associated resource requirements for LHC Computing. The results of Phase 3 will be presented to the experiments and CERN, in support of further Computing Model development for the Computing TDRs. The central theme of Phase 3 will be to base the model studies on large-scale prototypes at each stage, including the large scale event simulation, reconstruction and physics analysis studies planned by some of the LHC experiments starting next Spring. This will enable MONARC to face the key issues and problems of distributed data access, processing and analysis in a real working environment. By studying and modeling these real cases, MONARC hopes to develop more efficient and cost-effective strategies for LHC data analysis, making best use of the resources at CERN and the Regional Centres. The MONARC Simulation tool-set will also be further developed so that it can provide more realistic assessments of the site configurations, and the ability of a given distributed architecture to support the required workload within the limits of acceptable turnaround time.
In the following sections, we briefly summarize MONARC’s status, the motivations for Phase 3, the goals, scope and schedule of the project extension and its sub-phases, the equipment needs, and the relationship to other Projects and groups in CERN/IT.
The MONARC Project has successfully met all of its major milestones, and is well on the way to meeting its primary goals, including
· identifying first-round baseline Computing Models that could provide viable (and cost-effective) solutions to meet the simulation, reconstruction and analysis needs of the LHC experiments
· providing a powerful (CPU and time efficient) simulation toolset that will enable further Model studies,
· providing guidelines for the configuration and services of Regional Centres, and
· providing an effective forum where representatives of actual and candidate Regional Centres may meet and develop common strategies for LHC Computing.
The progress of MONARC is documented in its Mid-Project Progress Report[1] and the talks by H. Newman and I. Legrand at the LCB Computing Workshop in Marseilles[2] (October 1999). The MONARC Technical Notes[3] cover the specifications for possible CERN and regional centre site architectures, regional centre facilities and services, and the testbed studies used to validate and help develop the MONARC Distributed System Simulation, and to determine the key parameters in the candidate baseline Computing Models. The principles, features and examples of runs of the MONARC Simulation are described in Section 3 and the technical Appendix of the MONARC Progress Report, and in an upcoming paper on the System’s recent developments[4]. A series of short papers on: the structure and operational experience with the Simulation system (using the results of the Analysis Process Working Group); the work of the Architectures Working Group; and the testbed studies and simulation validation in local and wide-area network environments, will be completed in advance of the CHEP 2000 Conference.
The motivations for MONARC Phase 3 were spelled out in the Progress Report in June 1999:
“We believe that from 2000 onwards, a significant amount of work will be necessary to model, prototype and optimise the design of the overall distributed computing and data handling systems for the LHC experiments. This work, much of which should be done in common for the experiments, would be aimed at providing "cost effective" means of doing data analysis in the various world regions, as well as at CERN. Finding common solutions would save some of the resources devoted to determining the solutions, and would ensure that the solutions found were mutually compatible. The importance of compatibility based on common solutions applies as much to cases where multiple Regional Centres in a country intercommunicate across a common network infrastructure, as it does to sites (including CERN) that serve more than one LHC experiment.”
A MONARC Phase 3 could have a useful impact in several areas, including:
· facilitating contacts, discussions, interchanges, for the planning and mutually compatible design of centre and network architecture and services (among the experiments, the CERN Centre and the Regional Centres)
· providing a modeling consultancy and "service" to the experiments and Centres
· providing a core of advanced development activities aimed at system optimisation, and pre-production prototyping
· taking advantage of MONARC’s synergy with (and complementary to) the work on distributed data-intensive computing systems beginning this year in other "next generation" R&D projects[5], such as those on Grid Computing.
The Phase 3 study will be aimed at maximizing the workload sustainable by a given set of networks and site facilities, while reducing the long turnaround times for certain data analysis tasks. Unlike Phase 2, the
optimization of the system in Phase 3 would no longer exclude long and involved decision processes, where a momentary lack of resources or “problem” condition could be met with a redirection of the request, or with other fallback strategies. These techniques could result in substantial gains in terms of work accomplished or resources saved.
Some examples of the complex elements of the Computing Model that might determine the (realistic) behaviour of the overall system, and which could be studied in Phase 3 are
· Resilience, resulting from flexible management of each data transaction, especially over wide area networks
· Fault tolerance, resulting from robust fall-back strategies and procedures (automatic and manual, if necessary) to recover from abnormal conditions (such as irrecoverable error conditions due to data corruption, system thrashing, or a subsystem falling offline)
· System state tracking, so that the capability of the system to respond to requests is known (approximately) at any given time, and the time to satisfy requests for data and/or processing power may be, on average, reliably estimated, or abnormal conditions may be detected and in some cases predicted.
MONARC in Phase 3 could exploit the studies, system software developments, and prototype system tests scheduled by the LHC experiments during 2000, to develop more sophisticated and efficient Models than were possible in Phase 2. The Simulation and Modelling work of MONARC on data-intensive distributed systems is more advanced than in PPDG or other NGI projects in 2000, so that MONARC Phase 3 could have a central role in the further study and advancement of the design of distributed systems capable of PetaByte-scale data processing and analysis. As mentioned in the PEP, this activity would potentially be of great importance not only for the LHC experiments, but for scientific research on a broader front, and eventually for industry.
MONARC Phase 3’s central goal is to develop more realistic Computing Models meeting the LHC Computing Requirements than were possible in the Project’s first two phases. This goal will be achieved by confronting the Models with realistic large scale “prototypes” at every stage, including the large scale trigger, detector and physics performance studies that will be initiated by some of the experiments in the coming year. By assessing these “Use Cases” involving the full simulation, reconstruction and analysis of multi-Terabyte data samples[6], MONARC will able to better estimate the baseline computing, data handling and network resources needed to handle a given data analysis workload.
During Phase 3, MONARC will participate in the design, setup, operation and operational optimization of the prototypes. The analysis of the overall system behavior of the prototypes, at the CERN site and including candidate Regional Centre sites, will drive further validation and development of the MONARC System Simulation. This is expected to result, in turn, in a more accurate evaluation of distributed system performance, and ultimately in improved data distribution and resource allocation strategies. Strategies that will be recommended to the experiments before their next round(s) of event simulation, reconstruction and analysis studies.
As a result of this mutually beneficial “feedback”, we also expect to obtain progressively more accurate estimates of the CPU requirements for each stage of the analysis, and of the required data rates in and out of storage and across networks. We also expect to learn, in steps, how to optimize the data layout in storage, how to cluster and re-cluster data as needed, how to configure the data handling systems to provide efficient caching, and how to implement hierarchical storage management spanning networks, in a multi-user environment.
In addition to the large scale studies of simulated events initiated by the LHC experiments, MONARC will develop its own specific studies using its Testbed systems[7] to explore and resolve some of the problems and unexpected behaviors of the distributed system that may occur during operation of the large-scale prototypes. These in-depth studies of specific issues and key parameters may be run on the MONARC testbeds alone, if adequately equipped, or in tandem with other large computing “farms” and “data servers” at CERN and elsewhere.
In the course of studying these issues using testbeds and prototype systems, we expect to identify effective modes of distributed queue management, load balancing at each site and between sites, and the use of “query estimators” along with network “quality of service” mechanisms to drive the resource management decisions.
One technical benefit for the HEP and IT communities that will result from MONARC Phase 3 is the development of a new class of interactive visualization and analysis tools for the distributed system simulation. This work, based on new concepts developed by MONARC’s chief simulation developer I. Legrand, has already begun during MONARC Phase 2. Based on the initial concepts and results, we are confident that by the end of Phase 3 we be able to make available a powerful new set of Web-enabled visual tools for distributed system analysis and optimization, that will be applicable to a broad range of scientific and engineering problems.
Following the ORCA3 software release [8] and the CMS High Level Trigger (HLT) 1999 milestone, it became evident that a coordinated set of future ORCA releases and HLT studies of increasing size (in terms of the numbers of events and data volumes) and sophistication would be required. In order to carry out the ORCA4 release of the software and the subsequent HLT study in the first half of 2000, two of CMS’ major milestones have been advanced to next Spring:
· Simulation of data access patterns March 2000
·
Integration of databases and mass (tape) storage March
2000,
where we will use large volumes of “actual” (fully simulated and reconstructed) data[9]. This has led to a strong and immediate demand for MONARC’s help with the design and optimization of data structures, data access strategies, and resource management, to make good use resources at some Regional Centre sites as well as CERN.
In a similar vein, ATLAS is planning large-scale studies using large samples of GEANT4 data, and ALICE is planning a series of increasingly large “data challenges”.
In the course of MONARC-assisted studies such as these, working closely with the experiments, MONARC is confident that it will be able to progressive develop more realistic Computing Models, and more effective data access and handling strategies to support LHC data analysis.
The preliminary schedule for Phase 3 covers a period of approximately 12 months, starting when Phase 2 is completed. The completion of Phase 2 will be marked by the submission of the final MONARC Report on Phase 1 and 2, in March 2000.
We foresee that Phase 3 will proceed in several sub-phases:
·
Phase 3A: Decision
on which prototypes to exploit and/or build. Develop general plan for
cooperative work with the LHC experiments and CERN/IT. This will require a
§
Joint
MONARC/Experiments/Regional Centres Working Meeting
·
Phase 3B: Specification
of resources and prototype configurations
·
Setup of simulation and prototype environment
·
Phase 3C: Operation
of prototypes; operation of MONARC Simulation System; analysis of results
·
Phase 3D:
Feedback between prototypes and studies with MONARC Simulation; strategy
optimization.
The MONARC Phase 1 and 2 Report will contain a proposal for a somewhat more detailed set of milestones and schedule.
The equipment needs for Phase 3 involve access to existing or planned CERN/IT facilities, with some possible moderate upgrades depending on the scale of the prototype simulation/reconstruction/analysis studies to be carried out by the LHC experiments. An disk and memory upgrade to the existing Sun E450 server (MONARC01) purchased by CERN for MONARC will also be needed.
While the equipment requirements will be better specified in Phase 3A, we include a list of preliminary requirements for discussion with CERN/IT, and for planning purposes:
§ Access to a substantial computing and data handling system managed by CERN/IT, consisting of a Linux CPU farm, and a Sun data server, linked over Gigabit Ethernet to internal and external networks
§ Access to a Multi-Terabyte robotic tape store
§ Non-blocking access to wide area network links to the main (potential) Regional Centres[10]
§ Temporary use of a large volume of tape media (e.g. for ALICE).
There is a specific need to upgrade the Sun MONARC01 server, to make it a sufficiently capable “client” that will be used together with the larger system indicated above:
§ Memory upgrade to at least 1 Gigabyte
§ Attachment of RAID disk array of at least 1 Terabyte.
During MONARC Phase 3, we expect to take advantage of the substantially higher bandwidth network connections (in the range of 30 to 155 Mbps) that will become available this year between CERN and Europe, Japan and the US. We will work with CERN/IT to better understand the technical requirements and means to best use these networks to further study and prototype the LHC distributed Computing Models, as well as the requirements for reliable and secure high throughput connections to key points on the CERN site.
During MONARC Phase 3 we intend to continue our close collaboration with the LHC experiments, and also to work in closely with the CERN/IT groups involved in the development and use of large databases, as well as data handling and processing services. Our role with respect to the LHC experiments will be to seek effective strategies and other common elements that may be used in the experiments’ Computing Models. While MONARC will have its own unique role, using distributed system simulations to optimize present as well as future large scale data analysis activities for th LHC experiments, we will also keep close contacts with present (PPDG) and future Grid Computing projects in the US (GriPhyN) and in the European Community.
For the MONARC Collaboration:
Harvey Newman,
CALTECH/CMS Laura Perini, INFN-Milan/ATLAS
[4] I. Legrand, ``MONARC Simulation System and Graphical User Interface for Interactive Distributed System Optimization’’, to appear in January 2000.
[5] Details on the synergy between a MONARC Phase 3 and R&D projects such as the recently approved "Particle Physics Data Grid" (PPDG) project and the proposed “GriPhyN” (Grid Physics Network) project may be found at http://www.cern.ch/MONARC/docs/progress_report/longc7.html . Also see the PPDG and GriPhyN Websites at http://www.cacr.calech.edu/ppdg and http://www.phys.ufl.edu/~mre/ .
[6] One example, related to the CMS High Level Trigger studies, is described briefly below.
[7] At CERN, in Italy, Japan, and the US.
[8] This is the first release of mainstream OO software by an LHC experiment involving persistent objects, and qualified for initial use in support of trigger and physics performance studies.
[9] A typical dataset for one of these studies, as discussed with CERN/IT, would be 106 events, requiring 1010 SI95-sec for production processing (simulation and reconstruction) and 1-to-several Terabytes of diskspace.
[10] Examples include Italy, France, Japan and the US. In the latter two cases, at least 10 Mbps of bandwidth is expected to be available for dedicated mission-oriented and distributed system development purposes, starting in the Spring of 2000.