summarizes the evolution of the total batch CPU needs of CERN experiments up to 1992. It has been assumed that in providing the UA1 need of 45 units during the four months of data taking it is possible to make use of 50% of this capacity for other purposes when UA1 are not taking data. All the remaining non--LEP experiments, including other work at the SPS collider have been assumed to be equivalent to approximately two LEP experiments, which is an assumption allowed by the uncertainties described above. Total Batch CPU Needs for the CERN Physics Programme It has been assumed that the components of the MUSCLE CPU estimates from Table 1 can simply be added. Any spare capacity outside data--taking periods will be more than offset by code development and calibration activities which were explicitly neglected by MUSCLE. also shows an estimate of the CPU cycles used for networking, trivial interactive response, and various other system services. It is assumed that this can be kept down to 15% of the total needs. The estimated CPU needs of physics experiments invariably exclude such overheads, which must nevertheless be included when planning the size of systems to be installed.

Type of CPU

Mainframes used as scalar processors

This category includes such machines as IBM, IBM--compatible, Cray (used on unmodified code), and the larger VAXes. There is no doubt that almost all computing for HEP can be performed on such mainframes, and, as explained later, the differences between the (list)price/performance of the various machines is small. The large fraction of HEP computing which involves high volume data handling, linked to substantial CPU needs, is ideally suited for conventional mainframes.

Mainframes used as vector processors

On most modern mainframes vector processing hardware is either built in, or a relatively cheap optional extra. Thus only benefits can accrue from allowing the compiler and the vector hardware to do whatever they can to speed up unmodified HEP code. However, a substantial body of experience shows that the resultant increases in speed rarely exceed 10 to 20%. Although it is too early to present a final verdict on the success of 'vectorization' through code modifications, it is now widely accepted that it would be a mistake to expect any substantial speed up for reconstruction and physics analysis code. By its nature this code has to be flexible, and therefore unstable. Vectorization of this code to achieve overall speed improvements of a factor two may well be possible, but at a cost in manpower and code maintainability which most physicists would consider unacceptable. There is considerably more optimism about the prospects for speeding up Monte Carlo simulation through algorithm re--design rather than code modification. Work is now in progress to produce a vectorized version of GEANT3 although it is too early to make predictions of the benefits. It has already been demonstrated [Footnote: e.g. K. Miura (Fujitsu USA), Lectures at the CERN Computing School, Oxford, 1988. ] that the stable, experiment independent, components of a shower Monte Carlo can be re--written very effectively for a particular vector machine, but up to now, no complete Monte Carlo program has been vectorized. Vectorization of Monte Carlo code may make it possible to generate simulated data at the level of detail, and with the statistical precision, merited by current and future detectors. Continued efforts in this area should be strongly encouraged.

Parallel computing

Most HEP computing exhibits inherent event--level parallelism. This parallelism offers opportunities to control the cost and expand the flexibility of HEP computing. To define the role of parallel computing for HEP in the 1990s, it is necessary to examine both the fraction of HEP computing which can be parallelized, and to specify which of many possible parallel architectures should be used. Most experiments estimate that 40 to 70% of their minimum needs could be provided by parallel computing systems. Compared with the use of conventional mainframes, significant but tolerable inconvenience is expected, which is partially offset by the hope that cheap parallel systems could make it possible to exceed the minimum CPU requirements, with corresponding benefits for physics. Exploiting HEP parallelism by allowing one experiment to run a job using all the CPUs of an IBM 3090 brings little or no benefit. [Footnote: Some large programs may be required to use the CRAY processors in parallel because they might leave insufficient memory for other programs to exploit the remaining processors. This form of parallelism is not considered further in this report. ] Benefits arise when the processors used in parallel are much cheaper per unit of CPU power than mainframes. Price/performance can be improved by, for example, using mass--produced CPU chips, or by restricting I/O and inter--processor communications possibilities. Processor 'farms' have been used in HEP for some years. Physicists who have replicated existing designs have found them cost--effective solutions for stable 'production' programs involving limited I/O. Whether the various emulator and 'ACP' programmes have been cost--effective as a whole is more difficult to assess. A major justification for the development of such processors has been their use as event filters in data--acquisition systems. Thus many would argue that the development costs should be ignored when evaluating the processors for off--line use. Whatever the true costs of their development, these processors certainly allow a cost--effective expansion of CPU resources for any experiment in which they are already heavily used. The optimum future parallel architectures for HEP are not immediately obvious and this working group believes that the CERN (or even HEP) community should initiate a continuing and coordinated parallel computing project. The project should be rooted in reality, by requiring that it provide almost immediately a parallel computing service to users, while exploring the most promising approaches for the future. The most promising approach for early investigation appears to be that of using commercially available processors adequately linked to the data handling capabilities of a centre's mainframe system. The major challenge would then be to make the parallel processing system as almost as easy to use as a mainframe of the same total power. Other approaches, such as building special processor boards, or buying complete parallel computers, should also be investigated.

The Move towards Personal Workstations

Central time--sharing services are still a cost--effective solution to the provision of general--purpose interactive computing for an organization of CERN's size, [Footnote: A paper of A.Osborne and L.M.Robertson, prepared in February 1987, calculated the average cost of central computing, including all overheads (full costs of staff, management, energy, buildings, etc.), as 137 SF per standard CERN CPU hour. This is approximately double the cost calculated by taking into account only the cost of IBM hardware. Thus the cost of the current VM service (supporting 450 simultaneous users at peak times, and 2,300 different users each week, consuming a total of 800 CPU hours) might be estimated as: 110 kSF per week; 250 SF per peak user per week; 50 SF per user per week.

The cost of the basic hardware of a simple workstation today, including a minimal contribution to the cost of a "boot--host" or disk server, is 15 kSF plus 25% per annum maintenance. If we assume a three year lifetime for the station, this represents only 170 SF per week per workstation. The hidden costs for workstations are almost certainly higher than those of the Computer Centre, where economies of scale are possible, especially if manpower--intensive functions like disk backup are included. No realistic assessment of these costs has been carried out (to our knowledge), but it is clear that the cost per peak user is very similar for both types of service. ] especially when measured in terms of cost per individual user of the service. Measured in terms of cost per active user at peak times, however, personal workstations of the VAXstation 2000 or Apollo DN3000 class are very competitive, and in addition provide a quality of service to the end--user which is incomparably better than that of a time--sharing service: human interface with bit--mapped screen, windows, pop--up menus and mouse; guaranteed availability - no delays, queues, interruptions outside the control of the user; personal choice of important details of the system. The evolution towards personal workstations as a vehicle for general--purpose interactive computing started early in 1987, following a substantial improvement in price/performance over most of the market, and is now well under way.

However, there are a number of points which must be considered.

Personal workstations cannot be shared Personal workstations are personal - there must be enough of them to ensure that a convenient one is available when required. The result will inevitably be that these devices will often lie idle, but, as the above cost comparison shows, we should not worry about this if the workstation is used regularly at peak periods. If some batch work can be run on workstation clusters this should be treated as an added bonus. User Productivity It is clear that the end--user will be much more productive when using his personal workstation than he would be using a central interactive service. However, workstations are real computers, and require to be operated and managed: hardware installation and maintenance; software support; file--system maintenance; network monitoring; etc. It is important that the user does not exchange his new--found productivity for a part--time job as systems programmer cum operator cum network manager. To obtain the full benefits of the workstation a level of central service, staffed by specialists, equivalent to that provided for the central time--sharing services must be made available. Integration with Central Services Workstations must be very well integrated with central services and with each other. While enjoying the advantages of his dedicated computer system, the user still requires to share data with his collaborators and to have access to central data storage facilities, batch services, printing services, networks, etc. Integration of workstations with the central IBM service has improved substantially at CERN during the past twelve months, but there remain many deficiencies in terms of both functionality and performance. Networking The high bandwidth required between personal workstations and their file servers will have a significant impact on the general networking infrastructure of CERN and its community. While this in turn will necessarily place constraints on the topology of workstation clusters and services, it is essential that these are minimized, and the potential needs of workstations taken account of in the evolution of the network infrastructure, even at the expense of the installation of substantial over--capacity. The Fast--Moving Workstation Market The technology of workstations continues to advance at breakneck speed, and shows little sign of slowing down. Apart from the problem of rapid obsolescence, this also poses serious support problems. At present CERN has effectively standardized on VAXstation/VMS and Apollo systems, in order to simplify support. This standardization has been the result of a judicious choice of products and the provision of (albeit limited) central support. While there are no immediate reasons to question the choice of these two systems, we must ensure that we benefit fully from developments in the market, and the selection of systems supported should be regularly reviewed. The de facto standardization of almost all suppliers on the use of Unix, ASCII and IEEE floating point (with DEC a notable exception) should limit to some extent the problems of support. Low--end workstations/PC--s This study has assumed a definition of a workstation which includes a certain minimum CPU power, memory size, graphics screen and mouse, but also the provision of a conventional operating system with (Fortran) program development environment. It is the last factor which most clearly distinguishes personal computers from personal workstations. However, with the advent of machines like the Macintosh II this distinction is becoming very blurred. It is clear that the Mac II is an extremely attractive system, which is perfectly capable of running large Fortran programs, and has good networking facilities in addition to its excellent human interface. It will certainly be used in substantial numbers at CERN as a low--end general--purpose workstation.

Data Handling

The MUSCLE report made quantitative estimates of LEP data volumes, and assessed some of the benefits conveyed by keeping some of the LEP DST data on disk storage. The duplication of data needed for distribution to universities and regional centres was not included. Wider discussion of the MUSCLE data volumes has tended to show them as lower limits, particularly with respect to the 'Master DST's' which may be bigger than the MUSCLE estimates by large factors. Noting that many non--LEP, non--collider, experiments have or expect raw data rate of thousands of tapes per year, it is reasonable predict total CERN data volumes of 8 to 10 times the MUSCLE single experiment estimate. This implies at least 300,000 active tape cartridges in 1992 assuming use of 200 Megabyte cartridges. In the absence of fibre optic networking, distribution of data to universities and regional centres will require at least a similar number of cartridges. It is reasonable to assume that raw data are copied once at most, but master DST's will have to be copied many times if off--site physicists are to have access to more than a fraction of their data. The MUSCLE report set a target of 100 Gigabytes of disk space at CERN per LEP experiment by end 1991. This represents 1.4% of the expected data volume, and would suppress a large fraction of the tape mounts, in addition to facilitating physics analysis. It was also urged that this disk capacity was complemented by automated cartridge handling. Such a system would probably have a capacity of 10% to 20% of the total data volume. As argued above, it is reasonable to assume that an optimized storage hierarchy at CERN would have about 8 to 10 times the capacity needed for a single LEP experiment. Together with system needs estimated at 20% of the total, this leads to a target of over 1000 Gigabytes disk storage, and automated handling of at least 40,000 cartridges by end 1991. Automatic migration of relatively inactive files from disk to tape--cartridge storage would be of considerable benefit in managing experiments' DSTs. For physicists' "personal" files, which can easily and justifiably reach hundreds of Megabytes per person, such automatic migration is even more important.

Networking

The existing CERN site network is only marginally able to support data access from workstations, and does not support bulk data transfer between computers. Physics analysis at CERN would benefit from improvements in both areas and efforts are underway to achieve this. There is no physics reason why access to data from universities and regional centres should need a lower performance than CERN--site networking. This runs contrary to conventional networking philosophy which assumes that traffic falls off rapidly with increasing distance. Although it is financially unreasonable to expect that wide area networking should always offer local area network performance, the effectiveness of off--site physics analysis will increase as this goal is approached. A recent study [Footnote: E--A. Knabbe (CERN--DD) "What would the LEP experiments do with 2 Megabit/second link(s)?", September 1988. ] of how LEP experiments would use 2 Megabit/second links showed that the links would be used by a wide variety of services (paralleling the wide variety of services offered on the CERN site). For many activities remote users would have access similar to that on the CERN site, although a truly integrated environment for the location--independent analysis of HEP data will require much higher speeds. Although 2 Megabit/second links are not sufficient to provide totally transparent access to arbitrarily large data samples in real time, their effectiveness can be enhanced by having sufficiently large data storage facilities at the remote end of the link. Frequently accessed data can then be stored at the university or regional centre. The management of these distributed data stores could, in principle, become totally automatic. A substantially integrated environment, supporting analysis activities requiring access to Gigabytes of data, could be created by using networks based on optical fibres between universities and regional centres and between the centres and CERN. The current capacity of such links is in the range 34 to 144 Megabits/sec., and the same physical fibres will be able to support at 4 or 16 times these rates within the next few years. The true commercial cost of such fibre--optic networking would be low enough to make its early use by HEP very attractive. In reality, its early use in Europe would require a significant change in PTT tariff policies.

EVOLUTION OF CENTRALLY ORGANIZED COMPUTING AND NETWORKING AT CERN AND AT REGIONAL CENTRES

Introduction

The next five years will be a challenging period in the development of computing for offline High Energy Physics, as we progress towards a distributed computing environment, exploiting the fast--developing technologies of workstations and networks, at the same time ensuring that the needs of the early years of LEP are met effectively.

The main elements in the evolution of centrally organized computing, at CERN, in national and regional centres, and in the universities, are:

• The growth of batch capacity for LEP data analysis, involving the expansion of conventional general--purpose facilities at CERN and in major centres, the large--scale exploitation of cheap parallel computers, and the integration of these with cartridge tape and disk storage facilities on a scale new to HEP.
• The establishment of personal workstations as the principal tools for general--purpose interactive computing. The technological and economic conditions are ripe for this development, but the challenge will be to develop the support services, the network management and the tools to integrate workstations with each other and with central services for file management, batch processing, etc.
• The development of distributed co--operative processing for interactive data analysis, with different components of a single program executing simultaneously on a powerful workstation and on one or more central CPU and file servers.
• The introduction of high performance wide--area networking, which will enable the above developments to evolve on a European--wide basis and allow the individual physicist to work effectively from his home institute.

It is important that this evolution take place smoothly. While we must press ahead with the introduction of new technologies for distributed computing, we must also learn how to exploit these effectively, and be careful not to abandon prematurely existing proven services.

Evolution of Batch Capacity

There are three main sources of "production" batch capacity available to high--energy physicists: the CERN central computing services; large national physics institutes or computing centres; smaller systems such as departmental VAXes, emulator farms, off--duty personal workstations, etc.

CERN Computer Centre Batch Facilities

By the end of 1988, the CERN Computer Centre will provide two batch services, based on the following equipment.

1. IBM 3090--600E/VF - 39 CERN units (scalar);
2. Siemens 7890S - 12 units (no vector capability);
3. Cray X--MP/48 - 32 units (scalar).

These figures are the nominal capacities of the computers. In practice, a significant and growing proportion of the IBM service will be used for interactive computing, which consumes about 20% of the IBM--compatible capacity delivered at present. An additional 10% of the CPU cycles are consumed by general services, such as network file servers, mail handling, and printing. This percentage will increase significantly with the development of distributed computing. The requirement to increase the general--purpose batch capacity to over 150 CERN units by the end of 1992 may be met in a number of ways. This section discusses some of the general factors involved, and suggests a model which is reasonable today in order to enable costs to be estimated. In practice the route chosen will depend to a large extent on future product announcements and the results of financial negotiations with manufacturers.

Competitive and complementary services

Competitive batch services were first introduced into the Computer Centre at the end of 1976, when an IBM system was installed alongside the existing CDC 7600. The original motivations for installing a second manufacturer's equipment were to provide a wider service for users, allowing them to benefit from the different strengths of the different systems, and to provide greater possibilities for compatibility with services in outside institutes, avoiding placing the latter in a position where they were under pressure to buy from CERN's single supplier. These remain good reasons for continuing to operate complementary services today. However, the past twelve years have demonstrated that this competitive environment has additional advantageous effects: on the suppliers, in terms of quality of service as well as more imaginative commercial terms; on the staff providing the services; on the users, who are encouraged to avoid dependence on proprietary technology - software, operating systems and hardware. The scale of CERN central computing is sufficiently large for the operation of two services, and it is important for the flexibility and vitality of the central batch services that the competitive element remain.

Operating systems

Evolution of the capacity implies also evolution of the operating systems which can best provide it. There is clearly a strong incentive to provide a stable environment for the users, and operating system changes and the introduction of incompatible hardware must be kept to a strict minimum, while ensuring that obsolescence is avoided and a reasonable flexibility to exploit new hardware is maintained. These conflicting requirements are hard to reconcile. In preparation for LEP, the IBM service is being moved to a new base on the VM/XA/SP operating system. This will ensure that the service will be able to exploit the hardware innovations which will appear during the next few years, particularly in terms of memory size and multi--processor configurations, at a minimal inconvenience to the user. Similarly, the Unicos operating system on the Cray will enable either of the Cray architectures (the X--MP/Y--MP and Cray--2/Cray--3 lines) to be used. There is therefore no need to make major changes in operating systems on these services during the next five years.

There is, however, considerable activity on the part of most computer manufacturers and in various standardization bodies in the area of Unix, which promises to provide a single standard user interface on all computer systems currently of interest to offline High Energy Physics (e.g. Cray, IBM 3090 and compatibles, DEC VAX, Gould, Apollo, IBM PC, Apple Macintosh). While there is ample scope for argument about the attractiveness of Unix as an operating system, we now have practical experience of good manufacturer--supported implementations (e.g. Unicos) which demonstrate that it has no fundamental flaws (and indeed has many advantages) in our environment. The benefits of a standard user interface are clear, but we must carefully weigh these against other factors like additional functionality, ease of use, and manufacturer commitment, before deciding to abandon excellent systems such as VMS and MAC OS. IBM's AIX system for the 3090 is particularly interesting in view of the company's declared commitment to it, and a number of deficiencies perceived in the current versions of alternative 3090 operating systems. However, the first release is not scheduled until 1989, and there are important features missing, such as tape and batch support. We should actively study the AIX system when it becomes available, and also investigate how we can best influence the various Unix standardization activities (POSIX, X/OPEN, the Open Software Foundation).

Cost comparison of Cray and IBM--style computing

The IBM--compatible and Cray batch services are very different in many respects, the most significant being that the Cray has a substantially higher potential vector performance, but has a limited disk configuration (due to the cost of the high--performance disks which alone are supported on Cray machines), and lacks virtual memory. Today, and probably during the five--year period being considered, the vector capability of these machines will not be of major significance in overall throughput, although individual programs may make dramatic gains, and in turn perform substantially better on the Cray. A brief analysis of comparative list price costs [Footnote: The list price of a Cray Y--MP system with 8 processors, 32 Megawords of main memory and a 128 Megaword "solid state disk" is $20M. 200 Gigabytes of disk space cost about $10M. The total cost of a system with a capacity of about 96 CERN units is therefore $30M, or about 500 kSF/unit ($1=1.6 SF).

The list price, including standard academic discount at 20%, of an IBM 3090--600S/VF, which would deliver about 50 CERN units, together with 256 Megabytes of main memory, 256 Megabytes of extended store and 100 Gigabytes of disk space is about 27 MSF, or about 550 kSF/unit.

The list price for a four processor Amdahl 5990--1400, with 256 Megabytes of main memory, 512 Megabytes of extended memory, and 100 Gigabytes of disk space, but no vectors, and which would deliver about 50 CERN units is about 30 MSF. A discount would normally be expected. ] of IBM and Cray systems which will be available at the beginning of next year shows that these systems are rather competitively priced in terms of scalar computing capacity per Swiss franc. Thus, while Cray--style computing does not have to be financially justified in terms of vector usage, the inconvenience of restricted memory and disk capacity will limit the type of work for which these machines will be suitable.

Manpower implications of the Cray service

It is clear that the provision of two separate batch services using different equipment has direct manpower implications. These are most significant during the installation of the hardware and introduction of the service. However, the style of batch service offered on the Cray, aimed at a relatively small number of very large users, restricts the continuing support manpower considerably, particularly in comparison with the manpower required to run a general--purpose service like that offered on the IBM systems. About 5 staff are required to maintain the Cray service in addition to those which would be required to run a similarly restricted service provided by duplicating the hardware and software environment of the general--purpose IBM service. This may be considered as the cost of running complementary batch services. Although it is not possible to quantify the value of any single element of a financial negotiation, the competitive environment generated by the two--manufacturer policy for batch capacity has been an important factor in the achievement of keen commercial terms for computer hardware in recent years, largely, if not completely, off--setting the cost of the additional manpower involved.

The multi--purpose CERN VM service

The VM service operated on IBM and compatible computers by the CERN Computer Centre provides a number of quite different facilities, the most important of which are: major batch service; general--purpose interactive service; "front--end" support for the Cray; printing and text--processing facilities for many other computers and workstations, in particular the central VMS service; file server for workstations and small computers.

Operating this general--purpose service clearly requires much greater manpower than that needed for a straightforward batch service, due to the variety of facilities to be supported and the overall complexity of the system. It must also be noted that the ancillary services consume a non--negligible fraction of the CPU capacity of the configuration.

Over the next few years, as discussed in later sections, the VM service will play an increasingly important role as a network file server, which will require a substantial amount of high--priority CPU capacity. COCOTIME [Footnote: The CERN committee which allocates central computing resources to experiments.] and the Computer Centre will find the task of scheduling resources increasingly difficult in this complex environment.

IBM--compatible systems

IBM--compatible systems have served the industry well, both in improving the price/performance of IBM--style computing but also in stimulating IBM to build powerful processors which perform well in the scientific and engineering markets. However, the competitive response of IBM has also included the provision of obscurely specified hardware and microcode enhancements which are essential for the efficient operation of the machine using IBM software. For example, the new VM/XA/SP operating system cannot operate on CERN's Siemens 7890S computer, installed less than four years ago. Great care must therefore be taken in assessing the overall financial benefits of IBM--compatible systems.

DEC batch capacity in the CERN computer centre

The DEC services in the CERN Computer Centre have been set up in order to provide interactive computing facilities. However, due to the fact that an interactive service must cater for peak daytime loads, some 60% of the installed capacity is available for batch. At present this amounts to only about 4 units, including the capacity of the engineering service VAXes, but the recommendation below for the expansion of the interactive service could lead to somewhat greater capacity being available for batch. This will be of interest to small experiments which are thereby able to do all of their computing on VAX. The capacity available will not, however, exceed 10% of the total general--purpose batch capacity at CERN, and therefore does not justify the operation of a full--scale batch service.

DEC is not considered here as an alternative to IBM--compatible and Cray for the main central batch facilities, because of the large number of CPUs required to achieve a reasonable capacity, although the cost of an appropriately configured system is comparable with the others. [Footnote: A cluster of four of DEC's largest systems, the four--processor VAX 8840 with 128 Megabytes of memory, together with 50 Gigabytes of disk space and 4 disk controllers, costs about 10.5 MSF with standard discount. This should deliver a nominal total of 24 units on sixteen processors. The multi--processing efficiency of the computers and our ability to schedule 16 processors efficiently for general--purpose batch are unknowns, but we might expect a degradation in throughput perhaps as high as 20%. The cost per CERN processing unit is therefore in the range of 440k SF to 550 kSF.] While we can expect more powerful CPUs to be delivered by DEC over the next few years, the other manufacturers will also introduce new equipment, and it is not clear that the relative position will change significantly in the short term.

Model for evolution of the central batch services

Replacement of the Siemens 7890S The first step is the replacement of the aging and obsolescent Siemens 7890S, which is not capable of running the latest version of the VM operating system, and which will provide less than 25% of the IBM--compatible capacity after the installation of the IBM 3090--600E system at the end of 1988. The Siemens should be replaced at the end of 1989, about six months after the startup of the LEP machine, by a system with at least 50 units of scalar capacity [Footnote: This will bring the total IBM--compatible capacity to 90 units, of which we assume 20% will be used for general--purpose interactive work.] This could be purchased from IBM itself, or from a compatible supplier. Clearly, the chosen system must run the latest version of the VM operating system, with reasonable confidence that it will not become obsolete in this sense during its four--year lifetime. Possible candidates today would include a four processor Amdahl 5990--1400 or an IBM 3090--600S.

Assuming that a 40% discount over list price could be achieved, the cost of this operation will be around 18 MSF.

Note that the existing IBM computer, installed originally early in 1986, will itself have to be replaced by the end of the period being considered.

1. Cray System

   The Cray system was installed in the spring of 1988, and it is therefore too soon to assess fully the effectiveness of this style of service for the different types of processing required for LEP physics. It is clear that it will be better suited to work which is CPU--intensive, and that a very large disk capacity is excluded due to the high cost of the disk equipment. The I/O limitations are, however, sometimes over--emphasized - the system has excellent tape support and the 50 Gigabytes of disk space which should be available early next year will enable a good tape staging service to be provided to support most Monte Carlo and production programs.

   Vector support will become available on almost all high--end processors over the next few years for relatively little additional cost. The ability of high--energy physics to benefit from such hardware would bring significant financial savings, even if the overall improvement in throughput is relatively low, due to the high total cost of the investment in such a large system. In order to stimulate the pioneers in vectorization a machine with an outstanding vector performance should be maintained beyond the life of the present Cray.

   Subject to the satisfactory development of the Cray service, the current X--MP/48 should be replaced during 1991 by a system with at least 80 units of scalar capacity. The desire to minimize change will make the Y--MP system an obvious candidate (the eight processor model should deliver over 90 units). However, it may well be more attractive to install the Cray 3 system, if it is available at that time, providing a substantially better scalar performance with its GaAs technology, and a very large real memory. Clearly any other comparable supercomputers which may become available must be carefully considered, with due attention paid to the importance of the needs for a competitive environment and minimal change of operating system mentioned above. The system should include at least 100 Gigabytes of disk space, and we estimate a total cost of around 30 MSF, assuming a substantial discount.

2. Disk Capacity

   The evolution of the disk capacity of the IBM--compatible service is covered in a later section on data storage.

CERN Computing Centre - Evolution of Batch Capacity

The nominal CPU capacity of the IBM service has been reduced by 30% in 1988, rising to 40% in 1991, to cover the part used by general--purpose interactive and distributed computing services.

Large Regional Centres

In order to make a major impact on LEP data analysis, regional computer facilities must include very large disk configurations. In turn, to make efficient use of this expensive investment and justify the personnel costs which will be necessary to manage the DST data and associated calibration files, substantial CPU capacity must also be available. Because of the practical management difficulties involved, an individual collaboration will be able to organize its large--scale computing at only a limited number of physical sites. We conclude that a centre which will make a significant contribution to several experiments must provide minimum resources of 100 Gigabytes of disk space, an automatic cartridge tape loader with a capacity of 5K tapes, and a CPU capacity of 20 units. [Footnote: This represents an investment of about 13.5 MSF (100 Gigabytes cost 3 MSF; a one third share in a tape loader costs 0.5 MSF; 20 units of IBM CPU power costs about 10 MSF at academic list price). ] Increased CPU capacity would make increasingly effective use of the disk space, and, if adequate network connections to CERN are also available, [Footnote: See a later section for discussion.] the centre could be a very attractive focal point for a substantial part of the analysis of an individual experiment.

We therefore suggest that the conventional batch computing capacity available to high energy physicists be concentrated in a number of large, adequately configured, regional or national centres, to provide a total processing capacity of at least 200 units.

Other Batch Capacity

Emulators and parallel processors

The experience with the 3081E and 370E emulators, and at Fermilab with the ACP, has already demonstrated that these systems can be used effectively in many cases where the program is relatively stable. In order to maximize the cost benefit of such systems, it is important that they are provided with an adequate level of operational support, including tape handling.

At the low--end of the CPU market, the cost of raw processor power is falling at an astonishing rate, which will continue to stimulate the development of cheap parallel processor systems. These range from straightforward assemblies of micro--computers which could be exploited in the same way as the existing emulator farms, to very specialized systems such as the Meiko Computing Surface.

A number of parallel processor "farms", established in universities, regional centres or in the CERN Computer Centre, well integrated with the institute's central computing services, to provide easy and cost--effective access to the main data storage facilities, could make a major contribution to the general computing capacity of HEP. Once established, these services would provide a framework which could later be very rapidly expanded in capacity, for relatively small additional cost.

We therefore make the following recommendations.

1. Commercially available parallel computers, or systems which can be assembled from commercially available processors, should be studied actively, as a coordinated effort between CERN and interested universities and regional computing centres. An immediate aim would be to identify commercial systems which could be used to provide parallel processing services in HEP computing centres. A longer term goal would be to do research in parallel computing relevant to high--energy physics needs.
2. During 1989/90 parallel processing farms, providing a minimum of 20 units of capacity each, should be installed in several of the larger regional centres, and at CERN. These should be integrated with the central services of the computing centres, to provide operations and software support, and access to the central data storage facilities.
3. The pilot service at CERN should provide about 25 units during 1990, rising to 150 units by the end of 1992.

Private batch capacity

We note that substantial batch processing capacity will exist in the form of private computing facilities, owned and operated by individual collaborations. Much of this will be in the form of emulators and other parallel processors, often purchased for a specific task associated with data collection but capable of being used for more general offline analysis at other times. These facilities are outside the scope of this section of the report.

Batch capacity in university physics departments

The aggregate capacity of the small computers installed in university physics departments will grow substantially over the next few years. Some fraction of this will be available for data analysis, depending on the ability of the physics departments to supply data to the computers and of the collaborations to organize the distribution of work to the many small units. The main contribution of these is likely to be as tools for a department's own staff, thereby enabling them to perform more of their work while at home. See also the discussion in the later section on interactive data analysis.

Workstations as batch processors

The number of personal workstations installed at CERN will grow dramatically over the next few years, and may reach a total of 1000 within four years, representing a nominal CPU capacity of several hundred CERN units. Since the workstations will be under the control of the individual users it is unreasonable to imagine that any sensible fraction of this capacity could be used by a central service. However, individual experiments should be encouraged to take advantage of spare capacity on their workstations for suitable tasks such as Monte Carlo.

Evolution of General--purpose Interactive Facilities

General--purpose interactive facilities include a program--development environment, text--processing and document handling facilities, electronic mail and database services, together with good connections to the main local and wide--area networks, to ensure that the user is able to control effectively his activities on central services at CERN and in regional centres. To an increasing extent, interactive services and facilities will be used for data analysis which, because of the rather special requirements for CPU power, data storage and network bandwidth, is discussed separately in the next section.

The following discussion refers specifically to CERN, but it applies also, with appropriate scaling, to regional centres.

Workstations at CERN

The benefit to physics analysis of a rapidly expanding use of personal workstations has already been discussed. In order to ensure that the evolution towards a personal workstation for every physicist and engineer at CERN may proceed as efficiently as possible, we make the following recommendations.

1. Centralized support for VAX/VMS and Apollo workstations should be strengthened, with the ultimate aim of making the purchase, installation and management of a personal workstation as easy as using the central VMS service.
2. The bandwidth, reliability and facilities available between workstations and the central services should be improved substantially. Planning should aim to provide an instantaneous 30 kilobytes/second for file transfer between an individual workstation used for general--purpose work (see the next section for the needs of workstations used for physics analysis) and the central IBM service, adequate capacity being installed to meet the aggregate demand at peak times. [Footnote: Note that this implies high--priority CPU capacity as well as cables and communications controllers.] The central VAX/VMS service will form an important focal point for users of VAXstations, and sufficient capacity should be provided to enable them to take advantage of this facility for file storage and other services.
3. The investment in the networking infrastructure of the laboratory should take account of the projected growth in the use of workstations (see discussion in the section on interactive data analysis).

The Future of Central Time--sharing Services

While it is clear that there will be a steady and rapid growth in the use of personal workstations, the life of the central time--sharing service is far from over. As we have noted above, workstations are today cost--effective only for intensive users. The demand for the central VM and VMS services continues to grow, and shows no sign of diminishing. We expect that the move of users from the central services to workstations over the next two years will be more than balanced by the increase in the number of new users of the central services, and that the demand at peak times will double during the same period, which will see the growth in the management information and engineering services, and the startup of LEP. Thereafter, we expect to see the rate of increase diminish, and demand eventually tail off.

The central interactive services are at present provided by the VM service (at the level of about 20% of the installed IBM--compatible capacity), and by the VMS service (which has been installed specifically for this purpose). The popularity of both of these services indicates that they are effective and complementary, and this and the advantages of a competitive environment enumerated above lead us to recommend that they should both be maintained and developed. The current load problems of the VM service will be largely resolved with the introduction of the VM/XA/SP operating system which will enable the service to expand to take advantage of additional processors and memory which are already installed. It is assumed that the VM interactive service will continue to use up to 20% of the installed IBM--compatible capacity, and that no specific recommendation for expansion is required.

The VMS service is, however, badly overloaded at present, and an effective increase of around 60% in its capacity is required merely in order to provide an acceptable service. This should be installed without delay. In addition, the capacity of the service should be doubled before the end of 1989. The total cost of the initial upgrade, including a modest 10 Gigabytes of disk space will be 1.4 MSF. The cost of the subsequent doubling in capacity, replacement of obsolete equipment and growth of the disk capacity to a total of 100 Gigabytes, is likely to cost about 7 MSF at standard prices, over two years.

The growing use of personal workstations implies a growing demand for centralized file and CPU server facilities (increased productivity will lead to increased consumption of resources as well as increased output). The VM and VMS central time--sharing services are the natural vehicles for such support, and as users move towards workstations they will in any case go through a phase where their working environment is split between the central service and the workstation. We can be sure that any new investments in the central VM and VMS services will be of long term value for workstation support, even if the time--sharing demand drops faster than we predict.

Interactive Data Analysis

The use of more or less powerful workstations for interactive data analysis gives rise to the following requirements, which are in addition to the general needs of workstations, discussed in the previous section. These requirements are stated in terms of CERN, but it is clear that they can be extrapolated to apply to the larger national and regional centres used by HEP.

1. Access to Physics Data

   All current models for LEP data analysis assume that the master "data summary tape" (DST) will be stored on cartridge tapes managed by the central VM service, with successive levels of subsets (team and personal DSTs) stored on disks on the same system. In some cases it may be possible to pre--select a relatively small subset of the personal DST (say 5--10 Megabytes) and transfer that to the workstation for processing during a working session of a few hours. This mode of working can be accommodated within the relatively modest data rates required for basic workstation support. However, in cases where this pre--selection is not practical all of the external data consumed by the analysis program must come from the IBM file server. This gives rise to a requirement for very much greater data rates than can be achieved by any existing product. The minimum requirement is binary file access at an aggregate data rate of 500 kilobytes/second per cluster of 10 workstations of the VAXstation 2000 or Apollo DN3000 class, rising to 1 Megabyte/second by the end of 1989. The demand will continue to rise faster than the technology available.

2. Very High Performance Networking

   The processing power of the workstations will continue to grow at a rate far outstripping developments in deliverable network bandwidth. It will be necessary to envisage the installation of the highest performance network facilities available, specifically for the support of the data analysis workstations. It is important that the development of distributed physics computing is not restricted by the considerations of a general--purpose network infrastructure. It is likely that the coming 100 Megabit/second FDDI (Fibre Distributed Data Interface) standard will be an appropriate choice in the medium term.

3. Co--operative Processing

   In order to benefit from the different characteristics of central mainframes and workstations, and to minimize the effects of practical constraints such as insufficient network bandwidth, considerable investment should be made in the development of co--operative processing techniques. Already the Physics Analysis Workstation (PAW) [Footnote: PAW - Physics Analysis Workstation, Comp.Phys.Comm. 45, 1987; The PAW Users' Guide, CERN Program Library Writeup Q121.] developments permit the separation of the user interface (on a workstation) and the Zebra data server (on a mainframe). Future plans envisage the extension of these techniques to provide "object servers" which may be distributed with some freedom between the workstation and specialized central services such as the Cray (for CPU--intensive tasks) or the IBM (for data storage manipulation). The effects of this style of working on the scheduling of the central resources may be profound, but the problems raised must be tackled with vigour both by COCOTIME and by the technical staff involved. Co--operative processing will become a major factor in the analysis of LEP data.

4. Distributed DST File Servers

   We have seen that the centralization of DST data poses a major problem for distributed computing. It may be necessary to consider the use of distributed file servers, in order to relieve the bottleneck of the central IBM file server. We must note, however, that the task of handling the DST data for LEP requires considerable CPU capacity [Footnote: The MUSCLE Report estimated that an IBM 3090--600E will be required to perform this task for the four LEP experiments in 1991.] , and so a distributed file server will be itself a substantial system, resembling more a mainframe than a workstation. The decision to install distributed file servers is largely a financial one, and at present the economics favour IBM--compatible systems, particularly due to the price of disk storage. In the immediate future, therefore, we recommend that the task of providing a central file server facility should be the responsibility of the IBM/VM service.

   It is important that collaborations include estimates of their needs for such facilities in their computing requests, in order to ensure that appropriate funding and planning may be organized. Note that the figure of 20% of the IBM service which was allocated above to general--purpose distributed computing support does not, by a very substantial amount, include the costs of support for interactive data analysis. It is assumed that this is included in the quoted batch CPU requirement of 160 units.

5. Costs and Staffing

   There are at present about 100 Apollo and 100 VAX workstations installed at CERN. We have been experiencing exponential growth, but it is expected that the rate of increase will settle down to about 200 new systems per year, leading to a ceiling of some 1000 stations. The useful life--time of a station might be expected to rise from about three years today to five years by 1992. We expect about 10% of the stations to be "high--end" systems costing an average of 80 kSF each, while the others will cost on average 30 kSF each, giving a total annual investment of 4.3 MSF. Central support costs, including special servers, gateways, software licences, etc. will bring the total annual cost to around 5 MSF. Maintenance charges for these systems are at present very high, amounting to about 25% of the (heavily discounted) purchase price per annum.

   The most efficient way to staff workstation support activities will be to provide a well--managed and strongly coordinated central service, rather than using large numbers of part--time people with a variety of employers, interests and priorities. Estimates for central staffing are as follows, assuming that half of the workstations are Unix--based, and the other half use VMS.

   1. Management and basic systems support: 5 engineers;
   2. Development of distributed computing support: 3 engineers;
   3. User support: 2 engineers;
   4. Systems management and operation: 1 technician for each 50 nodes.
6. Workstations in Regional Centres and Universities

   As the use of personal workstations for interactive data analysis evolves, they will become a requirement at all institutes participating in the analysis. Large centres with substantial disk capacity will therefore require to provide services to support these workstations and integrate them with their central processing and file server facilities. The larger institutes and CERN should also provide software and systems management support services for the smaller institutes.

   Once distributed co--operative processing is established as the normal way of computing for offline physics at CERN and in the major regional centres, there will be a strong demand for improved integration between these centres and CERN, and with smaller institutes and individual physics departments. We should be aiming for the development of distributed data analysis on a European--wide basis, with the individual physicist able to work effectively from his personal workstation, using databases and DST files located at a number of different institutes. This will require not only a high level of coordination between the main computing centres, but also high--performance reliable networks. The requirements for the latter are covered in the following section.

7. Data Representation

   The distributed computing environment which we envisage will highlight the inefficiencies of using different floating point number representations on the various computers involved. Although most workstations now use the IEEE standard format (with the exception of VAXstations), each of the mainframe types commonly used within high energy physics uses its private format.

Networking Implications

This report 'Computing at CERN in the 1990s' includes a part dedicated to general questions of networking. This section is therefore restricted to a brief summary of the requirements of centrally organized computing.

On--Site Networking Infrastructure

Short term evolution

The on--site Ethernet Service must continue to grow and be improved in a number of areas.

1. The planning, installation, management, monitoring and maintenance of the Ethernet infrastructure must be provided as a central service.
2. Installation planning must take account of workstations, and their requirements for connections to central services. Ample capacity must be installed in areas where the growth in the use of workstations by physicists and engineers will take place.
3. The reliability and availability of network services is crucial to the success of distributed computing. The central operation must ensure that major high--level services are monitored and appropriate corrective action initiated when necessary. It should be noted that the complexity of networks and network services is such that problem determination requires highly skilled staff and sophisticated network management systems.

The Domain token ring infrastructure for Apollo workstations must be maintained, with appropriate bridges to the Ethernet infrastructure and central services.

Medium term

The medium term (the next three years) will see the migration towards 100 Megabit/second FDDI token ring, initially as a "backbone" interconnecting the Ethernet and Domain infrastructures and the central services. An early priority in this development, which may well dictate the topology of the initial network, must be to ensure high bandwidth connectivity of workstations used for physics data analysis and the central IBM/VM service. It should be assumed that within three years, most new physics analysis workstations will be of sufficient power to require connection directly to the FDDI network. [Footnote: By that time, the cost of an FDDI connection should be comparable with that of an Ethernet connection, making it the network of choice if the cable is available.]

We should also expect to see FDDI coming into use to connect remote peripherals to the central mainframes (e.g. magnetic tape units installed in experimental areas), and as a means of interconnecting mainframes from different manufacturers within the Computer Centre.

In preparation for the longer term, CERN should acquire experience with the 100 Megabytes/second High Speed Channel (HSC), [Footnote: ANSI Committee X3 T9.3.] which will initially become available as a means of connecting very high performance graphics workstations to supercomputers. It should also, however, develop as a standard inter--processor channel suitable for the Computer Centre.

Wide--Area Networking

Short term evolution

The EARN and HEP-- DECnet networks which have developed over the past few years with emphasis on connectivity and guaranteed delivery provide good general--purpose facilities suitable for remote job entry and mail, and for the transfer of small to medium files. The European--USA links provided are very important features of these services. Both DECnet and the HEP X.25 network also allow elementary remote login. The ability of EARN to increase its services to include full--screen remote login and inter--process communication facilities will be major factors in its continued success, as will the improvement of the capacity planning for its backbone network. In the event that a pan--European network for research and development emerges from the Eureka COSINE project, an adiabatic transition from the above services, including transatlantic gateways, would be vital.

Effective high--bandwidth connections between CERN and the major national and regional centres are required by the end of 1989, in order to enable the efficient management of physics production and data analysis work at these centres and at CERN. These links will provide database access and file transfer (of calibration data and programs, and perhaps small samples of physics data), in addition to remote job entry and other facilities. Since much of the communication would be automated, such as direct access from programs to controlling databases at CERN, reliability is vital. Efficiently operated 2 Megabit/second links will be required at least between CERN and Aachen, RAL, IN2P3, Saclay, and an Italian site (Bologna?) and in due course also CIEMAT, DESY, NIKHEF and the USA. Experience with such links should be obtained as soon as possible. It is likely that neither HEP, nor the wider European research community would benefit from a completely shared network for all activities. The high traffic rate needed by HEP for distributed data analysis is inappropriate for general purpose networks. However, low--level multiplexing of HEP and general purpose traffic over the same fibres does seem a promising approach.

Medium term

The use of personal workstations will extend the user's normal computing environment to his office desk, in a form which he will be able to duplicate conveniently in his home institute and at CERN. The degree of integration with centralized file and CPU servers, which the CERN or regional computing centre offers him locally via LANs, will become an increasingly important factor in his productivity, giving rise to a demand for general--purpose communication links between CERN and the major computing centres in the member states, supporting 10--100 Megabit/second data rates. The current proposed experiment to connect CERN to several German laboratories using 10 Megabit/second channels on a 140 Megabit/second PTT fibre will give useful experience in this area and should be actively supported by CERN.

Personal workstations, and the development of distributed computing which they will stimulate, offer the first real opportunity for physicists to work effectively away from the CERN Computer Centre; the real cost of high--performance networking is now sufficiently low to enable them to do this from their office at CERN, or from their desk in their home institute; HEP must ensure that the tariff policy of the European PTT monopolies does not prevent European physicists from taking advantage of this situation.

Data Storage

Mass Storage Evolution in the CERN and Regional Computer Centres

This section summarizes the requirements described earlier in this report.

1. Growth from 200 Gigabytes at CERN today to 1000 Gigabytes by the end of 1991, implying a growth rate of 300 Gigabytes/year at a cost of 11 MSF [Footnote: Assumes 200 Gigabytes of IBM--compatible storage at 30 kSF per Gigabyte, and 100 Gigabytes of other (VAX, Cray) storage at 50 kSF per Gigabyte.] per year.
2. Automatic cartridge loader at CERN capable of handling 200 mounts per hour by the end of 1991. The system should be able to handle cartridge tape drives connected to Cray, IBM and DEC, using a single cartridge store.
3. Minimum of 100 Gigabytes at any regional or national centre playing a major role in the processing of any large experiment.

Distributing the Data

On--site distribution

The DST data required for LEP physics analysis, as estimated in the MUSCLE Report, poses two major problems for storage and distribution.

1. There will be a great deal of it: a minimum of 800 Gigabytes for each experiment by the end of 1991 (some estimates are a factor of four higher). On that timescale, economics will dictate that all but 10--20% of this data must be held on cartridge tapes, but even then the tape mounting rate will be very high.
2. The computing power required to maintain the master DSTs and prepare the subset DSTs for subsequent analysis will be twice the computing load needed for the final physics analysis, requiring the power of an IBM 3090--600E for the fraction of the work done at CERN.

Thus, managing the LEP DSTs will require a lot of disk space, serviced by a high--capacity cartridge tape robot, and supported by powerful CPUs. Economic (the cost of disk space and CPU power, and the possibility of sharing robots and tape drives) and practical (ease of management and operation) arguments will favour a centralized solution, at least for the first phase of LEP. The principal drawback to such a solution is the difficulty in achieving sufficient network bandwidth to feed the data to the workstations where the final processing will take place. This is mainly due to the current relatively poor throughput of heterogeneous network connections to IBM, rather than to any fundamental weakness in the concept of a centralized file server. [Footnote: DEC LAVCs achieve sustained aggregate network data rates of around 250 kilobytes/second, more than twice that available with a single Ethernet connection to the IBM. Even higher rates can be achieved between some 80386 based PC--s. This is due to the use of light--weight protocols and efficient implementations of them. While the former may not be easily applicable in a heterogeneous situation (where standard protocols must be used), we can expect to see the development of more efficient implementations, and the use of "protocol engines" in the hardware of the communications interfaces, in a 2--4 year timescale. In the meantime, it seems reasonable to connect multiple Ethernets to a central file server, using several physical controllers, in order to achieve higher aggregate throughput.]

We must study carefully early experience with LEP data analysis using workstations, in order to understand better the issues of data distribution. In the meantime, strategies for file servers should remain flexible and be reviewed regularly in order to be able to benefit from changes in storage costs and technology.

Distribution of DST data to external institutes

During the next five years, improved wide--area networking will enable individual events, or even occasional complete tapes, to be transferred between CERN and external institutes with some convenience, but until optical fibre bandwidths become available, almost all of the DST data will be distributed on cartridge tapes. In order to assist in this massive operation the following central services should be provided.

1. Tape copying service, including the allocation and labelling of the new cartridges.
2. Direct mailing service, whereby a cartridge can be allocated, labelled, written and then packaged and sent by express carrier to an external institute.
3. Centralized cartridge tape database holding details of tape history and location, implemented using Oracle and available to collaborations locally or remotely, for inclusion in their book--keeping systems.

The cost of tapes and freight should be charged to the requestor of the data, probably through a CERN team account.

Storage Management

This report has already described the need for a data management for the whole CERN community. This problem is too complex, and depends too greatly on the situation at regional centres and universities, for a technical solution to be proposed here. Technical solutions, almost certainly starting as small scale trials rather than global services, should be pursued by a committee reporting to the management of European HEP. A more specific recommendations can be made about a more limited goal. The need for automatic disk--to--tape migration of user files and experiments' DSTs has been described in section . This facility must be made available on the central IBM system at CERN by early 1990 at the latest. If suitable commercial software will not become available in time, it should be written at CERN or, possibly, be commissioned from a competent software house.

EVOLUTION OF CENTRALLY ORGANISED SOFTWARE SUPPORT AND DEVELOPMENT

In this chapter we address the questions of software support. We concentrate on system--related software and on general questions of software support. The following chapters will go into more detail on specific areas where software support of an application--related nature is seen as an existing or potential problem.

The Changing Landscape

The distributed computing environment described in this report could revolutionize physics analysis. However, the increasing complexity of the hardware infrastructure must not be allowed to produce increasing complexity for the user. Every effort must be made to render the appearance of the operating system, and of the data storage, as location--independent as possible. At the same time as hardware complexity is increasing, so is the complexity of HEP software. The software for any current large experiment is well beyond the point at which any single person could understand all the details. A later chapter will describe the promise of software engineering techniques for the design of HEP software. In addition, it seems inevitable that coding will eventually have to be done at a far higher level of abstraction than that of current 'high--level' languages such as Fortran. The creation of an HEP--specific 'even--higher--level' language is an option meriting further study.

Operating Systems

An environment in which physicists must use only one viable, but unexiting, operating system may well be more productive than one in which they must use several better, but different, operating systems. This is the context within which HEP should give serious consideration to an increased use of Unix. For all workstations except those from DEC, Unix is now the default operating system, and on many workstations there is no other possibility. At CERN the CRAY runs under Unix, and VAXes and VAXstations could (but usually don't) run under Unix. However, until now, Unix has not been available for the IBM systems which are the centre of data--handling and computing for most experiments. With the announcement that AIX, (IBM--Unix) will become available for IBM mainframes during 1989, it becomes possible for physicists to contemplate using only one operating system. The "Open Systems Foundation" initiative, involving Apollo, DEC and IBM among others, offers the hope that the different implementations of Unix will really appear very similar to users. It would certainly be premature to recommend that HEP convert rapidly and totally to Unix. However, it would be unrealistic to recommend that no Unix service should be offered on central IBM systems until its success was assured. Few physicists have any experience of being able to do all the analysis of a large experiment within one operating system, and an abstract study of its advantages would almost certainly be judged inadequate. It is recommended that an AIX service be started on the CERN IBM system as soon as the AIX specification becomes adequate to meet HEP needs. This service should be started on a small scale and its expansion should be driven by user demand. Some flexibility will be necessary in assessing whether AIX has reached an adequate state of development. It is not recommended that CERN write batch and tape handling systems to provide essential facilities missing from early versions of AIX, but smaller deficiencies might be repaired by CERN effort. This recommendation implies that some new manpower must be found, or that there will be some degradation in the support for the existing CERN services. It should also be noted that, even though great benefits are expected from the use of a single operating system, exchange of data between systems will still be hindered by their different data representations, except between systems using the IEEE/ASCII form

Languages

The working group has not attempted an evaluation of alternatives to Fortran, or even of the impact of the likely evolution of Fortran. For all its inelegance, and lack of safety features, it seems certain that Fortran will remain the main language for HEP code well into the 1990s. No existing or planned language, including, of course, Fortran, provides machine independent tape or network I/O. HEP has overcome this problem with utilities such as the ZEBRA--FZ package, which are effective, although naturally less convenient than if they were integral parts of the language. Fortran is notoriously lacking in data--structure concepts. Many more modern languages have some support for data--structures, although it is not clear that they could support the complexity required for handling HEP data. Fortran at least allows the freedom to provide sophisticated data--structure support through additional packages such as ZEBRA--MZ. The ensemble of HEP packages which run on top of Fortran, together with Fortran itself, form a powerful environment for HEP data analysis. It would not be easy to guarantee that a move to another language would improve the overall environment. Most physicists do not have the time or background knowledge to perform a thorough review of languages before starting an experiment. The CERN or HEP community should periodically review developments in languages, both to add weight to their input to Fortran standardization committees, and to identify any serious alternatives to Fortran.

Software Support for Experiments and Packages/Libraries

Support for Experiments

Ten years ago, most experiments at CERN expected and received support from CERN, or a major laboratory, to write the experiment--specific software. This help normally took the form of programmers assigned to the experiment, and was especially important for real--time programming and data--acquisition. Today, CERN and the major laboratories are not the sole repositories of computing expertise, and the case for this type of software support has weakened. However, there remain some good arguments for the provision of some experiment--specific support. Even in the largest experiments continuity is a major problem; the efficiency of the software effort can be greatly enhanced if at least some people can work full--time for many years. It is very difficult for universities to provide these people, especially if they must work largely at CERN. In the case of smaller experiments with shorter timescales, it is inefficient to require that, for example, university physicists become experts in a standard data--acquisition system which they will only use for a few months. The current level of experiment--specific support is widely criticised by physicists. This criticism may reflect the unclear (or non--existant) algorithms used to allocate support personnel, and the lack of coordination between CERN and other centres, rather than truly inadequate resources. CERN, and the regional centres and laboratories which can also provide such support, should review urgently how support is organized, and how this organization should be improved.

Packages and Libraries

Central support for widely usable packages and libraries has been extremely successful for many years. The CERN libraries, and packages like GEANT3, HBOOK and ZEBRA are used not only within the CERN community, but also for many HEP experiments worldwide. A relatively small centralized effort has prevented much wasteful duplication, and has made high quality software available to experiments which could not afford to write it themselves. It is very unlikely that a similar success could have been achieved merely by hoping for cooperation between experiments. In spite of this evident success, there are still some problems:

• Existing central support is overstretched;
  • widely used products are sometimes not sufficiently tested and documented before release,
  • there is little or no room for additional projects.
• Although informal relations with users have been mainly excellent, there is no formal mechanism for making requests and assigning work. As a result, some users, particularly in smaller experiments, feel the system is unfair, and that only the needs of large and vociferous experiments will be considered.
• As with experiment--specific support, there is no mechanism for coordinating these activities on a European scale.

An additional, and even more complex problem, is the determination of which software should be bought commercially. There can be no doubt that widely available commercial software, meeting the needs of HEP, should not be duplicated. The success of less mature commercial products, where HEP is, perhaps, a major fraction of the market, is much less clear, and existing experience does not make it easy to formulate a simple policy about what should be bought and what should be written. These decisions should be made on a case--by--case basis by a technically competent body representing the CERN community. The existing coordination problems, particularly in relation to CERN and major centres, make it impossible to say whether manpower levels are grossly insufficient or not. It is recommended that a software support committee be set up by European HEP with the brief to:

1. Examine support manpower at CERN and at other European laboratories and centres. Assess whether manpower levels are adequate to provide the coordinated software support needed to minimize wasteful duplication.
2. Coordinate support activities among all laboratories and centres willing to participate. This implies that software support personnel be instructed by their local laboratory to perform at least some of their work under the direction of the software support committee.
3. Advise, case--by--case, on the relative merits of buying commercial software as opposed to writing packages within HEP. This advice should take into account the extent to which commercial products meet HEP needs, the support which the HEP community would have to give to either HEP--written or commercial products, and the purely financial implications for (at least) the whole of the CERN community.

The committee should not limit its consideration to simple extrapolations from existing packages and libraries. The distributed computing environment, and the increasing complexity of HEP software make it necessary to explore new areas for HEP software support. Examples of such areas are support for a location and device independent user--interface to data, and possible HEP--specific very high level languages which could simplify the coding of data processing and physics analysis.

SOFTWARE SUPPORT: DATA BASE SYSTEMS

The questions specific to data base systems have been looked at by a separate group, with the following guidelines:

• Emphasis should be put on the user needs, and thus most contacts have been with users, rather than designers of data bases.
• Adequate commercial data base systems are to be given preference over homemade developments, wherever available. Limits do, however, exist due to specific requirements of users like access speed or simplicity of use.
• Due to the specific structure of our collaborations, the transfer of data base information between institutes is a vital constraint as important as portability of code and event data.

Our group has used as means of contact with different groups a widely circulated questionnaire, followed by detailed contacts with selected users, and all our conclusions are based on this input. L.Adiels, F.Bruyant, G.Gopal, A.Osborne, P.Palazzi, E.Rimmer, and M.Sendall have contributed substantially.

We have observed a large amount of duplication of work in this area. We therefore present clear recommendations in an attempt to limit the development of data base applications to a small number of acceptable solutions. We are conscious of the fact that the present recommendations still need further discussions with both users and the designers of data bases, before final solutions can be adopted.

Whenever specific products (e.g. Oracle or ADAMO) are mentioned in the recommendations this is not to be taken literally. They have been chosen since they are used at present and define the minimum requirement we have for any future product. We are fully aware of the rapid development in this field.

In this report, the word "Data base" is used in a broad sense and matches with the definition given in the ECFA report: [Footnote: Databases and bookkeeping for HEP experiments, ECFA Working Group 11, ECFA/83/78 (Sept 83).] " "A collection of stored operational data used by the application system of some particular enterprise." "

In principle a data base need be no more than a set of computer--based records, but the term is generally used to signify that the data are stored in separate files various types of record different purposes is possible.

Data Base Applications in Present Experiments

Addresses of Collaboration Members

The purpose of this information is to keep the name, address, telephone number both at CERN and in the institute for administrative purposes. The integration of this information with electronic mail addresses has been made by a few experiments.

Among the solutions adopted, some experiments have no mailing list, others use card image files with command files to use them, later experiments like Aleph and Delphi use the NADIR system. NADIR makes use of the Oracle relational data base system to store experiment members names, addresses, telephone numbers and electronic mail addresses. Interactive updates and queries are performed via Oracle full--screen forms. Various ADAMO tools are used to extract the Oracle information into portable files which are then exported to member institutes which do not have the Oracle system. ADAMO--based tools are then used to reconstruct and access local direct--access files and include an electronic mail distribution system called AMSEND. One experiment uses HYPERCARD on a MacIntosh.

Most experiments express their satisfaction with the adopted solution, even card image files when the number of participants is sufficiently small. It should be noted, however, that these data are purely for internal use. A single data base collecting all CERN users could be beneficial to all. But the latter question was not raised in the questionnaire and is thus the interpretation of our group only.

Electronic Mail Addresses

The aim is to allow to send mail electronically to a user's family name, or to lists of people, without the need to know his account(s). This mail is automatically converted to his preferred mail address and optimally routed. This tool is heavily used to inform the members of a collaboration rapidly and easily.

The presently most frequently used method to retrieve electronic mail addresses seems to be NAMES under VM. Two LEP experiments (Aleph and Delphi) are using information stored via NADIR and used in AMSEND, developed in Aleph. NADIR stores, as well as geographical addresses, the collaboration members home computer, his login id(s), networks and mail servers. Other experiments rely on simple files. All experiments found their system adequate and preferred that the updating of both the mailing lists and the electronic mail addresses be made directly by the experiments (e.g. by the group secretaries).

The DD division has set up an Oracle based system called EMDIR (electronic mail directory) where each user is invited to enter his preferred electronic mail address. This system is unpopular (it has a poor user interface), many users are not keeping it up--to--date and because of this it cannot be relied upon by collaborations. None of the experiments which replied mentioned that they are using it, although the authors report "thousands of queries per month". They intend this to be "the basis of future mail systems".

In summary, as is the case for the geographical mailing lists, the information is used exclusively within the experiment itself.

The DD division has recently implemented an Oracle data base of the login identifiers on the central systems of all registered central computer users and this data base includes the CERN telephone directory. This perhaps could become one part of a common geographical and electronic mail addressing system for all CERN experiments where the experiments themselves would furnish the other information about their collaboration members.

Experiment Bookkeeping

The purpose of bookkeeping is to keep track of data taking runs, experimental data, storage media (disks, tapes) and program versions used in the different phases of event reconstruction and analysis.

Amongst the current experiments, several use simple systems, like card images, others use random access files with KAPACK or the Oracle RDBMS and some experiments have just started with Oracle or will start soon and are still undecided. Opal intends to use Oracle to hold "log--book" information. It seems that this is a place where a common effort could lead to a general solution.

On--line Data Bases

There is no single set of data base tools today to span the whole range of on--line applications such as storing detector constants and calibration data, histogramming, error logging and run parameter logging. This has led to a large variety of individual solutions ranging from simple files to KAPACK based applications or even the usage of a RDBMS such as Oracle. The CP--LEAR experiment will use Oracle online on a VAX but only as an indexing system pointing to other logical and physical entities (histograms, tables and files).

Description of the Detector Geometry

The aim is to provide a detailed geometrical description of the detector components and their evolution in time. It is needed in the off--line programs for the simulation and the reconstruction of the events and for graphics display. The system must be able to cope with a simple description, as used in reconstruction, and a very detailed one, as used in the simulation. A fast access time is crucial, given that the simulation or reconstruction will frequently access it to get answers of the type "where am I" and "how far can I go".

Again, a wealth of homemade solutions have been developed. Some experiments use random access files based on ZBOOK, KAPACK or ZEBRA. The manpower investment in this area is very large (several man--years per experiment, on average). The main difficulty resides in the extremely fast access required. This explains why different experiments have tried to optimize the data storage to their needs by using different solutions. It also excludes the direct usage of Oracle for the data access at execution time.

The need for a DBMS to provide fast access to Detector Description data was identified very early on in Delphi. At first (1984) they developed a system based on KAPACK, tailored specifically to this particular application. So long as the originally designed Data_Structure was adhered to the system worked very well. However, Delphi were not happy to freeze the design of the Detector Description Data_Structure so far ahead of data--taking.

So, a generalized DataBase Management System - CARGO, also based on KAPACK, was developed, care being taken not to tailor it to the needs of any specific application. The Delphi Detector Description - Geometry Data at first (1986), Calibration Data later (1988) - has been successfully structured according to the conventions of CARGO. Fast access to this data for Simulation, Analysis and Graphics programs is provided via an application specific access package (DDAP) which structures the data requested by the User Program into a ZEBRA division.

The Geometry File contains the detailed Geometry for each of the 18 or so sub--detectors including the beam pipe, coil and the cable duct areas.

A first version of the calibration constants have now been installed in the calibration file for almost all the sub--detectors and programs are now beginning to use this data.

The L3 experiment has built a detector description data base using the ZEBRA (CERN data structure management package) direct--access package RZ. The RZ package allows a stored ZEBRA bank to be identified by up to 100 4 character long keys. L3 has written a Fortran data base package (called DBL3) on top of Zebra RZ to provide the user interface to their RZ data base. Their system stores changes to data as update records rather than changing base records and they also store pointers (in the form of file names) to their multiple RZ files. The user of DBL3 is normally returned a ZEBRA pointer to his requested data, the system having read it from disk into ZEBRA memory. DBL3 took about one man--year to build and they feel it will provide the functionality they need for many years. In addition they have built a package for interactive access to the data base using the KUIP (Kit for an User Interface) component of the PAW (Physics Analysis Workstation) system. Since PAW is itself based on ZEBRA and RZ this was a natural extension.

The Opal experiment will also use an RZ based system for detector description but make the general remark that parts of their data would be suitable for a relational data base.

A general solution for experiments needing the functionality of a commercial data base such as Oracle but having constraints of performance or portability may consist in a hybrid solution of maintaining the master information in Oracle and providing translate programs that generate normal Fortran files.

This is the solution adopted by the Aleph experiment who store their detector description constants in an Oracle data base including validity time--stamps. The data definition is made using data modelling with tools from the ADAMO package. The data are then extracted into RZ or BOS (the DESY data structure management system) banks which themselves can be stored in Fortran direct--access files. Each Oracle table becomes a BOS/RZ bank with the numbers of rows and columns stored as the first words of the bank (this amounts to extracting a single "view" from the relational data base). The Oracle data base contains data from the different detector components (including the beam pipe, cables etc.) and updating is done by 1--2 people for each of them.

Calibration Constants

The purpose of this package is to store the calibration constants and to keep track of their changes as function of time.

Similar solutions are adopted as for the detector geometry, since the two need to be linked together, the main difference being that the volume of calibration data is normally much larger than that for detector description. Many of the same comments apply as for the previous section. Opal will use RZ direct--access files, while Aleph will include in their Oracle data base time stamps and pointers to their calibration data (e.g. file names or tape numbers).

Event Data (or Memory Management Systems)

All the answers received so far were related to the management of the event data in memory. All existing memory management packages are mentioned (BOS, HYDRA, ZBOOK, ZEBRA), though most of the new experiments tend to standardize on ZEBRA. None of these applications involve a real data base. In particular no experiment mentioned the need of a data base for direct access of event data. Opal envisages an event server which can recall events from specific blocks on 3480 cartridge tapes and expressed a wish for a tape management system at the computer centre.

Bookkeeping of Program Versions

The intention was to find out whether people feel the need to keep track of program versions in time, in order to allow the reprocessing of events in the same conditions (and with the same bugs) as they were a long time ago. Also here, the question was not clear enough and all answers were related to source code management.

Histograms and other Physics Results

The purpose is to keep a catalogue of results/presentations for on-- and off--line. In the on--line context, this may be monitoring histograms, and in the off--line context, reconstruction results or physics data.

In the answers, a majority expresses support for PAW.

Software Documentation

A data base could keep the whole software documentation and the description of the data.

Several experiments use simple files. The only applications using data bases are from Aleph using the data model tools from ADAMO to provide software documentation (in addition to write--ups on paper).

Opal uses an internal system called OpalDOC but would merge this with the DD CERNDOC system when some bugs in it had been fixed and when it accepts TEX formatted files.

List of Publications

The purpose is to have an up--to--date list of all publications for the experiment. This should include the title, the list of authors, the reference number, the data and subject keywords.

Currently, the experiments use simple homemade solutions. They are satisfied with their solutions, but some wonder whether in the long term a more elaborate system may not be preferable, provided the access remain sufficiently simple. The Particle Data Group uses Oracle. The EP Electronics are using ISIS and find it hardly adequate.

Other DB Applications

A need expressed by experiments and mechanics groups is to have a data base application which can handle the registration of parts of the equipment and their life history. The Opal experiment uses a "4--th Dimension" data base on a MacIntosh for the inventory of their electronic modules and find this very convenient with the drawback of no easy access for the general members of the collaboration.

The DD Online Computing group use Oracle to maintain the pool of online equipment (several thousand items). This is now used by a wide range of people for arbitrary queries and has been extended to include experiment hardware configurations of equipment not owned by the group and also to include information on maintenance contracts. This runs on the IBM VM system and makes use of the Oracle report generator. They did not consider using a PC based system because of lack of connectivity with other central data bases. They feel there is a gap between the very competent technical support offered by the DD data base section and the end user community (indeed the User Support group feels they would need dedicated Oracle experts to do other than refer users to this section).

An interesting data base application is the one set up by the DD Fastbus System Management team which allows experiments to generate code which is used to initialize a Fastbus configuration (this contains full configuration and routing tables, arbitration levels and so on). Fastbus is a multi--level triggering system where the hardware is fully addressable and functionally defined. By using an Entity--Relationship model they showed the data describing a Fastbus configuration to be relational in nature. They thus use an Oracle data base on the central VAX cluster to store the basic descriptions of the different Fastbus modules and the relations between them. A given experiment uses a set of Oracle SQL forms (some tens of thousands of lines of SQL*FORMS containing many inter--table checks) to build the description of their particular configuration. A Pascal program then reads the resulting data base and generates the required configuration initialization code in an ordinary VAX file which can be down--loaded to the experiment concerned. Use of Oracle gave them the benefits of a backed--up data base with a high--level of integrity, possibilities of roll--back and multiple concurrent access.

Present DB Applications in Experiments: Conclusions

The most striking fact appearing in the answers from the experiments is the large variety of solutions adopted. Several experiments expressed their dissatisfaction about the chosen solution. Such a situation seems to have originated from the regrettable lack of general purpose data base applications software adapted to their problems. The design and development of such general purpose applications needs expertise with a given DBMS and manpower. A simple ad hoc solution is often "cheaper" for the individual experiments, but, summed over the experiments, it is certainly more manpower consuming than to provide once a general package. Some exceptions should be noted, for instance the NADIR system developed in the Aleph collaboration. It is also striking that this package was adopted by other collaborations, like Delphi, although it needed some "customizing" before it could be used. NADIR is a good example of an application very close to being a general solution for mailing lists, which is now being worked on by the MIS unit. This example shows that when a sufficiently general application is provided, the experiments will tend to use it, independent of the complication of the underlying DBMS.

A general observation is that experiments are using both commercial relational data bases and the traditional HEP data management packages, depending on the nature of the data concerned. Some translate the relational data base into a HEP package for reasons of portability.

Broad Areas for DB Applications and their Requirements

Several areas can be identified where the use of a DBMS is clearly justified. and is quite similar for the different experiments. Each of these areas has specific requirements and needs different application software for access and update by the users. Some applications already exist, most of them need, however, further development. Even where this may require a sizeable effort, overall harmonization seems to be the only way to avoid the present wasteful approach of multiple ad--hoc solutions, each of which is manpower--intensive.

We will first discuss the fields where a HEP--wide solution can easily be envisaged. For these applications a complete tool package should be written by some DB "professionals" in consultation with an advisory team" from different experiments.

This is then followed by a short discussion of data base applications where a general purpose package seems to be - at least at the moment - not in sight. We believe, however, that at least for the modelling phase and for the transportation of data bases general tools can and should be provided.

DB Applications where Generalized Packages are Possible

Experimental Administrative Database

As mentioned above this type of data base should contain all data necessary for the information exchange (mail AND electronics mail) between all members of a collaboration such as address, phone number, office, function, responsibility, affiliation, network addresses, etc. It will contain only few, well defined, tables. To avoid unnecessary duplications and work this data base should be coupled to already existing ones like: user registration, telephone directory and personal data.

All members of a collaboration need read access to (part of) the data base via panels and/or via coupling to high level language programs. However, since probably only reduced functionality is needed at some places the access can be realized on an extracted d/a file or even a sequential file. All other access might be (should be?) limited to the secretaries of each collaboration. At least part of the data base should be accessible HEP wide.

It would be desirable if each member of a collaboration could initiate the changes to the DB as far as his own data are concerned. In practice this would essentially mean that a panel system has to be provided which creates the necessary update corrections sent then to the Data Base Administrators (e.g. the group secretary). Since probably only reduced functionality is needed at some places the access can then be realized on an extracted d/a file or even a sequential file.

Since the requirements are almost the same for all experiments, a general data base package including all necessary tools for the communication with the data base and for the transport to/from external institutes should be provided by CERN. As a starting point the NADIR system could be considered.

This application can best be realized on top of a relational data base. If done so it will be quite easy to increase the functionality with time, whenever needed, without having to redesign the already existing parts of the data base.

Tools are needed for:

Electronics and material Bookkeeping

With this database we want to keep track of all electronics modules and materials used inside a collaboration. Therefore data such as module description, address of manufacturer, purchase information, inventory and location are entered.

The requirements are quite similar to the ones mentioned for the experimental administrative database. In addition coupling to a general data base describing the internal functionality of modules should be envisaged.

This data base will be accessed only by few members of a collaboration. Therefore it can be kept centrally at CERN with some access from outside institutes. The package itself should ,however, made available to all collaborating institutes to have their local bookkeeping system identical to the one used centrally.

As for the experiment directory this is an important area where a general product should be written and maintained.

Tape and Production Data Base

In a large experiment it is difficult to follow the history of a single event throughout the different phases in the off--line analysis. Therefore a database is needed which keeps track of files and program versions used for real and MC--events. In addition a database which contains information on storage resources (tapes, optical disks etc.) is needed.

This data base has a simple and well defined structure and could be realized on almost all DBMSs. What is important here is that read and write access is needed from terminals and from higher level programs (e.g. reconstruction programs) in real time. It has to be accessible - may be not with the same functionality - from all institutions of a collaboration.

As for the previous application areas the structure is well defined and a general package can easily be designed. If realized on top of a RDMS, experiment dependent additional information can easily be integrated.

DB Applications for which Tools could be Specified and Written

Detector Description

In a complete detector description database the geometrical description of the detector components, the description of the electronics path to each sensing device, the deficiencies, the alignment corrections and their variation with time have to be entered.

In general, this leads to a complicated structure with many different tables (100--1000). To cope with this complexity of the data structure and to allow for easy navigation through it a RDBMS is needed. In particular, there are basic records and, separately, update records which are merged with the basic records only at the time of access.

Access to the data base is needed by many different applications and packages in read and write mode. Access time is crucial, hence the data base is probably read in once and all access is made to a structure in memory as long as there are no changes in the time dependent constants.

In this complex system updates to the data base have to be done under the control of the Data Base Administrator of the collaboration.

The data base has to be available (with reduced functionality) on all computer systems at CERN and in the outside institutions.

Unless one introduces strong restrictions a general package cannot easily be provided. However, it would be desirable to model an example data base which could be used as a starting point for experiments which have a small number of (simple) detector components.

In addition to the tools already mentioned for the experiment directory, more and probably application dependent tools are needed, especially if the commercial DBMS cannot be accessed directly at all places.

Calibration Constants

Calibration constants used for (online) corrections of experimental data present generally a huge amount of unstructured data. Therefore, for most experiments there is no strong need to store them directly into a data base. However, the data have to be accessible via the database. It would be sufficient to enter the access information into the data base and keep the data on separate files outside (but nevertheless as part of) the data base especially since some coupling to the detector description data base is needed.

The access to the data base is almost exclusively done from higher level language programs.

Data Base Systems: Recommendations

General recommendations are presented first. They are followed by further recommendations specific to certain areas.

As mentioned at several occasions in this report, a large duplication of effort has gone in developing real or pseudo "data base applications" inside the experiments. It frequently happened that a simple ad--hoc approach was chosen, which in the long term proved to be too restrictive, once the volume of data increased or the data base was needed for applications it was not originally designed for. It also happened that commercial systems were tried and found too slow to be acceptable. The most frequent arguments used by opponents of data bases remain: -- Slowness of access, especially in the area of the detector description and calibration constants. This slowness now seems to have been partly overcome. -- High learning threshold, which is undoubtedly a strong argument in case of simple problems like lists of publications. This is, however, purely a question of user interface and it could be overcome by providing general purpose applications with a suitable (graphics?) interaction with the user. The intricacies of the underlying DBMS can be completely hidden to the user.

Specific applications should be written as part of the general support software. This should be the result of a concerted effort between the DD division and the users. Suggestions are given in the specific recommendations below.

Education and Training

Although DBMS have found a wide acceptance in the engineering world (e.g. LEP), there is still a reluctance for their use by physicists, who often prefer their "homemade" solution. It should however be realized that, in the future, experimental data are more likely to be managed by a DBMS than by our existing packages, which remain tightly linked to our prejudices on Fortran as the main language used for our programs. We recommend that: " An effort should be made to make physicists in experiments more aware of the potentialities of commercial DBMS for their applications. This could be achieved by intensifying training in the areas of data models (software engineering) and DBMS. "

Design Support Team

It should be avoided that the effort to write General Purpose or applications code be duplicated by the different collaborations, as is often the case today. To help overcoming the initial difficulties and the long term maintenance, we recommend that: " Manpower should be made available to support centrally the experiments, starting with the design of the data base and continuing during the whole life cycle, including the implementation of the application dependent code. This support team should also ensure the long term maintenance of the General Purpose applications described below. "

As the design of a data base application involves software methodologies and as software methodologies make implicit use of data bases, the same support team could cover both areas. This will help to speed up the implementation and to ensure that the resulting application can be used by several experiments.

Data Model Software

It should be realized that the usage of a DBMS for a specific application requires one to several man--years of work to design the structure of the data and to write the necessary SQL and/or SQL*FORMS code. Several areas described below will need such application code to be written. It has been stressed at several occasions that this task is greatly helped by the use of a formal Data Model. This immediately provides a link with the "Software engineering group", but we independently recommend: " A package should be provided to design interactively a Data Model and to store this definition in the form of a dictionary in ZEBRA files. The Entity--Relationship Model and related software from ADAMO should be considered as a first step in this direction. This would allow to profit from the experience gained and possibly from existing tools, including commercial ones. "

Portability of the Data Base Information

The Oracle Relational Data Base Management System has found a fairly general acceptance in the user community as the standard data base system. It is already being used and will be used even more in many different applications. Some features which were stressed by users at several occasions are its handling of concurrent write access, its user interface via SQL*FORMS and the possibility to implement tools on SQL to guarantee the data integrity. A serious problem with Oracle is its price, which prevents it from being used throughout the HEP community. As an indicative value, it costs approximately 25 kSF for a VAX780 and 10% per year for maintenance. Other tools have to be paid for in addition. A solution must be found which allows also the smaller institutes to use the data. " A package should be provided to map data from Oracle to a ZEBRA (RZ) data structure. The reverse could also be implemented, provided a data model describes the structure of the data in the DB. A decent user interface should be written on top of these files to allow the users to inquire about the information contained in this structure and to update it. Tools provided with the ADAMO package could be used to learn from the existing experience and could possibly be used directly as part of the proposed package. "

This approach guarantees that the information can be used and updated in the collaborating institutes free of charge.

Area of Experimental Administrative Data Bases

Several data base applications, all using Oracle, have been developed, like the User Registration DB, the EMDIR system, several mailing lists from experiments using NADIR and the AMSEND electronic mail system. Data are often duplicated and contradictory, because some are out of date. It is recommended that: " A data base should be set up covering all CERN (or HEP?) users and other people related to experiments. It should link with information from existing data bases. It should include the functionality required for experiment mailing lists and experiment specific data. Control of the data, i.e. entering and updating the information, should stay within the experiment concerned. We further recommend that a study be made on existing tools and their performance, in order to coordinate any future efforts, such as those being made around NADIR and EMDIR. The functionality should cover at least the one of the NADIR and AMSEND systems. "

Area of Bookkeeping Data Bases

LEP experiments seem not to have invested much manpower in this area yet and a common solution still seems possible. Therefore, we recommend: " A solution should be researched and developed urgently, in common between the LEP experiments, in the area of tape bookkeeping, to avoid duplication of effort. "

Area of Documentation Data Bases

The part of the software description related to program code and data description may come automatically with the Data Model tools and is considered as part of recommendation 3.

In addition, "paper documentation" will still be required and makes this an important area to be looked into. CERNDOC seems to have basically the right functionality, but it should be coupled to other data bases, like the CERN Library data base and experiments administrative data bases. It is also essential that the data can be transported and printed throughout the HEP community. We recommend that: " The redesign of existing documentation data bases (CERNDOC, HEPPI, ISIS etc.) into a common data base system (e.g. Oracle) should be envisaged. "

Area of Detector Description and Calibration Constants Data Bases

This is probably the area with the largest investment of manpower and the largest savings if a common solution could be found. What makes it particularly difficult is the large data volumes encountered and the high speed needed for accessing the data. It seems unlikely that the existing choices for the detector description and calibration constants data bases could be modified, given the imminence of the data taking for the experiments involved. It also seems difficult to provide a completely general solution, given the large differences between the detectors.

It is at least expected that the experiments will store the information in a DBMS and in ZEBRA files. The availability of software to handle a data model and to convert between a DBMS and ZEBRA could give some help. Some guidelines and assistance for structuring the data would also be helpful. The experience gained with the usage of the large data bases set up by the LEP (and other) experiments should be exploited to try to define further support tools and minimize duplication for post--LEP experiments. A study group should be set up around the end of 1990, after a sufficient experience is accumulated.

Area of Interactive Data Analysis Data Bases

PAW datasets are expected to play this role. PAW is already widely accepted by the user community, and no new software needs to be developed. The design of PAW explicitely does foresee do use PAW datasets as data base; the present PAW implementation, however, seems to be lacking the built--in possibilities for identifying the contents of the various directories. In--line identification seems necessary to make PAW data exchange inside and between experiments largely self--documenting.

Data Base Systems: References for detailed reading

• Databases and bookkeeping for HEP experiments, ECFA Working Group 11, ECFA/83/78 (Sept. 83).

• P. Chen, The entity--relationship model - Toward a unified view of data, ACM Transactions on Database Systems, Vol 1, N0 1, March 1976.
• S.M.Fisher, P.Palazzi, The entity--relationship model of ADAMO, Aleph report ADAMO Note 3, March 25, 1988.
• R.Mount, Database systems for HEP experiments, Computer Physics Communications 45 (1987) 299.
• A.Putzer, Data--base system in High Energy physics, Proceedings of CERN School of Computing Troia, September 1987 and Heidelberg University report HD--IHEP--88--02.
• Z.X.Qian, F.Blin, W.G.Moorhead, P.Palazzi, NADIR, the New Aleph DIRectory, Aleph report ADAMO Application Note 10, November 11, 1986.
• Z.Qian et al., Use of the ADAMO data management system within Aleph, Computer Physics Communications 45 (1987) 283.
• E.M.Rimmer, Programmed initialization of complex data collection and control systems , CERN--DD/88/6, March 88.

SOFTWARE SUPPORT: SOFTWARE ENGINEERING

This subject was considered by a separate working group. To define what is meant by software engineering: 'the application of engineering--like methods for large scale software production'. In this report we consider the methods and tools which are likely to become a necessity for such complex projects as will be experiments (and accelerator control) at a future hadron collider.

The major recent trend in HEP experiments and one that promises to continue, is that experiments have become very large, in terms of equipment, numbers of physicists and data volumes. Experiments, in addition, have lifetimes measured more in decades than years. Necessarily, the software, both on--line and off--line, required to analyse and control these projects has also grown in complexity and volume. The issues of how to build, manage and maintain large software projects are addressed by modern software engineering (SE) methods and techniques. Furthermore, SE methods will undoubtedly be essential to enable the integration of both hardware and software in future real--time systems.

The question of which programming language(s) should be used is not discussed by this working group. It should be stressed however, that interesting developments in such areas as automatic code generation and the like are not likely to be commercially provided for a Fortran environment with typical HEP memory managers and associated data structures.

We have no intention of recommending the precise tools and methods that should be used. This is a field which is rapidly evolving and it would be presumptuous to presuppose what might exist in 5 years time. We do, however, base many of our conclusions on the Structured Analysis/ Structured Design (SASD) methodology that, thanks to pioneering work by the Aleph collaboration, has gained a strong foothold in the HEP world, both at CERN and at other HEP laboratories.

In the following, we consider in some detail analysis and design methods and related tools which constitute the relatively recent major innovation in software engineering. We also discuss briefly the tools required at the coding and implementation level. This includes code management which, traditionally, has been the most widely used implementation tool.

Analysis and Design, Methods and Tools

Modern analysis and design methodologies provide systematic techniques for the analysis of a problem or project, together with design strategies for translation into software. The basic components of a typical method are:

• The notion of the life--cycle of software, or its existence between the inception and the end of its utilization. In analogy with a manufacturing process the following approximate steps exist ; specification of requirements, analysis of requirements, design, implementation, testing and maintenance.
• Abstraction or the concept of construction of a logical model without regard to physical implementation. This enables a clearer understanding of the problem at hand, followed by consideration of the practicalities of implementation at a later stage. Complex problems are broken down into subproblems and the latter subdivided until the resulting problems become intellectually manageable.
• Graphical methods to enable a thorough analysis and design of the project concurrently with the production of the documentation. Diagrammatical techniques, with the goal of partitioning the problem into manageable portions, are vastly superior to monolithic tomes of written prose.
• Data modelling techniques provide a clear understanding of the data, and may provide a path to physical data structures.
• A Data Dictionary consisting of specification of the data elements, their inter--relationships and descriptions. The data dictionary provides a central backbone of data definition and documentation for all the data throughout all phases of the software.
• Automated tools to enable manipulation, editing and verification of the various diagrams, specifications and data descriptions and their inter--relationships. Integration of all tools via a common data base is essential to be able to go back and forth between information relating to various phases of the project.
• Design criteria aimed at producing specifications for independent modules which may then be built by smaller teams. The final refinement of the design diagrams results in specifications not far removed from actual code, and indeed there are developments in the area of automatic code production.
• Management procedures to both ensure the quality and monitor the progress of the work in hand. At each stage of the life--cycle real deliverables enable a proper assessment of the project. Improved quality of the product, especially in the critical analysis and design phases, is a built in, rather than added on, feature.

In addition to the above, 'reverse' engineering tools are used to analyse code and produce diagrammatical descriptions in order to verify what the code really does. Given the addiction of HEP to Fortran and other concepts such as memory managers, these tools have, in general, to be home grown.

Historical Background

The term 'software crisis' was coined in the 1960s as recognition that large software and system development was, in many instances, out of control. The various studies made of large projects indicated that a major contributor to this problem was the lack of a systematic approach to software design and development. The concept of 'software engineering' emerged with the idea of applying engineering--like techniques to the various stages of the software life cycle. With the advent of significant new ideas on Structured Design and Structured Analysis [Footnote: Some literature: W.P. Stevens, G.J.Meyers, L.L. Constantine, Structured Design , IBM Systems Journal 1974; T. DeMarco, Structured Analysis and System Specification, Yourdon 1978; C. Gane, T. Sarson,Structured Systems Analysis, Prentice--Hall, 1979; G.J. Myers, Composite/Structured Design, Van Nostrand Reinhold, 1978; E. Yourdon, L.L. Constantine, Structured Design, Prentice--Hall, 1979. ] in the early 1970s together with the concept of the software life cycle, it became possible to formulate frameworks that incorporate the known techniques for software production and the appropriate management practices.

Situation in the HEP comuunity

Whilst the concepts of software engineering have been embraced in particular by the aerospace and oil industries over that past decade, the start of any significant involvement in the HEP world can be traced to 1984. The workshop on "Software in High Energy Physics" in 1982 provided the HEP community with a chance to meet and discuss with people who not only advocated but had actually used these techniques. Several groups, notably the Aleph software group, expressed further interest and, with the aid of the results of several comparative studies Aleph selected SASD as a suitable methodology. This selection was based on the criteria that SASD supported all phases of the life--cycle, was widely used in industry, was relatively easy to learn and use and was itself supported by (at that time rather limited) automated computer tools. This view was confirmed by a tutorial on software design techniques in 1983.

Subsequently courses in SASD were given at CERN and towards the end of 1984 SASD was formally adopted by Aleph for both the on--line and off--line software.

In the following summary of the existing situation, SA refers to the analysis part of SASD and SD to the design part.

• Aleph : SA is used extensively for off--line and on--line data acquisition software and SD is used for certain subsets. Many extensions have been made by Aleph, sometimes because the early versions of the tools did not support required features such as, for instance, data modelling for which the ADAMO toolset [Footnote: Reference: Qian Z. et al, Use of the ADAMO Data Management System within Aleph, Comp. Phys. Comm. 45 (1987), 283-298. ] was built. Support of the methodology and tools, including 'reverse' engineering tools such as program analysers (for example PRODES [Footnote: Reference: J. Bunn, PRODES, a tool to analyze and describe a Fortran Program , ADAMO Application Note 3. ] is done by the group.
• SPS Control System : SASD has been used to analyse the existing controls software and design modifications for the SPS to become a "multicyling" machine. This work [Footnote: Reference: R. Bailey, J. Ulander, I.Wilkie, Experience with using the SASD Methodology for Production ofr Accelerator Control Software, SPS/AOP/Note 87-16 ] comprises a major revision and upgrade (essentially a complete rewriting) of the large body of applications programs used for controlling the SPS.
• LEP Control system : Following the lead of Aleph and SPS Operations group, SASD techniques are used for creation of the applications programs to control LEP. The work is split into an analysis section (using SA), comprised of accelerator physicists and engineers, plus a design section (using SD) consisting of computer professionals.
• Fastbus system management : SASD methods have been used to make a system to automate the management and control of large Fastbus systems. This work [Footnote: Described in: E.M. Rimmer, A Database for Fastbus DD/87/23, Oct. 1987 ] has been a collaboration between DD, Aleph and Delphi and is relevant to any Fastbus setup.
• Opal : SA is used for the off--line reconstruction program, although, as noted below, there is no official commitment by the collaboration to use these techniques.
• OBELIX : SA is used for a pilot project, with the intent to use it for off--line and data acquisition software.

There is currently no official CERN support for software engineering analysis and design methods and tools.

Partly as a result of the Aleph experiences, the FNAL experiment D0 adopted the same methodology for its software development. Subsequently FNAL has negotiated a license for use of a modern commercial system based on SASD. The first users of this tool are D0 and the FNAL data acquisition group. Consultancy support is eventually foreseen.

Towards the end of 1986 and following into 1987 courses on SASD were given to experimenters from the two HERA experiments, the HERA machine group and the DESY computer centre. It was decided to adopt SASD for both on--line (ZEUS) and off--line (ZEUS and H1). In November 1987, after evaluation and comparison of existing products, a common analysis and design tool [Footnote: CADRE/Teamwork, by Cadre Technologies Inc, Providence, R.I. This tool has been selected by (amongst others) : FNAL, DESY for HERA, Cornell for CLEO and CERN for SPS controls. ] was selected, with licences negotiated by DESY and provided for the experimental groups. The ADAMO toolset is used.

Large Software Projects at CERN

In this chapter we attempt to summarize the experiences of large scale software production both with and without the use of software engineering methods. For the former we have looked at major experiments including Delphi, EMC, L3, Opal, UA1 and UA2. The software for the older experiments (EMC, UA1 and UA2) was written well before SE techniques became fashionable. Delphi and Opal use SE techniques to some degree, but the methods have not been officially adopted by the collaborations. In addition we reflect on the SPS controls software prior to 1985.

The conclusions on the use of SE techniques are based on the experiences of Aleph and, to a lesser extent those of ZEUS and H1 at HERA and D0 at FNAL, together with those of the SPS and LEP control systems. Experiences at the European Space Agency have provided specific insights into project management possibilities.

Software Projects that have not Used Formal SE Methods

The conclusions and experiences below are an average and not necessarily applicable to all projects.

Organizational Tools

Reliance is made on dynamic memory and data managers to organize data structures. The latter are in general well documented, both with text and (often) diagrams, and contain some elements of data dictionaries. Access to data banks is sometimes provided via a higher level package, which thus acts as a sort of data model. There are clear analogies with the SE principles outlined above.

The Problems

There exist general problems that do not specifically result from the lack of adoption of SE methods ; rather they reflect the attitude of the management of the project that software building is a second rate activity compared to, say, building detectors. This often results in no formal managerial structure for the software team which itself is often largely composed of young physicists who have little or no experience working on large software projects. Furthermore there is usually no means of obtaining suitable training. Reliance is made on consultancy provided by in--house experts who often themselves have received no formal training.

One of the main reasons why SE has not had a bigger impact on the more recent experiments is the lack of money for automated tools and, until recently, lack of good tools themselves.

The following is a summary of other problems which are addressed by SE methods and techniques.

• The lack of precise design specifications leads to wasted effort particularly when code writing is not centralized.
• Documentation is often poor and sometimes non--existent. Often the only way to know what is going on is to read the code.
• Due to a lack of suitable communication methods, it is often the case that software specialists are unable to get feedback and input from physicists not directly involved with the software. These same physicists feel cutoff from the software activities.
• Project management and progress assessment is difficult due to the lack of any management tools.
• Frequently, the people who write the code have left the experiment before the first real data has been processed. The resulting combination of poorly designed software with (often) minimal documentation results in serious maintenance problems.
• At the SPS, the previous approach to writing accelerator controls software consisted of providing a powerful interpretive environment where users were free to write their own programs. A decade of such practices resulted in diverse packages, often undocumented, that constituted a system which was very hard to maintain. A major upgrade, affecting nearly all sub--systems, was essentially unthinkable given the then current practices. An alternative approach to writing control systems software has been to rely on the accelerator expert explaining the requirements to programmers who translate directly into code. Experience has shown this is not the most efficient way to work.

Conclusions from Experiences with SE Methods

It should be emphasized from the outset that SE techniques are not a panacea. Rather, the use of these methods brings with it an awareness of the necessity for proper managerial structure and proper training. The adoption of a particular computer aided tool, however versatile and powerful, will do little to solve the problems of large scale software production without the necessary commitment to apply the systematic engineering--like concepts inherent in SE methodologies.

For SE to be effective, the project management must understand the principles, support the effort and be prepared to make serious commitments in money, for the necessary hardware and software tools, in time, for training and learning, and in manpower at the start of the project.

The methodologies comprise a wide selection of techniques and practices, and it is up to each project team to select the sub--set they feel comfortable with. Though the HEP environment differs from that in industry in terms of management structures, it is clear that it should be as unthinkable to work on a complex software system without following guidelines and having some level of expertise as it would for the non--expert to take a soldering iron to the inside of a Fastbus module in the hope of improving it.

To cope with the problem of high learning restrictions before physicists can start to work on large software projects, it should be recognized that not every one needs to be equally skilled. Alternatively, in this domain of specialized skills, a certain division of labour is required.

In the SPS controls software system upgrade the results have been quite positive with relatively straight forward design and coding following a proper analysis phase. The system performs well with very few changes needed in the requirements specification.

Use of SE methods works both in a traditional HEP environment, where there is a broad overlap between analysts/designers and those writing the code, as well as a more structured situation, such as LEP controls, where accelerator physicists analyse the requirements and professional programmers perform program design and implementation.

The following is a summary of more specific conclusions:

• It is necessary to have formal training by experts who themselves should preferably be knowledgeable in the context of the particular project. After training, there has been a good acceptance of the methods, in particular by younger members of the community.
• Graphical analysis methods make systems easy to visualize and have been found to be very useful for communication and documentation between workers in the field and interested parties such as the project management or newcomers, who themselves have had no formal instruction. Data Flow Diagrams of Structured Analysis have proved to be a very good analysis tool and are often used in the broad design as well. Although the analysis and design is self documenting, discipline is required to keep the diagrams up to date in the light of subsequent modifications.
• Data modelling is of paramount importance, as it extends from the analysis phase down to code writing and data presentation thus providing continuity throughout the life cycle. For HEP the Entity Relationship (ER) model [Footnote: P.P. Chen, The Entity--Relationship Model - Towards a Unified View of Data, ACM Transactions on Database Systems, Vol.1, No.1(1976), 9-39 ] has been found to be very useful. The mapping onto the relational (or tabular) model is very easy, making it convenient for data base design work. With suitable tools, such as those developed within Aleph , physical data structures can be built and documented automatically from a data definition language. Data may also be manipulated and validated. The corresponding graphical depictions of the data model are termed Entity--Relationship diagrams (ERD). For certain applications in information analysis and data base engineering, other data models are preferred.
• In the context of a complex multi--processor, multi--tasking, on--line environment, SA tools which enable modelling of control and real--time aspects have proved to be very useful in providing a clear understanding of the required functionality of the system [Footnote: Reference: T. Charity ,R. McClatchey, J. Harvey, Use of Software Engineering Techniques in the Design of the Aleph Data Acquisition System , Comp. Phys. Comm. 45 (1987) 433-441 ] The advantages of problem analysis, partitioning into components and early documentation are as valid in this context as for the off--line. Of particular importance in this area are the newer additions to SASD i.e. State Transition diagrams, for modelling system response to stimuli, and Data Flow diagrams for modelling the data and flow of control.
• Automated computer tools for diagram manipulation, editing and verification are essential. Aleph has been restricted by the lack of widely available tools. An important feature in any such tool is that it be of an open nature rather than a closed system, thus permitting extensions to be built as required and to allow different tools to be used on the same project. Aleph have made important extensions, particularly in the areas of data modelling and 'reverse' SE products that analyse written code.
• Today the relevant hardware environment for use of SE tools is a workstation with its large graphics screen, window capabilities, adequate CPU power, text editor, compiler, etc.
• Although the analysis and design features have been successful, there are still gaps, or less precise methods, to go from design to the actual coding.
• The 'walk--through' or suggested method of analysis, design and code review has proved to be useful. The reviews of the analysis were usually done by groups, with each person in the group standing up in turn to show his work, and the rest looking for deficiencies. The temptation to correct the work rather than assess it was largely resisted, as this would waste time and offend the person who had done the work. Code reviews are time consuming and were carried out by assigning an individual to review a set of subroutines. Such reviews, plus the necessity to merge input from different sub--projects, require relatively frequent meetings of representatives from the entire software team.
• Fairly simple but effective techniques may be used to monitor the progress of the overall project. Time estimates are made for each sub--project and for the relevant milestones (whose definitions are agreed in advance). The study of the evolution of regular reassessments of these time estimates enables the project managers to spot delays and problems before they impact seriously on the overall project. Commercially available tools like MacProject (APPLE) are useful in this context.

To summarize, software engineering techniques have been successfully used in Aleph, accelerator controls and other CERN projects. The methods have been very helpful in providing a clear understanding of the problem at hand and as a means of communication of ideas within the groups both between experts and non--experts. Similar conclusions have been drawn by groups from FNAL, DESY and other HEP laboratories. In the past, the lack of suitable automated tools has been a restriction. This situation has now changed. Good tools are available and further improvements can be expected. The majority, if not all, of the groups mentioned in this report have selected the same SASD based tool (Teamwork) which is therefore becoming the de facto HEP standard.

Requirements of Outside Institutes

Software development does not require full--time presence at CERN and is therefore an area in which physicists in the outside institutes contribute to experiments. This is especially true of small groups in universities which lack the technical resources to make contributions to detector hardware. Some of these groups have local computing resources, such as medium sized mainframe computers or workstations, others rely on network access to remote computer centres at CERN or elsewhere. Some institutes have quite limited resources for buying new computer equipment and software.

Within the above constraints, the users in institutes outside CERN have much to gain from SE techniques. The tools for breaking large problems down into well defined parts which can be tackled independently are of particular benefit to the community outside CERN. Such subproblems can be taken on by individuals or small groups of users in the outside institutes. While the work is done independently, the common use of SE tools ensures consistency, so that the code which is produced should fit together correctly. At the level of a common data dictionary the different groups may maintain close communication. The diagrammatic tools of SE, such as data flow diagrams may also improve communication between groups at remote sites. However, as already noted above, SE tools will not remove the need for discussion, and regular meetings between contributors to a software project will continue to be important.

Training in SE methods and tools must be provided for users from outside CERN. Where possible this is probably best done nationally to save on substantial travel costs; this is easiest in countries with a large national laboratory, e.g. Rutherford Appleton Laboratory in the U.K.. In any case, courses generally only cater for a limited number or people (15 or so) for maximum benefit. Good documentation is clearly a necessity - both pedagogical and reference. The special problems of training students need to be considered. In general it would be sufficient that limited training be given so that they could use the results (e.g. of analysis and design) of SE techniques.

Standardization between the different HEP laboratories would be highly desirable since many institutes are involved in collaborations with more than one laboratory. Standardization with other users of national laboratories would also be useful, although probably harder to arrange. The benefit of this to the outside institutes is that costs of hardware, software licenses and expert support may be shared. A more general benefit is that physicists will not have to learn new tools when they change experiments.

At present, some of the best SE tools are available only on workstations which are still relatively expensive items; the software licenses can also be very expensive. Before committing the CERN community to a particular computer aided software engineering (CASE) package, the financial implications for the outside institutes must be fully considered. It would not be acceptable for HEP groups outside CERN to be excluded from contributing to the software effort due to the choice of an unnecessarily expensive set of tools. In view of this, it may be worth considering systems which are available at relatively low cost and which run on cheap or existing hardware.

The problems of maintaining a central database for SE tools will have to be faced. One of the key ideas in (some) SE methodologies is the data dictionary. This at least must be maintained centrally, although copies may be made for reference. The problems of maintaining such a database, with updates coming over wide area networks, must be addressed.

Other Coding Support Tools

The software engineering environment is completed by implementation and coding tools. In this area tools have already been widely used in the past, and experience exists. We present here a brief review of the more important tools and their functionality. No mention is made of standard compilers, and we also exclude Documentation and Data Base system tools, discussed in other parts of this report.

Source Code Management

Source code management means the use of a tool which helps to maintain different versions of a program (or other text file). Various tools have been in use by most experiments over many years. Several exist today and have shaped the habits of their respective user communities. No tool exists which clearly responds to the modern interactive workstation--oriented development environment.

An overview of the situation in the HEP community is the following:

• Patchy and Historian are currently the only available tools for those who need a portable code manager. Patchy will be used as long as there is someone to maintain it. Historian continues to be offered for those who need the features not available in Patchy. The list of such features could include binary program management in the near future.
• BASE is a recent portable product written to specifications of the HERA experiments, by a commercial software house. It targets a completely new type of source code manager. The product is presently being installed on DESY's IBM system. BASE clearly bears watching.
• CMZ is an interesting new extension of the conventional source code manager concept. It offers convenient interfaces to the operating system via a Unix--like command language. Though a private initiative of CERN--connected people and currently unfinished, it has attracted significant interest in the CERN community. The terms of availability of this product have not yet been defined.

There is no consensus concerning source code management requirements in the community. In view of the rapidly changing environment, new projects are not recommended in the area of code management tools. In the long term, it is possible that Unix will offer a sufficiently portable and friendly environment which may obviate the need for portable tools. Opinions differ considerably as to when and if this will happen.

Other Implementation Tools

In addition to the functionality traditionally offered by source code managers, other important tools exist that help in the development and maintenance of software projects. The following is a partial list of some of the more useful categories.

• Syntax--Directed Editors "understand" the syntax of the language. In addition they provide coding templates and (sometimes) partial compilation/execution capabilities. The best known such example is LSE (DEC) which, although not so useful for Fortran code generation via expansion, is very useful for ADA. It can be customized for completely different applications such as entry of DDL in the ADAMO syntax.
• Static Analyzers take source code as input and perform checks such as searching for undefined variable, inconsistent common block lengths and conventions not being followed. Other functions include the production of program calling trees with plots and checking for use of recommended library routines. Currently such tools have been developed at CERN and have been widely distributed. [Footnote: Documentation: J. Bunn, FLOPPY User's Guide, CERN/DD/US/112; H. Grote, FLOP : a Fortran Oriented Parser, CERN/DD/US/13. ]
• Program Testing Tools of which the most common and essential variety is (symbolic) debuggers, enable the monitoring and modified execution of programs with possibilities to display the source code as it is executed. The best example is probably the VAX/VMS debugger although IBM's IAD is functionally equivalent. Path Coverage analysers, such as PCA (DEC), show which program paths have been executed or not. Such tools are useful for performance analysis as well. Regression testers, such as DTM (DEC), attempt to ensure that programs under development run correctly by establishing benchmarks and then comparing with runs made with new developments or modifications.
• Optimization tools, such as PCA (VAX/VMS), SPY (CDC,IBM/MVS) and IAD (IBM/VM), enable the identification of parts of programs that consume the most CPU time. Optimization should, however, be done late in the development of a program, bearing in mind the probable loss of clarity of the code.

An important aspect in the choice of support tools is the selection of a coherent set within a common framework. There exist several ambitious projects [Footnote: The Toolpack Project from Numerical Algorithms group and PCTE : Portable Common Tool Environment (ESPRIT). ] that attempt to create a global software environment consisting of an integrated suite of software utilities.

In order to improve efficiency of software production, programmers must change their habits to make increased use of such tools. This will require both learning time and financial investment. Effort is required to keep abreast of new developments in this rapidly changing field and also to provide local support for selected products.

Software Engineering: Recommendations

To Project Management

Large software production is a serious activity and deserves, from the start of the project, as much attention as detector building. Adequate software tools are required as well as the necessary computer hardware. " Software engineering tools and methodologies have been shown to be effective provided sufficient support is given by the project management. Use of such tools alone will solve little, unless the required managerial structures and practices are adopted such that very large collaborative efforts may be partitioned into smaller teams working in parallel. "

Our choice is "pay now or pay lots more later". [Footnote: From: J. Manzo, On Managing Large Scale Projects : Some Simple Principles for Developing Complex Systems ,Comp. Phys. Comm. 45 (1987), 215-228 ] In this context later could mean never or at best only partially fulfilled physics goals from experiments employing vast manpower and financial resources.

Training

" Training in the concepts and ideas of software engineering is essential and should be provided by (repeated) courses given by experts from HEP. " Realistically it should be recognized that, given the range of skills needed to design and construct large scale software, there has to be some division of labour within the project team. Different skills are required by the project managers, say, than by those working more at the implementation end.

A central compilation of the documented experiences from past and existing projects would provide the necessary complement and back--up to formal training.

Support and Consultancy

Support and consultancy is required on several, not necessarily independent levels. " A central support group should be set up to evaluate European HEP requirements and negotiate licences for the necessary tools. This same group would also act to provide feedback to and from the vendor for bugs, problems and improvement, and help with product installation. " This group should coordinate the use of SE tools across different projects and different HEP laboratories, and build and maintain extensions required by the HEP community. " A special interest group should be formed consisting of users of SE tools both at CERN and the outside institutes. " The functions of this body would be to provide communications between the groups involved, to decide on the common tools required and to make sure that where possible, progress may be made on common problems. " Day to day consulting on conceptual problems is best dealt with by expertise from the project itself. This requires an establishment of a base of expertise in each project. "

Software Tools

Automated computer tools are essential. " A powerful set of tools capable of supporting development of a variety of projects (batch oriented systems, real--time applications and data base modelling) should be selected with a view to standardization across the CERN and HEP community. The relevant hardware environment is the workstation. "

Adequate, though not perfect, tools exist today. It should be remembered that like any other tool, they will evolve, and new ones will have to be obtained as requirements change.

Licences

" CERN should negotiate site--wide or HEP--wide licences for groups working on experiments at CERN, and for other required applications. "

Requirements of the Outside Laboratories

Software development will remain a strong, major activity of the outside institutions. When negotiating licences it will be important to keep in mind the limitations of hardware and financial resources of these institutes. Standardization of SE tools, both CERN--wide and HEP--wide is very relevant in this context. Training for outside users should be provided at CERN although not exclusively.

SOFTWARE SUPPORT: GRAPHICS

The subject of graphics support was considered by a separate working group. We gratefully acknowledge the kind collaboration of P.Schilling and J.Olsson from DESY.

Graphics in HEP

Graphics Applications

Within the last years the range of graphics applications within HEP has become extremely large, from the simple display of histogrammed data to highly interactive dynamic event viewing. This growth in the kinds of application programs has been paralleled by the very fast development of graphics devices and workstations. The types of applications considered in this document are as follows:

• Detector design: CAD and physics simulation in order to optimize detector performance and save beam test time.
• Software development: Detector representation, interactive display of reconstructed events, debugging pattern recognition programs, etc.
• Experiment analysis: Event and detector viewing, physics parameter representation, multi--dimensional statistical analysis, histogramming. (Both in 'batch' as well as 'interactive' modes.)
• Publications: use of graphics displays with a graphics editor to produce figures for output onto high--quality hardcopy devices.
• Human interface for control applications.

With respect to the above list of applications we will concentrate essentially on software development and experimental analysis which will be, together with the publication and human interface area, the main use of graphics during the coming years of the LEP era.

Implications for Software and Hardware

The large range of devices available from numerous manufacturers (all with different native graphics interfaces), the large number of graphics applications mentioned above, the need to run programs on several types of computers (workstations and mainframes) inter--connected through networks, and the large number of physicists participating in the current and future experiments, are all factors which lead to the following requirements:

• transportability,
• device independence,
• common interfaces for on--line and off--line applications,
• metafiles to exchange graphics information at any level,
• access to multi--window environments,
• fast response times,
• good availability.

It is essential that a common approach to graphics and user interfaces is taken by programmers in different areas so that modules of code will work together in a compatible fashion.

The different layers of HEP graphics applications can be classified in three levels:

• Basic picture generation using primitives for lines, characters etc.
• Library functions necessary for physics analysis and event viewing e.g. Histogramming, jet drawing etc.
• Physics application interfaces.

The first 2 of these levels will be discussed at some length as the fundamental building blocks of all graphics applications systems. "

Whatever choices are made for hardware and software standards for the first generation of LEP experiments, they must take into account the need for a smooth evolution of the applications programs. "

Graphics Software and Interface Standards

Basic Picture Generation

The large number of graphics packages (Application Program Interfaces, or APIs) like WAND, GPR, GSR, Template, DI3000 etc. cover the full range of possible hardware in a non--transportable way. Thus, since it is not possible to standardize application programs to only one type of graphics machine, the 'natural' solution has to be provided by graphics standards. (Ref: Report on the Status of Graphics Standards, D.R. Myers, DD 88--14.) Of course, if use is made of features in a standard, which are not supported in hardware and hence have to be emulated, then the performance will suffer. Nevertheless, as an example, the software emulation of 3D displays, although slow on low--end graphics machines, does allow the development of 3D code on cheap hardware, and may also be sufficient for applications which require 3D but do not need real--time transformations.

As a minimum, the features required of a standard include the use of hardware primitives, characters, segments and real time transformations (when available). However, at the top end of the scale, a particular device may contain features not supported by the standards. It should be noted that the current standards, such as GKS and PHIGS, provide inquiry functions to allow one to find out at run time which features are available, and thus to decide which algorithm is best suited to the occasion.

Whilst using a graphics standard, such as GKS, for all new graphics devices can result in a significant loss of performance compared with what could be attainable utilizing the hardware features directly from the application program, to a large extent performance depends on how much effort is available when writing the device drivers. Performance is certainly the reason why some people using workstations prefer the native graphics package to any standard. However, there have been several notable cases where poor performance was found to be due to the use of a standard in a completely inappropriate way. "

Training and experience in the use of GKS (or any other standard) is essential to produce well written and efficient graphics applications, and CERN support in this area is necessary. "

In the more complicated examples of HEP application programs, for example event viewing and analysis programs, the complex data structures within the application program are represented in the structures within the application code, for example in Zebra or Bos banks. Since a dynamic hierachy of coordinate systems is not required it is sufficient to map the physics data structure to a flat graphics structure, such as a set of GKS segments. This also avoids the complexity of programming graphics data structure editing functions.

GKS does not provide explicit support for workstation 'window' systems. However, assuming that the workstation operating system provides a window manager to manipulate the windows, GKS does allow multiple windows to be used, with some limitations, and this feature has been implemented at the driver level for several existing workstations. "

From the experience gained so far, GKS provides a very good standard low--level graphics application interface, giving a reasonable compromise between portability and performance. "

Basic Library Functions

However, as GKS is by definition a 'kernel' system, it is recommended that a graphics library of higher--level functions should be developed, based on what has been done in various experiment over the previous two years. The working party believes that the supported basic library should be layered on top of GKS, and this can be used as a starting point for those who wish to adapt the library to non--standard picture generation software package. This library will include dedicated HEP primitives, some of which are described as graphic macro--primitives in the Higz manual, as well as features such as histogram production, helix drawing, jet drawing, etc. This would help to prepare a smooth transition of the existing code toward new hardware and graphics standards which might be used in future experiments. "

We recommend that a set of high--level graphics tools and functions based on GKS of particular use in HEP should be provided as part of the CERN Program Library. They may be written by groups or individuals throughout HEP, but the work should be coordinated at CERN, which should also provide long--term support. "

Distributed Graphics

Interactive graphics will run increasingly in a distributed environment, hence it is essential that we gain practical experience in X--Windows implementations, and thus be able to give sensible advice to those in the community wishing to acquire software products to support distributed graphics. As X--Windows is a very low--level protocol, it will also be necessary to evaluate the various tools available layered on top (DECWindows, Open Dialog, etc.) and to choose one, or at most two of these as the CERN supported HEP interface to window systems. Moreover, X--Windows is not a graphics package, so an important area to explore will be the co--existence of X--Windows with layered graphics packages (perhaps using the X--Windows extension called PEX).

Also we have to solve the problem of communication from local interactive graphics programs with remote databases containing thousands of histograms, DSTs and raw data, as well as to define a framework open to the evolution of the hardware to new systems (Unix) and to new languages (C, PROLOG...). "

We believe that networked graphics, where all or part of a graphics application runs in a host remote from the graphics workstation, will be of increasing importance during the period addressed by this report. We recommend that CERN effort should be put into gaining experience of windowing implementations, with a view to selecting one or two as supported interfaces for the HEP community. "

With the introduction of the CRAY running UNICOS, Unix has made a definite appearance in HEP, and we expect to see expanding use of this operating system. The CERN libraries are supported already on Unix, and one can forsee the use of other languages which complement Fortran in the graphics area. The use of Unix would certainly ease the use and support of a wider range of workstations, including those from SUN, Silicon Graphics, Ardent, etc. Our goal is to provide more choice via the use of open systems.

Document Processing

Another area which is still developing is the integration of graphics with documents. The signs are that CGM will emerging as the accepted picture storage medium for such applications. However as many hard--copy devices manufacturers are providing built--in Postscript interfaces, we should not ignore this de--facto standard. In addition, the 'appendix E 2D metafile format' of GKS is now an acceptable standard, but the lack of useful 3D Metafile standard must be corrected.

Graphics Hardware

It is acknowledged that many of the major institutes in the large collaborations plan to use at least one 3D workstation and several 2D workstations in addition to normal terminal utilization for software development and experimental analysis.

The range of possible graphics hardware is very wide, and is evolving at a rate much faster than the progress in the preparation and analysis of experiments and the lifetime of software packages. Applications exist needing high--resolution, 2D and 3D, colours or grey levels (sometimes with a palette of 256 colours for smooth shading), and high--resolution hard copy facilities. For the coming LEP experiments wire--frame representations seem to be adequate, but we believe that for future experiments (e.g. at LHC) we may well be interested in solid modeling techniques. Nevertheless, there are many less--sophisticated applications, and so hardware with a range of performance levels is required. In most cases we need full interaction with the screen, both to change dynamically the aspect of the graphical representation, and to select physics quantities by 'picking' their graphical representations.

At both ends of the performance range there is a need to use simple graphics terminals on mainframes for program development, as well as 3D graphics machines with hardware processors to do real time transformations. Although the trend is clearly to replace terminals by workstations, due to the life--time of the range of machines currently existing or being bought, terminals will still be in use during the period covered by this report. We have also found that workstations currently available with a typical performance of one MIPS are not sufficiently powerful, either for compilation and linking of large programs, or for good graphics performance. Fortunately, the trend continues to be towards faster machines, and affordable devices are appearing with 4 to 10 MIPS. This is already sufficient for reasonable 3D performance for many applications which do not need smooth transformations in real--time.

At the high--end, CPUs are approaching the speed necessary to perform dynamic rotations of 3D wire frame objects without special transformation units. This has interesting implications for the utility of display lists, as their usefulness is less obvious for machines at this level. Although ultra high--level graphics performance is becoming available in the some workstations, we are more interested in enhanced CPU power, since our work concerns CPU intensive interactive computing using complex data structures rather than computer design applications.

Already in the 'MUSCLE' report, as well as in another chapter of the present work, networking between mainframes and workstations or clusters of workstations has been identified as being an important issue. We certainly endorse that conclusion. Tools which optimize file transfer and the transmission of graphics commands over heterogeneous networks will be of the utmost importance for graphics applications. Whilst it might be necessary to group workstations (perhaps of a similar type) in clusters with a database server, the ideal is clearly to be able to work in a heterogeneous environment with the minimum number of constraints. "

We recommend that CERN coordinate the definition of application--oriented benchmarks for 2D and 3D devices, these being based on GKS 2D and 3D code for the basic picture generation. In addition CERN should, in cooperation with the institutes, perform benchmark evaluations. "

Graphics: General Conclusions

We endorse the choice of GKS as the graphics standard for HEP for the coming LEP experiments. GKS will exist and be developed for more than 5 years, and new hardware capable of running PHIGS can certainly support drivers for GKS--3D, although these might need to be written or commissioned. The use of GKS--3D has the important advantage that 2D modules of code may be used within 3D applications without modification, and that application programmers do not need to learn a second interface standard. However, during the period under consideration it will be necessary to monitor developments in the field of graphics standards, especially as a new generation of standards is scheduled to appear during this time. "

CERN should provide support for the basic graphics packages and be responsible to ensure that they run on a range of equipment, as well as for distribution and documentation. CERN should also evaluate and test new terminals and drivers. It is essential that sufficient manpower with the necessary technical knowledge be made available to fulfil this task. "

Application programs used in big collaborations are written in several places using a range of hardware. Once developed, these programs are used by physicists who are not software experts. The support to maintain the applications and test them in new environments must be provided by named people within the experimental groups. If these people can be located at CERN, so much the better. "

In order to facilitate the communication necessary for decision making we recommend that an HEP Graphics Steering Group be formed with representation from CERN, collaborations, institutes and major computer centres associated with HEP graphics processing. It may be sensible to look for institute representation from country working groups where they exist, since such groups cut across boundaries between collaborations in a positive manner. "

SOFTWARE SUPPORT: USER TRAINING

Introduction

This topic was considered by a separate subgroup, the work of which has proceeded primarily via discussions with working experimental physicists and programmers. There have been no questionnaires distributed to the HEP community and no response has been received from the implicit invitation to contribute in the CERN computer newsletter.

CERN provides major central computing facilities for experimental physics. There is, in general, a small overlap between the activities of the professional staff running the central computing facility so their training should come from commercial sources. Often CERN staff are involved in the development of system software in collaboration with the manufacturers and this contact takes on the role of training. Similarly, clerical and technical staff are encouraged to attend courses organized by the Technical Training Committee. Having ascertained that the Education Service has no surplus manpower, terminals, workstations or conference rooms, we have made no in depth study of this area.

The Physicist's Demands

We found that the majority of physicists rely heavily on the documentation and working examples of the libraries and program products which they use. Many have acquired new skills through reading books, guides and manuals.

There was a wide range of responses to the idea of receiving formal training. Some physicists were prepared to attend a course which suited their needs; most supported the idea that training should be available but found it difficult to fit it into their own work schedules and a few suggested that if they needed to attend a course to use a package then this alone was proof that the package and its documentation were inadequate.

Documentation and Examples.

Most physicists drew attention to their need for greater clarity, simplification and general improvements of the user guides, reference manuals and other documentation of the libraries and program packages they use. Two detailed points were that working examples should work and there should be more simple examples in the documentation of the major packages currently in use at CERN.

The online FIND and HELP facilities were generally appreciated as was the assistance which may be obtained via the User Consultancy Office (immediate help with simple problems and requests for information, distribution of user manuals and access to reference manuals, and the transfer of difficult problems to the best available expert). There was however some concern, expressed particularly by physicists making occasional visits to CERN, that even online documentation may be out of date and examples do not work because they are not fully maintained in the evolving CERN computing environment.

In general, physicists working outside CERN were content to receive printed copies of documentation from their experiment and from major CERN products by air mail, and corrections by telefax. A few expressed a preference for a service, to deliver documents and updated pages over the electronic mail networks. This is difficult to achieve because the printing and publishing facilities on the Central CERN computers are incompatible with those in the HEP community as a whole.

Everybody, from the worst hackers to the most talented and prolific producers of the routines and packages used by experimental physicists found it difficult and time consuming to produce and maintain documentation in English. The experts, with their familiarity and detailed knowledge of their working environment, also find it difficult to appreciate the needs of the end user.

" To respond to the expressed needs of producers and consumers we recommend that a qualified and experienced technical writer should be recruited. "

There are a large number of creative people at CERN it is not too surprising that it has often been a long and arduous process to reach agreement on a standard such as common procedures for the production of technical notes, papers and other forms of documentation. However, given the considerable dissatisfaction with the documentation within experiments and the manuals for many CERN packages, we would expect that the authors and managers of code should approach any new initiatives on the structuring of documentation with an open mind and a spirit of flexible co-operation on their part.

Formal Training Courses

Training courses for physics-computing lasting two to five days are given approximately once a month at CERN. The requests to attend these courses exceeded the number of places, especially for those courses with a large component of practical work using terminals and workstations.

We found several cases where physicists had not reached the point of applying to attend a course. For example, there were very important activities at CERN which took place during the course and the course was given some months before the new skills would be put into use and the knowledge would not be retained. The pressure to produce results, for the design and installation of an experiment or the publication of results at conferences, inhibits attempts to take the time to learn new techniques which might lead to more efficient working practices.

The demand to attend courses already exceeds the capacity and may be assumed to rise if there were better access and a choice of dates and extra subjects were introduced.

The studies of data-bases, software engineering and graphics in this report identify several areas where (200) experts within experiments should receive training and, with the introduction of the personal workstation, many of the (2000) physicists at CERN will also wish to learn new techniques.

" Therefore we recommend that training courses and tutorials should be given repeatedly "

For subjects given by visitors to CERN, or for physicists attending commercially run courses suited to CERN's requirements then the number of people receiving training will, most likely, be determined by financial constraints.

For topics in which there is already expertise at CERN, or a CERN based package, the people writing or maintaining the package often prepare and present the courses. This does give them some useful feedback on the package itself.

Once a course on a topic becomes stable and mature, or the level of demand requires the course is given many times a year, then the author's time is not spent most effectively in repeating the course.

Resources should be provided to maintain an appropriate repetition rate.

The courses may be given by trained demonstrators, the course may be recorded and video training introduced, and eventually a large component of the training may be given through computer based learning techniques.

As courses are given more frequently they may also become more modular.

The tutor does not have to cram in all topics of interest to people with differing levels of ability into the only one week course which will be given on that topic for two to three years.

Pre-requisite skills may be defined (for example the ability to use a workstation, a knowledge of the languages used in practical work and units of related courses) and units suitable for the technical expert in a collaboration may be missed by the end user physicist.

HEP regional centres should consider providing host facilities for their user community to attend live courses given on CERN based products by visiting CERN staff.

New Working Practices for Physicists

There was a strong feeling within the working groups of this report that resources should be allocated to educate the HEP community to exploit the advances of software engineering and computing technology.

It has been difficult to persuade busy, self--motivated physicists that the investment of their time in learning new techniques may lead to improved productivity (or faster physics). Indeed, there are physicists who have made almost daily use of the keyboards of teletypes, punch card machines and computer terminals for over a decade have not found the time to learn to type efficiently.

We have seen the start of the widespread introduction of personal workstations into offices at CERN, we expect 1000's by 1992. Thousands of physicists will use the CERN based Physics Analysis Workstation (PAW) package for interactive data analysis.

Personal Workstations

The estimated rate of the introduction of workstations leads us to believe that there will be one or two new users of workstations per day at CERN.

The new user, and visitors to CERN, should be able to sit with a relative expert for 2 to 3 hours and learn the basic use of the workstation, for example screen management, the help system, the file system and editing techniques. Other material could be divided into modules such as an introduction to the use of interactive debugers and the available software engineering tools; electronic mail, text processing and graphical data presentation; and remote login (terminal emulation), job submission and file transfer. " We recommend that there should be an introductory session for new users of workstations. "

There may be a demand for several such sessions each week. This has proved to be a very efficient way for a physicist getting started within the workstation clusters of large experiments but it is a workload which should not be carried by other physicists.

PAW - The Physics Analysis Workstation.

The Physics Analysis Workstation project (PAW) is an attempt to provide interactive data analysis tools across a wide range of mainframe computers, terminals and workstations. The project is strongly supported by CERN based experiments and the software has recently been generally released for user feedback. In short, PAW may become the interactive data analysis tool for thousands of physicist over the next few years.

Physicists outside CERN given the manuals of the PAW suite of programs (and told by their local system manager that PAW is now available) have reported that it is difficult to get started. Although the product itself could be easier to use, there is clearly scope for improvements to the manuals. We repeat that the introduction of more simple examples is a frequent request and the manuals of the PAW suite would benefit from the attention of a technical writer.

Courses given at CERN on the use of PAW have been oversubscribed and well attended. The excess requests from one course may completely fill the next course in three to six months time. The demand for an introduction to PAW is expected to rise as the LEP physicists move from installation to data analysis and as PAW becomes more mature.

The policies and priorities for funding within many institutes make it difficult for many physicists involved in software development and data analysis to come to CERN purely to attend a PAW course.

" PAW is an ideal candidate for the production of video training cassettes at CERN and for the courses with personal contact to be given by "demonstrators" rather than the authors of the component sub-systems. "

Software Topics.

The experts within experiments are aware of the pioneering work of Aleph in the use of software engineering techniques. Many recognize the shortcomings of traditional methods but now wish to concentrate on the physics analysis aspects of their experiment rather than start to make the significant investment to adopt formal methods within a collaboration after a few hundred thousand lines of code have been written.

It should not be left to the "next" experiments before some training in the use of software engineering techniques is introduced at CERN. Young physicists should have the opportunity to be trained to use these techniques as they will be used in industry or future experiments where they may eventually find employment. New projects could be undertaking using the techniques and packages which are available today; for example the data processing experts in the computer centre and the Lep experiments could co-operate in the design and implementation of their tape handling utilities.

We considered the possibility that the adoption of formal specification methods would allow physicists to concentrate on the development of algorithms and analysis and use trained programmers to implement the code. This proved to be very unpopular when suggested to working physicists.

Training in data modelling would also lead to an awareness of the potential use of data bases. We have found that there is a significant acceptance of SQL as a language with which to make database enquiries for book-keeping activities (during detector construction).

High level physics analysis languages are emerging as the LEP experiments draw near to data taking. These languages reduce the work to be done by a physicist analysing data but an expert may be needed when features are missing from the language. They give many advantages when comparisons of analyses are made as many cuts and definitions are defined within the package. At present there use seems likely to be private to one experiment and its own training resources. In the past such packages (like macro SUMX) did not gain a widespread user acceptance. It is considered to be a strength of PAW that user actions can be taken via interpreted Fortran files (via COMIS).

A Central Workstation Training Facility

We are entering an era where modern software working practices may be introduced into HEP work. The techniques which physicists and programmers should learn are discussed earlier in this report. Some subjects may be taught to a large audience in the CERN lecture theatres using traditional classroom techniques. However the bit mapped personal workstation provides the best user interface to the software tools which will become part of everyday working life. A large proportion of training activities should take place where the physicists may learn through relevant exercises on workstations.

" We recommend that the provision and operation of workstations and terminals for hands on training and demonstrations should be a central responsibility. "

A 'divisional workstation training office' in the computer centre has recently been established. This has five Apollo workstations, three VaxStations and two Pericom Graphic terminals. This gives a reasonable probability that a person attending a course may choose to use the same type of terminal as they use in their own work. A class size is limited to around twenty by the number of available terminals with two people sharing one keyboard. This also seems a reasonable limit for the personal attention of a tutor and the size and air conditioning of the office.

In round numbers, if this facility were to be used for training courses for five days each week and fifty weeks of the year then 2500 members of the CERN community, working on the frontiers of modern technologies, could each receive 16 hours training per year.

There may be a demand to enlarge this facility or to provide further units of a similar size.

The workstations have also been made available to experimental groups for private demonstrations of their own analysis packages and for evaluation of workstations in general. The processing power of the workstations may be exploited by appropriate processing tasks overnight and as a low priority activity during the hands-on sessions.

Videa and Computer--based Training

Video and computer based training have the advantage that the student may work at his/her own pace, at any convenient time, without the presence of an expert or teacher.

There is already some experience of the use of video training at CERN and in the production of video training cassettes. CERN computer centre staff have received video training in some basic VM operations and many find this an acceptable learning process. Some of these videos were recorded specifically for HEP use [Footnote: B.White visiting CERN from SLAC] and videos have been recorded at CERN for the safety training of the members of each of the Lep experiments. " There should be a library of video and computer based training courses at CERN. There should further be a public facility for a single user, or a small group, to use this material. "

Video and computer based training courses are available commercially for operating systems, programming languages, data bases, word processing and so on. A selection of subjects from a catalogue of 80 courses [Footnote: Computer Technology Group, Fox & Elder, U.K. ] is tabulated. The videos runs for 20 to 45 minutes and the list price is about 500 sfr (200 ukl).

Accelerator Computing Requirements in the Nineties

Working Group on Accelerator Computing Final Version, March 1989

Introduction

This report has been prepared in response to a request from the Steering Committee for the report "Computing at CERN in the 1990's" chaired by J.J. Thresher. To prepare the report a subcommittee of six members was formed with as chairman Roy Billinge who is the member of the editorial committee responsible for accelerator computing. The composition of the subcommittee was as follows. Roy Billinge as chairman and providing the input on MIS activities as chairman of SCAIP. George Shering as secretary and providing the input on networking as a member of TELECOM. Lyn Evans as specialist on SPS matters, and as a user of controls rather than as a builder. Claude Hauviller as liason with CAE, in particular the mechanical design and engineering aspects. Fabien Perriollat as the expert in controls and member of TEBOCO. John Poole as the expert on databases and to bring input from LEP/CLIC on the requirements for simulations. The members were charged to reflect their own views, those of their divisions, and those of the whole of CERN, not only in their specialist fields but also over all the subjects discussed. The subcommittee met six times, and this report summarises the discussions and decisions made.

Main Conclusions

Networked workstations will become the generalised computing tools, with about 1000 general purpose office workstations and 100 special purpose workstations in the accelerator divisions. Local and special purpose servers and networking support will continue to be supplied from within the accelerator divisions, but the accelerator divisions will increasingly rely on DD for central network support, for the main computing facilities, for the accelerator database engine, and for backup facilities. For all this they will expect as good support as the research divisions. It is agreed that CERN should remain a multi-vendor site as a whole. The accelerator divisions, however, retain the right to opt for a single line, either as a group or each division separately. Support for open systems such as UNIX, and products that will run on a variety of support engines, should be continued and even given preference. The accelerator divisions have a substantial commitment to IBM compatible office workstations (PCs), mainly of European manufacture. Network facilities and interconnections are being set up to capitalise on their use. DD support for the supply and networking of this line should continue, and all CERN-wide facilities made available on such networks. Coordination should be provided for the choice of software to be used, and its relationship with other CERN software options. Networking in the accelerator divisions is seen on three levels, the office LAN, the main controls LAN, and regional LANs for equipment. A reduction in diversity of networking technologies is desired. DD should provide strong central networking support, and should handle all aspects concerning licences, distribution, and grouped ordering. DD should also do the site network management, including allocation of names and IP numbers. A load of up to 1000 packets per second between the accelerator LANs, and to the centre is envisaged. About 100 home terminal connections will be required in the accelerator divisions. There should be a single central database management system for all accelerators, preferably running on the same computers. Over the period in question this should be ORACLE, which will be increasingly relied on for operational requirements. Thus an ORACLE piquet, and a piquet for its support engine, will be required. Accelerator design and modelling techniques will move in part to workstation networks, but many large jobs will still require the main computing facility. This could require between 0.2 and 1 CRAY XMP/48 year equivalent spread over the five years 1990 to 1995. With the increasing power of workstations on the one hand, and the better performance of networks on the other, the accelerator control systems would be interested in replacing their local computation facilities by on-line access to the central facility provided a suitable agreement on availability could be achieved. CAE will continue to be important with Euclid on VAX being the mainstay of the mechanical design, Autocad on the PC networks for mechanical drawing, and PCAD on the PC networks for electrical design. It is foreseen that that over the period the main priority for mechanical CAE will be the integration around a central database of design, drawing, computations, manufacturing, and documentation packages. PRIAM has become vital to the accelerator divisions. The support for embedded microprocessor systems should increase, and DD should support an integrated software workshop including tools for the specification, design, development and exploitation phases. This should work in liason with ORACLE. The central documentation service should be expanded to cope with technical and engineering documents, and should work in liason with ORACLE. Increased manpower will be required in the accelerator divisions for computing other than the control systems. In particular up to 15 people will be required for local informatics and networking support, working in liason with the central support services. Database application design will require another 15 to 20 people, which may become less if one team could do all accelerator applications.

Background and History

The Accelerator divisions are involved in computing in two specialist areas, namely Accelerator Design and Accelerator Controls, and in three more general areas of Computer Aided Engineering, Database usage, and Management Information Systems. They were early into computing with programs for lattice design, particle tracking, and field calculations. With the advent of computer control of the accelerators, the accelerator divisions had to build up a very considerable expertise in computer activities, including things like busses, language design, networks, databases. This also overflowed into other uses of the computers such as CAE and MIS so that the accelerator divisions often played a leading role. With the increasing penetration of the computer into all areas of CERN's activities, however, many of the accelerator control based activities became of general application, especially as the experiments grew to almost accelerator size. Also there has been considerable pressure on human resources in this area. Thus we expect to see a major shift in the nineties to the accelerator divisions becoming users of more generally supplied services, albeit very knowledgeable and discerning users.

General Model of Accelerator Computing

Schematic View of Accelerator Computing for the Nineties Figure 1 shows a schematic view of accelerator computing as foreseen for the nineties. Office workstations will progressively replace terminals, and will be networked together and to various servers. A bridge to the CERN Backbone will provide access to the central facilities and to other networks. It is reasonable to suppose that the accelerator divisions will remain grouped geographically into the PS, LEP and SPS areas, each with a local area network infrastructure. As the personnel distribution and political organisation of the accelerator divisions is not invariant with time, the division into geographical areas must not hinder logical communication between areas. The Main Control LAN will be connected to the CERN backbone for access to central services, but also to the office LAN as most of the accelerator division staff have responsibilities for accelerator equipment or operation and will want to have some interaction through the office workstation. Also many will be involved in control software development. Regional LANs will increasingly replace the more traditional equipment control highways such as serial CAMAC and MPX. These will be connected through the gateways to the other LANs for control commands, of course, but also for more general purposes such as database access and microprocessor development. Some direct access from the office workstations may also be allowed for equipment monitoring.

Workstations

There is essential agreement on the trend towards networked workstations, together with local servers, and increasing use of the central services over the network. It is agreed that a "workstation on every desk" is a reasonable aim. In the Accelerator divisions it is estimated that 1000 general purpose office workstations will be in use in the Nineties. In addition there will be a number of special purpose workstations, Vax Station or Apollo. Approximately, 30 will be used for mechanical design, 40 for controls (20 in PS controls, 20 in SPS/LEP controls), 20 for accelerator design and modelling, and 10 for database design. An attempt is being made to minimise the use of special purpose workstations, so a total of 100 seems reasonable. The accelerator divisions make extensive use of a wide range of industrial software, and so tend to prefer the IBM compatible office workstations for which there is the widest range of software and add on cards. There are, of course, other reasons such as history, European element. The PS, SPS, and ST have a policy on this matter, but in LEP both the IBM and Macintosh streams are present. It was agreed that divisions should have a policy in this area, and an adequate mechanism for implementing the policy. The advantages in concentrating on a single type of workstation include better support from the limited manpower available, better network facilities, easier communication, and a better spread of knowledge from other users. For office workstation application software a two level approach is recommended. The "high" level should be the best package for the specialist application (often, unfortunately, the most expensive). The "low" level should be a popular package running on the office workstation. Examples are found in many areas. In mechanical design it could be Euclid and AutoCad, in electrical design Daisy and PCAD, in word processing NORTEXT and a popular PC package, in databases ORACLE and Symphony or EXCEL, in software engineering Teamwork and PCSA. At the popular level there is a danger of diversification due to the individualism of divisions and groups. This might be controlled by recommendations from the appropriate Technical Boards, by selective training and support, and by providing interchange between the approved high and low end systems. Interchange is made easier if the low end system is the same as the high end, eg. Euclid or Oracle on a PC. This may not be the best answer, however, if the PC version is not a sufficiently popular one. To avoid diversification the low level package must be popular and easy to use. Management support is essential if a coherent advance in these areas is to be made.

Networking

As seen from the figure, there are three levels of networking in the accelerator areas: the office LAN; the Main Control LAN; and the Regional control LAN. For the office LAN, the accelerator divisions are following the TELECOM recommendations and soon all offices will be wired for Ethernet. It is thought that this will be adequate for the nineties. On the traffic this may generate, on average only 20% of the users may be using their workstation at any given time. However at peak periods, eg. morning, lunch, evening, up to 80% of the workstations may be in use for mail, looking at machine performance, or checking up on equipment. Thus the network should be configured for almost maximum utilisation at say one transaction per 10 seconds per workstation, maybe 10 packets per transaction, say 1000 packets per second over the three networks with a substantial fraction, say 50%, going to the central machines, mainly VM. The Main Control LAN is a subject which has preoccupied control system developers since the first successful introduction of the TITN network in the early 1970's. LEP have chosen the IBM Token Ring as it has a number of advantages for control system use. Networking now has a very wide application, however, especially within the large experiments. Much of the SPS/LEP networking effort on telephones and TDM will be combined with the DD effort in these areas in the future. The PS has only limited effort to devote to this subject. It was agreed that the accelerator divisions would be happy that a single entity, eg. DD/CS, should provide most of the LAN directions and support, with only support for local problems provided within the divisions. DD should also provide support for controls protocols, eg. TCP/IP moving later to OSI. This should include some measure of standardisation and support for Remote Procedure Call protocols of which at present there are three, the original one developed in PS plus two others, all different, one in DD and one in SPS. Connectivity between the Main Control LAN and the office LAN should be provided. This can take the form of MAC level bridges if the LANs are of the same type (Ethernet), or IP gateways if the LANs are different but use the TCP/IP protocols. DD should handle all matters concerning licenses and distribution of new versions, and provide grouped ordering. DD should also manage the main network file servers, especially from the point of view of backup. DD should provide the network management including names and IP numbers. Some critical servers may be local, managed by the controls groups, but others may be central and managed by DD. Increasingly it is believed that a regional control LAN should replace the serial control highways more traditionally used. Connection between the Regional LAN, through controlled gateways of course, should be possible to the central database and microprocessor development facilities, and also to the office workstations for monitoring and maintenance. Although it is agreed that Token Ring (a standard in its own right), Uti-Net, Apollo Domain, and Apple Talk will be around for some time yet, future decisions should move towards reducing LAN diversity where possible. Here again the Technical Board recommendations should be given management support, in particular the TELECOM recommendation for Ethernet. A number of home terminals will be required in the accelerator divisions, in connection with the more or less well specified piquet system. Although some engineers have expressed as desire to work from home, the mainline feeling in the accelerator divisions is that the official allocation should be restricted to those with on-line responsibilities. This will be about 30 per division, say 100 in total.

Databases

The accelerator divisions are heavy users of databases of all types and the first main application for Oracle was the LEP planning and installation. It is agreed that the primary repository of accelerator information should be a large central relational database. This is particularly important as the lifetimes of the accelerators and their associated equipment are increasingly outliving a given person in a given post. The lifetime of the database system should be at least ten years for reasonable stability. After an intensive inquiry ORACLE was chosen and used for the LEP planning and installation. It has grown until it now uses two big Vaxes, VXLDB1 and VXLDB2, running the cluster version of ORACLE. There is a support effort for ORACLE in DD of about 5 persons. Many users in the accelerator divisions are just now getting their applications running on ORACLE. It is agreed that the support effort is such that only one database management system can be handled at any given time. ORACLE Corporation seem to be keeping up well with the technology, and indeed are the leaders in certain fields such as SQL standardisation. It is therefore agreed that unless there is a major change in the market place, ORACLE should be the database system through most of the nineties. For accelerator data there was a consensus that there should be one single central database. This could be quite a big load. This would imply some changes in DD operations. All major upgrades should be made during accelerator shut-downs, ie. January-February. A 24 hour service with on-line backup capability and an agreed maintenance service are required together with a piquet service of Oracle and system expertise. The support required in the accelerator divisions should only need to be applications expertise. It is important that ORACLE be interfaced in a user friendly way to the office work station. Two approaches seem possible here. One is to use distributed database access such as SQL*net which essentially make the database available on the PC. The other is to use spread sheet programs on the PC and provide easy automatic transfer between the spread sheet and the database. ORACLE are working in both of these areas and the choice may be up to the user. Accelerator controls have been reliant on on-line databases, often in the form of files in the control network. It was agreed that the primary source of control data should also be the single central accelerator database, along with all the other accelerator data. The PS are currently moving their main control data base to ORACLE on VM. This raises two problems, speed of access and availability. The availability problem should be solved in any case as mentioned above. Several approaches to the on-line problem are being investigated. One is to retain control system files and provide programs to swap data between ORACLE and the files when changes are made. A second is to use a fast on-line database such as C-Tree and again have programs to interchange data. A third is to use an on-line relational database so that the links between the on-line and central database will be provided as part of the manufacturer supported distributed database. It is not yet known which of these approaches will be best, but the third will put more demands on the central database support group. A solid and fast network connection is required, possibly based on an Oracle product. There is a Remote Procedure Call interface to the Oracle VAX, but this is seen as a backup solution. Priority for small accelerator jobs could be required. It is important that Oracle be supported on a subset of agreed work stations with full networking, in particular the workstations used for accelerator control. A tool to interface with Teamwork would be required if teamwork becomes the standard SASD tool. DD should provide the interface between the standard tools to get the best results. Currently there are limitations on what can be held in and manipulated by Oracle. In the engineering database it is required to store the "geometry" of objects, not just numbers and strings. The whole concept has to be devoloped to handle this. At present work is going on to interface Euclid to Oracle. There is a need for unification of accelerator data and this will require concepts to handle 3D objects. At present dimensions are entered by hand. In 5 years or so we may have to replace Euclid, and it would be good if the eventual replacement product could be Oracle compatible. Graphics capability is also needed for applications in documentation, also for archiving and comparing waveforms to check if a signal looks as it should. Oracle and Euclid should be on the same network, so that if, say, we move a magnet using MAD we can see the new layout directly using Euclid. LEP and SPS have a machine element database which is used by the machine, survey and installation people. In the accelerator divisions, it is estimated that about 15 people will be involved in database design, and up to 30 software engineers, expert in database applications, will do applications programming. The number of designers may be reduced if there is a strong central nucleus of database engineers giving support to all divisions. On user feedback, Oracle has three user groups, the VM users, Accelerator controls, and the application developers (mainly LEP so far). This arrangement seems to be working well. When the PC users build up in numbers a separate group could be formed. User presentations will become more important as Oracle expertise begins to spread. It would be very convenient to have the same database for ADP and accelerators. Often in the design process it is important to have access to lists of manufacturers, and in operation to know the manufacturer, how many exist at CERN, contract numbers, etc.

Accelerator Design and Modelling

Although not so demanding as physics event analysis, accelerator design is increasingly a heavy user of CPU cycles on the big central machines. Requirements include lattice design, wakefield calculations, field calculations, particle tracking, 3-D structure design. Linac design can be particularly demanding in these areas. At present the main client is LEP with MAD, but as LEP moves into operation and more resources go into the design of new machines, the computing requirements will grow. Some indication can be had from the American labs. These range from 1000 hours per year of Cray time (SLAC), to one Cray XMP/48 year equivalent (SSC). Much of the work is still done on IBMs, however, so the load will be split. Accelerator design is becoming increasingly an interactive task requiring good graphics and a fast turn round time. There is a demand for the use of networked workstations, for example up to 20 Apollos for LEP, to meet this need. The control systems are increasingly using modelling techniques in the on-line control of the accelerators. At present these programs are run off-line, or on-line on number crunchers attached to the control system. All this involves extra work and complication which would be removed if the modelling could be done on-line on the central machines. This will require improved and more solid networking connections, faster turn around for remotely submitted jobs, and some declared availability. The availability need not be 100% as the modelling need not be used in the primary control loop, but will be required for operation mode changes, and optimisation studies. The availability maximum requirement might be similar to that requested for the database machines. Some statement on the on-line networked use of the IBM and CRAY are required from DD to enable the accelerator controls to plan correctly.

Computer Aided Design

The accelerator divisions have a high concentration of engineers, and have contributed heavily to the report on Computing for Engineering. Indeed more than half of the representatives on the committee for Computing for Engineering were from the Accelerator Divisions. Despite this, or perhaps because of this, some discussion and representation from CAD is also included in this report on Accelerator Computing. This is to ensure that the CAD influence on general computing requirements is not neglected. This is limited, however, to mechanical CAD, databases, and the most common requirements of the control system. To redo all the CAD requirements of the Accelerator Divisions would be to duplicate the work of the Computing for Engineering committee. Not to mention this area nor to consider its impact on workstations, networking, and the control system would unbalance this report on Accelerator Computing. Therefore any mention of CAD requirements in this chapter and elsewhere in this report should be considered as inclusive and not exclusive. For a view of the requirements of CAD the specific report on Computing for Engineering should be taken as the basis for requirements in the Accelerator Divisions. The rest of this chapter is concerned with mechanical design. As mentioned above, the subject of Computer Aided Mechanical Engineering is also treated in parts 3 and 4 of the report on Computing for Engineering. CAD started at CERN in 1982 with the introduction of EUCLID on a VAX780, and has expanded till in 1988 we have EUCLID-IS on a VAX8650 plus a 785 connected to about 30 dedicated work stations. This gives a total of 7 - 8 MIPS computing power. In the near future we will introduce VAX stations in a Local Area Vax Cluster (LAVC) with a 3500 file server connected to the cluster. DEC's new 6210 multi processor number cruncher and database server will replace the 785, and 7 new VAX stations will soon be put into service. Drawing started in 1984 with the introduction of AutoCad on PCs. At present there are about 50 Autocad licenses at CERN, bought at prices ranging from 7000 francs down to 1000 francs, clearly a case for coordination. Of these 25 are in the SPS, 3 in LEP, 3 in the PS, 1 in DD, and the rest in EP and EF. Work is in progress to link Euclid and Autocad. Next year there will be a mini-Euclid on PCs which will be the rough equivalent of Autocad, so Matra-Datavision themselves are not interested in links to Autocad. More than 60% of the LEP drawings have been produced with Euclid. The layouts for the whole machine are available on the database. In the SPS only 7 people, none engineers, use Euclid. This could be because the SPS are doing less in-house design and using more sub-contracts for design and development. The main use in the SPS, as in the PS, is for maintenance type changes in layouts, and for detailed layouts of tight systems such as target areas. There is only a little use of Euclid for CAM and links to the workshops, mainly in ST. Within 5 years CAM could be vital to CERN, with the use of CAD in-house for all design, and the diskettes sent to outside industry for manufacture. There will probably never be more than 5 or 6 numerically controlled machine tools at CERN. Communication is vital to be able to send designs around from system to system, even or especially to different systems. There is an IGES standard which Matra-Datavision are supporting which may help with this problem. Autocad was originally stand alone in use, but now many of the AUTOCAD systems are linked over Ethernet using NOVELL in the PS and SPS. The support required from DD could be quite different for the high and low level systems. The high level system should be staffed and run by DD, whereas for the low level system DD should organise licences, distribute software, provide courses, etc. There is not yet a link between Euclid and the finite element packages used at CERN. Euclid is a 3-D package, however, and can calculate weights, inertias, and other geometrical attributes. It is too early to have links with the finite element packages for stress analysis, etc, but this would be very desirable in the future. This might mean moving the finite element packages from the IBM to the VAX but this would put too big a load on the VAXES. At present we use ANSYS which is an American package and CASTEM from the CEA in France. There is a possibility of buying a CRAY vectorisation package for CASTEM. To reduce the cost of using expensive packages for limited applications, it may be possible to use them across the network in our sister institutes. For example a lot of the AA design work was done using programs (TOSCA) on the Rutherford computers. This could add to the wide area networking load. The current 7.5 MIPS of CAE VAX power will suffice in the medium term, going up to perhaps 10 MIPS, but with increasingly powerful work stations at the user end. A multi-purpose mid-range work station (PC with Autocad typically) will be required per engineer, about 30 in total for mechanical engineers in the accelerator divisions. A dedicated work station (Euclid type, probably VAXstation) will be required per 1.5 to 2 designers, again about 30 in total. Software licensing will become important as this could become an expensive item. Site licensing would be ideal, or perhaps a license for N simultaneous users with no restriction on the number of potential users. Personnel utilisation for mechanical CAD is at present 3.5 specific staff in DD plus another 2 in supporting activities. Note that 2 of the 3.5 are leaving at the end of 1988. Applications support in the divisions is about 4.5 full time equivalent, mostly on a part time basis. In the future it may be necessary to use fellows and people on shorter term contracts for CAD work. A new person can become productive after about six months training, which is a reasonable investment for 1.5 to 2.5 years productive work. People with a few years experience in CAD at CERN have no trouble finding jobs elsewhere.

Local Computing Centres, Servers, Central Support.

In the past the accelerator control groups have set up fairly powerful local computing centres because their needs were new and specialised. These centres grew to handle jobs such as accelerator modelling, controls program development, microprocessor software development, controls documentation, MIS applications for the controls and operations personnel. This will change, however, as CERN is setting up new central units to handle things like MIS and CAE. The PS have two NORD-500s, one for modelling and one for general applications. Also the PS have a local centre with two NORD-100s serving the secretariats. Running these computer centres takes time and money and is something the PS feels could better be done centrally. Originally the microprocessor work was done locally on the NORD-500 but now this has been shifted to the PRIAM VAX. A lot of the PS Controls group MIS work used to be done on the local computers but with the divisional emphasis on VM this is shifting. The controls documentation is still on the NORD but this could move to a CERN-wide scheme. For controls accelerator modelling, advantages are seen in doing this on the same central machines as are used for the accelerator design, using as far as possible the same programs. This will require, however, some decisions on the central machines' availability for networked real time job submission. Thus the PS see the eventual replacement of the local controls computing centre replaced by general central facilities. On the other hand the SPS are happy with a local facility for accelerator modelling (currently a NORD-500, possibly an Apollo DN-10000 in the future), as they find it easier to get computing time and a good interactive service. Also it is easier to connect to the control system. LEP would like to move to workstations for both accelerator design and modelling. As mentioned in several other parts of this report, the new model of accelerator computing is the use of personal workstations connected through a network to a variety of resources. This applies to both the office workstations and the control system workstations. The network resources take the form of servers. The control systems have file servers for their data and programs. For office servers, both the SPS and PS use NOVELL servers over Ethernet. These are used for PCAD, AUTOCAD, and general MIS applications. LEP is likely to go in this direction as well. This concept is replacing the traditional concept of the local computer centre run by the controls groups in the accelerator divisions. The servers can be located close to the people providing the support for that particular service, yet be available to anyone on the network, either in the particular division, or even throughout CERN. Thus the NORDs will see their role reduced to becoming servers on the network. In the case of the NORD-500s this will be to provide a real-time on-line computational facility. As mentioned above, this function could even be moved to the central machines using the network. The evolution of the administrative support NORDs will be in the direction of becoming a server on the network for the PS secretariats. A particular issue relating to local and central computing is backup. It is agreed that backup facilities across the network to centrally provided and run facilities is essential. This would apply to local computing centres, local servers, and even office workstations. For the servers a daily incremental backup is required, already done partially at the PS.

Control System Specific Issues

An important item is the cross software support for the Motorola 68000 family. This includes the RT kernel, at present RMS68K, and the cross compilers. at present C, Pascal, Modula II, and FORTRAN. Also the linker/loaders, symbolic debugger, and executive libraries. The aim should be to move to industrially supported products. The industrial cross development facility for OS9, UNIBRIDGE, is urgently needed for large project work in controls. The use of cross software must be carefully evaluated. It was generally agreed that cross development was about an order of magnitude more difficult than native development. Native development, however, requires provision for disks etc. at least in the development phase. One of the advantages of OS9 is that it can easily allow the development environment to be removed later. This was one of the driving forces for the 68000 in G64. An advantage of the cross software is that the central machine can provide better facilities for project management. In any case the accelerator divisions will require the PRIAM support for the 68000 and RMS68K to continue well into the nineties until a replacement has been found, run in, and a reasonable changeover period has elapsed. In accelerator control systems and equipment there is indeed a requirement for embedded microprocessor systems, say on a single card, for which a cross development environment is called for. However, much of the higher intelligence could be placed up a level where native development systems are used. Anyway DD should support a native system, for example native OS9 for the 68000 family. An Computer Aided Software Engineering (CASE) toolkit should be supported by DD with full support for the accelerator divisions as well as the research divisions. This should include tools for the specification, design, development, and exploitation phases of big software projects. It should work in liason with ORACLE. It should in particular cover the requirements of cross software projects on the CERN standard microprocessors. The choice of tools is a difficult matter. Often they are first chosen by user groups, eg. SASD, NOVELL, DV-DRAW. This is normal as the user is usually the first to perceive a problem and see a solution in the literature. Once it becomes established technology, however, it can be taken up and supported centrally. Who then should drive the evolution? The user who still best perceives the requirements, or the central support team who risk putting effort into things nobody wants? There was consensus that the initiative should normally come from the users. Note that the specification tools for databases are evolving on different but parallel paths with the SASD tools for program specification. The hardware, work station and networking aspects of the tools support should be set up, or at least specified, by DD. Accelerator controls are big users of electronic CAD and CAE, but with no particularly difficult requirements. The main uses will be for digital printed circuit board design. Services will be required for programmable circuits such as EPROMS, PALs, FPALs. This should include support for the compilers and provision of equipment for burning, equivalent or better than is provided at present. The use of artificial intelligence and expert systems is expected to have a big impact on accelerator controls and operation. At present a limited amount of manpower is being put into exploratory work involving th KEE system. When this reaches the stage of generalised application, considerable extra effort will be required to realise its promise. At this time a central support for these new facilities will be required. Controls requires a lot of documentation with the associated classification and printing. Much of this is done at present using control system specific facilities. In the future these should be provided on a general basis. Good quality (Laser) printing should be available on the corridor level for all applications including control. This raises a question of support for the corridor printers. An overall support operation could be quite expensive in manpower, so continuing with an ad hoc combination of secretarial and controls group support might be the only way to spread the effort.

N-manufacturer policy

There is agreement that CERN as a whole should not be dependent on any single manufacturer, and that competition is vital for good services and prices. Accelerator controls have had a 15 year involvement with Norsk Data which has illustrated the good and bad points of a single manufacturer policy. The SPS became very conscious of the bad points, and after a period of in house designed computers, have settled for an "open systems" approach. This involves the UNIX operating system and TCP/IP protocols, with hardware available from a number of sources. To retain some of the benefits of a single manufacturer policy the SPS has formed a close relationship with Apollo for workstations. The PS have also felt the need to move to workstations and have chosen the DEC Vaxstation. The PS have also decided to adopt the "open systems" approach with UNIX, and will run ULTRIX (DEC's version of UNIX) on these computers. This provides a common system across both controls groups, and will facilitate any future changes in hardware. DD already has a number of support activities in the UNIX, TCP/IP, Apollo, and Vaxstation areas, and clearly the Accelerator divisions expect this expertise and support to be continued and strengthened in the future. For the general purpose office workstations the accelerator divisions tend to prefer the IBM compatible PC types. Partly this is because of the wider range of general purpose software available. Also, however, the accelerator divisions are more stable and must take a longer term view on the re-usability of hardware and software which favours the use of the most widely spread material. This may result in a different choice from the research divisions which could prefer more state of the art material. Nevertheless it is essential that all CERN services be accessible from a single office workstation, so if there are "N-manufacturers" of office workstations at CERN, many central services will have to be made available on all approved types of workstation.

Support and User Feedback

There has been a feeling in the accelerator divisions that DD interprets its mission as too much in favour of the research divisions and not enough in terms of support for the laboratory as a whole. Hopefully this is changing with the establishment of the MIS unit and Computing Support for Engineers. Also the Accelerator Control Groups have been a bit "go it alone" in terms of computing technology, and again this is changing due to pressure on manpower. Hopefully in the nineties all divisions will work together to provide a range of agreed CERN-wide options. DD should see to all matters of purchasing, licensing and registration for the approved range of products. An important role of DD will be the networking and communication sitewide. It should be ensured that all equipment can communicate well, and with the outside world. This could be a main guideline in the selection of which material should be used. Technical support should be supplied by DD, in particular for the central services such as the central interactive service, central batch service, and central database facilities. However a local informatics support is also needed, and in several examples from industry, and in the accelerator divisions experience, a figure of 1% seems to be required, making about a dozen for the accelerator divisions. This local support effort should work closely with the centrally provided DD support activities. User Feedback can be provided on two levels. CERN-wide boards such as SCAIP and TELECOM with the chairman outside DD seem to provide good guidance at the top level. At the lower working level, boards coordinated by DD for special application areas such as data acquisition, MIS, CSE, PRIAM, DEC user coordination groups, etc., are appropriate.

Resources

The resources allocated to computing within the Accelerator divisions can be divided into three areas; Controls, Databases, and General Informatics. Controls are the dominant consumer of resources at present, but Databases and General Informatics are increasingly demanding. The accelerator control groups currently employ 90 persons for SPS/LEP and 44 persons for PS. Only a small fraction of this effort is directed to end user accelerator control applications. These applications are, of course, essential and should obtain more effort in the future. The small percentage of the total effort going to these end user applications is due in part to the old SPS philosophy that end user applications should be done by the end users themselves, ie machine physicists, equipment specialists, and operations personnel. PS and LEP controls have, however, officially adopted the professional applications programmer approach. The bulk of the controls effort has gone into hardware, system software, and applications support structures. There is some hope that rationalisation and the use of industrially supplied products will liberate more effort for end user applications in the future. Database applications were pioneered by LEP with ORACLE and are still limited in the PS and SPS divisions. As discussed in chapter 7, it is estimated that about 15 people will have to allocated within the accelerator divisions to program applications if the database benefits are to be fully realised. General Informatics support has grown up in a rather ad hoc way, partly within and partly outside the controls groups. About 10 people can currently be identified as providing informatics support in some way, but it is estimated that a more coherent effort of about 15 people will be required in the future to provide adequate local support, including local network support as requested by the TELECOM recommendations. Provision of strong central support in DD is essential if the above local support is to be most effective. Also good central support is vital to achieve integration and economy with the rest of CERN's computing effort. The accelerator divisions therefore support the requirements of DD for effort in the MIS, CSE and Database areas. It is pointed out, however, that a correct balance between central and local support must be achieved as no amount of central effort can replace support close to the application.

Computing for Engineering

at CERN in the 1990s 9th December 1988

Composition of the Working Group.

The main Working Group was composed as follows:

• Convenor: David Jacobs/DD
• Digital CAE/CAD: Ludwig Pregernig/EF
• Analog CAE: Guy Baribaud/LEP
• Mechanical CAD/CAM: Claude Hauviller/LEP
• Structural Analysis: Detmar Wiskott/DD
• Field Calculations: Hans--Horst Umstätter/PS
• Microprocessor support: Fabien Perriollat/PS
• Database Support: Josi Schinzel/LEP
• General Advisor: Pier Giorgio Innocenti/SPS

   Equipping TH with terminals:

     20 Mac's, PC's, Mac II's:                120 kSFr

     35 graphic screens, etc.:                 87.5 kSFr

      2 printers:                              24   kSFr

 

   Secretariat: Replacement of AES System:

     Workstations, screens, printer:           64   kSFr

 

   Library: Computerized library system:

     Hardware:                            200-250 kSFr

     Software:                            100-150 kSFr

     Peripherals:                              75 kSFr

     Optical disc storage system:             250   kSFr

     High resolution large screens (5 to 10)   75-150   kSFr