Draft EMI Testing Guidelines

This is still a draft, to be reviewed before moving it to the EMI web

Introduction

This page describes some guidelines that can be used by Product Teams for certifying the quality of their components before releasing them. We will use the following classification for different type of tests:

• Unit tests
• Deployment tests
• Functionality tests
• Regression tests
• Performance tests
• Scalability tests

Unit tests

Unit tests are meant to test the correctness of individual units or a group of related units. The definition of unit can vary with the programming language used, but in general the method is quite standard. Tools like CPPUnit, JUnit, PyUnit are generally used and they provide the necessary documentation to get started with unit tests using your favorite programming language.

Within EMI the goal of increasing unit testing is recognized. A high unit tests code coverage helps to improve the maintainability of the software. The target value of unit tests code coverage can be different for different components, therefore we do not provide numbers here, but in general:

• unit tests code coverage of [0,25%] should be improved,
• when reaching coverage around 80%, the decision to continue developing unit tests should depend on an ad-hoc analysis of the component under test.

Metrics for unit tests code coverage should be collected using tools provided by SA2 and periodically reported.

Unit tests should be performed automatically during the build of a specific component.

Deployment tests

Deployment tests verify that the component can be properly installed and configured on all the supported platforms. The deployment (installation + configuration) method can vary; the test should follow as close as possible the method used by the end users (e.g Yum + Yaim for gLite components). At the moment there is no recommended tool for this kind of tests.

Deployment tests should be done always on a release candidate, and their execution should be reported in the certification report together with the output of the command executed.

Functionality tests

In this context functionality tests cover the majority of the tests done on a component to verify it's correct (according to specification) behavior. Functionality tests should be performed according to the test plan of the given component. Depending on the component, it may or may not include interaction with other components, developed by the same or a different product team. When the interaction with an external component is needed, the testbed provided by SA2 with reference services can be used.

When the component is released with a CLI or API, these have to be properly tested: all the commands/methods have to be tested, with expected and erroneous input data.

Product teams should aim at automating as much as possible the functionality tests. The execution of the tests should be reported in the certification report, and it should include at least a short description of the test performed and the outcome (PASSED or FAILED). All the tests performed on the release candidate should pass.

Functionality tests should be performed always on a release candidates; exceptions can be done for the release of urgent bug fixes and special occasions agreed within the EMT.

Regression tests

In this context regression tests are tests that are meant to verify specific bug fixes. A regression tests should always be associated to a bug reported in the bug (defect) tracker. A Product Team should aim at providing regression tests whenever the bug fix can be automatically (with a script) verified.

The execution of regression tests should be automated, and may or may not be part of the test suite used for functionality tests. When regression tests are run as part of functionality tests, their execution should be highlighted in the certification report.

Regression tests should be performed always on a release candidate; exceptions can be done for the release of urgent bug fixes and special occasions agreed within the EMT.

Performance tests

Performance tests are tests that aim at verifying the performance of a component, which in many cases involves the measurement of the response time for specific service requests. Performance tests should verify how well the service behaves with nominal workloads.

The execution of performance tests depends on the service under test, it may or may not be automated, and can involve the use of external tools. In same cases the execution of performance tests may require the establishment of a specific testbed, and may involve sites and the coordination of SA2.

Performance tests should be done only when important changes in the component are being released and can be targeted to the study of specific parts of the component.. In general, every major release should include in the certification report a section for the performance tests, and the results should be compared with the ones of the previous release.

Scalability (load and stress) tests

Scalability tests are meant to verify that the component behaves according to its specifications when varying one of the variables that can affect its performance. Load and stress tests are included.

The execution of scalability tests depends on the service under test, it may or may not be automated, and can involve the use of external tools. In same cases the execution of performance tests may require the establishment of a specific testbed, and may involve sites and the coordination of SA2.

Scalability tests should be done only when important changes in the component are being released and can be targeted to the study of specific parts of the component. In general, every major release should include in the certification report a section for the scalability tests, and the results should be compared with the ones of the previous release.

-- GianniPucciani - 30-Aug-2010