Testing
ID | S69 |
|---|---|
Version | 1.0.8 |
Type | Overarching Standard |
Status | Effective |
Effective Date | Nov 28, 2025 |
Contracting Vehicle(s) |
Introduction
Ensures that Suppliers' software delivery test processes are of sufficient quality and rigour.
It is essential that Solutions delivered under the Catalogue demonstrate a high-quality approach to software development and testing. Within the area of software testing, there are various established Standards and many good industry practices in place.
This Standard does not seek to mandate adherence to a particular Standard, it does however, through compliance evidence, look to establish that an appropriate level of quality and rigour are in place with respect to software testing, with ISO/IEC/IEEE 29119 providing a reference point.
According to ISO, Standards are "Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognised industrial, professional, trade associations or governmental bodies". They are guideline documents, therefore not compulsory unless mandated. The Standards and external references within provide a clear benchmark for good industry practices which The Authority would expect a Supplier to follow. As such, they provide a guide to Suppliers on what level of quality The Authority expects from software development testing.
The International Software Testing Standard - ISO/IEC/IEEE 29119
An internationally agreed set of Standards for software testing that can be used within any software development life cycle or organisation:
ISO/IEC/IEEE 29119-1: Concepts & Definitions, published in September 2013
ISO/IEC/IEEE 29119-2: Test Processes, published in September 2013
ISO/IEC/IEEE 29119-3: Test Documentation, published in September 2013
ISO/IEC/IEEE 29119-4: Test Techniques, published in December 2015
ISO/IEC/IEEE 29119-5: Keyword-Driven Testing, published in November 2016
Within the context of the Catalogue, ISO/IEC/IEEE 29119 is used to provide a guidance checklist, used to establish Suppliers entering a Contracting Vehicle demonstrate quality within their software testing. Context and Risk-based Approach ISO/IEC/IEEE 29119 part 1 states "Software testing is performed as a context-managed process". It also states "A key premise of this Standard is the idea of performing the optimal testing within the given constraints and context using a risk-based approach". These statements stand true for almost any testing approach.
Further Guidance
Baseline Assurance Standard Requirements
The Baseline Assurance Standard (BAS) provides a proportionate, risk-based assurance approach for Solutions; balancing safety against efficiency by combining a minimum set of essential Requirements from the DSIC Overarching Standards. Completing the BAS is the first step to achieving full assurance with the Overarching Standards allowing Supplier Solutions to be published on the Buying Catalogue. Upon meeting this Standard, Solutions are required to meet any remaining Further Requirements in the Overarching Standard as applicable to the Contracting Vehicle within a period of 12 months.
All Baseline Assurance Requirements can be found here. Each Solution will be assigned a category of A, B or C that determines the level of assurance applied to that Solution. See the relevant category column to understand the assurance required for each Requirement. For information on Solution Categories see Solution Categories for Assurance in DSIC.
The following table of Requirements are the Requirements in the Baseline Assurance Standard related to the Testing Standard.
Applicable Contracting Vehicle(s) | ID | Requirement | Level | Category A | Category B | Category C |
|---|---|---|---|---|---|---|
Testing Requirements |
|
|
| |||
| TEST-1 | Irrespective of the Software Development Life Cycle (SDLC) implemented (traditional Waterfall to Agile), there will be organisational and project specific test processes underpinned by appropriate artefacts. Example evidence could include but not constrained to:
How are different types of testing either within a traditional SDLC or within Agile iterative-incremental development established, such as:
How testing automation is incorporated into the software build and distribution process, such as continuous build integration:
| MUST | Full Assessment Supporting evidence to include:
| Self-certification | Self-certification |
| TEST-2 | There will be Test Management Processes covering test planning and design. Example evidence could include:
| MUST | Full Assessment Supporting evidence to include:
| Self-certification | Self-certification |
| TEST-3 | There will be Test Management Processes covering test execution and management. Example evidence could include:
| MUST | Full Assessment Supporting evidence to include:
| Self-certification | Self-certification |
| TEST-4 | There will be Test Management Processes covering test completion and Reporting. Example evidence could include:
| MUST | Full Assessment Supporting evidence to include:
| Self-certification | Self-certification |
Test Environments Requirements | ||||||
| TEST-ENV-1 | The Supplier whose solution interoperates with the NHS national services and the applicable Non-Overarching Standards will provide the Authority with access to their system test instance(s), for assurance of the Capabilities and Non-Overarching Standards under the Contracting Vehicle. | MUST | Full Assessment Supporting evidence to include:
Note: The system software test instance(s) should be separate from the system software development instances, to support the testing and assurance activities of the Authority and other third party consumer Suppliers who would need to interoperate with the system software test instance(s) using the Non-Overarching Standards APIs. | N/A | N/A |
| TEST-ENV-2 | The Supplier will make access available to the requested version(s) of system test instance(s) within two weeks of the date of request by the Authority. These versions could either be compliant or versions for which compliance is being sought. | MUST | Self-certification | N/A | N/A |
| TEST-ENV-5 | The Supplier will ensure that a non-functional test environment is available to do Volumetric and Performance testing for their Solution. | MUST | Self-certification with Supporting Evidence Supporting evidence to include:
| N/A | N/A |
| TEST-ENV-6 | The Supplier will provision and support system test instances for User Acceptance Testing (UAT) and for training of End Users, enabled with all the Capabilities and national service integrations as per the Non-Overarching Standards under the Contracting Vehicle. Such instances shall be separate from the system test instance(s) provisioned to the Authority. | MUST | Self-certification with Supporting Evidence Supporting evidence to include:
| N/A | N/A |
| TEST-ENV-7 | The Supplier will ensure that their test environment(s) and system test instance(s) support robust access and security controls both at Solution level and Infrastructure level, so that only authorised users can access the test environment(s) and system test instance(s). | MUST | Self-certification with Supporting Evidence Supporting evidence to include:
| N/A | N/A |
| TEST-ENV-8 | The Supplier will ensure that the test environment is, as a minimum, available on all the working days of a week between 8 am to 6 pm. Any scheduled downtimes for undertaking maintenance activities should preferably be slotted outside the working hours and be notified to the Authority a couple of weeks in advance, to minimise the impact on ongoing or scheduled testing and assurance activities either being undertaken by the Authority or the connecting system suppliers. Any downtime required for deployment of emergency patch releases, should be notified to the Authority prior to the deployment. | MUST | Self-certification | N/A | N/A |
| TEST-ENV-9 | The Supplier will provide at least a minimum test data set (such as admin users, reference data such as user roles and access privileges, clinical reference dataset and any other applicable reference data sets etc.) for the Authority to be able to perform testing and assurance activities and to perform any basic admin tasks on the system test instance(s). The test data will contain Synthetic Data only. It will not contain any sensitive or personally identifiable Patient data. The Supplier will ensure that any maintenance releases/emergency patches deployed to their test environment does NOT erase any of the existing test data created by the Authority as this might adversely impact any ongoing testing and assurance. | MUST | Full Assessment Supporting evidence to include:
| N/A | N/A |
| TEST-ENV-10 | The Supplier will ensure that the system test instance(s) are capable of supporting concurrent testing by multiple external connecting system suppliers, where applicable under the Contracting Vehicle, allowing each to test independently of the others. | MUST | Full Assessment Walkthrough of the provisioned test environment(s)/system test instance(s) in an online session | N/A | N/A |
| TEST-ENV-11 | The Supplier will support backup and restore functions, to allow restoration of their system test instances(s) back to the original clean state or a previously backed-up state, if required. | MUST | Self-certification | N/A | N/A |
| TEST-ENV-12 | The Supplier will ensure that the performance of their system test instance(s) is comparable to the production system instance, albeit for the restricted volumes in the Test environment, in order to provide a realistic expectation of performance behaviour in the LIVE estate. | MUST | Self-certification | N/A | N/A |
| TEST-ENV-13 | The Supplier will provide all the necessary technical documentation for their provisioned API interfaces or bulk data interfaces to allow integration testing with external connecting supplier systems, where applicable to the Contracting Vehicle. | MUST | Full Assessment Supporting evidence to include:
| N/A | N/A |
| TEST-ENV-14 | The Supplier will provide the necessary guidance and access privileges to the Authority for interrogating Audit Trail records, system logs and API messaging logs which might aid the Authority in independent identification of any technical integration issues to minimise reliance on the Supplier. The Supplier will provide all the necessary support to the Authority in identifying and resolving any technical integration issues encountered during the testing and assurance activities. | MUST | Self-certification with Supporting Evidence
| N/A | N/A |
| TEST-ENV-15 | The Supplier will provide contact details of their system help desk or the team who are accountable for logging and resolution of any test environments(s) or system test instance(s) issues or incidents. The Supplier should advise the Authority of the process for logging the issues or incidents, and the expected timescales for resolution based on severity/criticality. | MUST | Self-certification | N/A | N/A |
Live Environments Requirements | ||||||
| LIVE-ENV-1 | The Supplier will support use of appropriate and limited Synthetic Data in their production (live) environment to support any smoke testing of the software build in the production (live) environment. It will be possible for the synthetic test patients to be clearly demarcated from the real patients. | MUST | Self-certification | N/A | N/A |
Further Requirements
Suppliers will not be required to meet any Further Requirements to achieve full compliance with the Testing Standard.
Roadmap
Items on the Roadmap which impact or relate to this Standard