/
Testing v1.0.1

Testing v1.0.1

11-04-2023

ID

S69

Version

1.0.1

Type

Overarching Standard

Status

Retired

Effective Date

Framework(s)

 

Description

Ensures that Suppliers' software delivery test processes are of sufficient quality and rigour.

 

It is essential that Solutions delivered under the Catalogue demonstrate a high-quality approach to software development and testing. Within the area of software testing, there are various established standards and many good industry practices in place. 

This standard does not seek to mandate adherence to a particular standard, it does however through compliance evidence look to establish that an appropriate level of quality and rigour are in place with respect to software testing, with ISO/IEC/IEEE 29119 providing a reference point.

According to ISO, Standards are "Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognised industrial, professional, trade associations or governmental bodies". They are guideline documents, therefore not compulsory unless mandated. The standards and external references within provide a clear benchmark for good industry practices which NHS Digital would expect a supplier to follow. As such, they provide a guide to suppliers on what level of quality NHS Digital expects from software development testing.

The International Software Testing Standard - ISO/IEC/IEEE 29119

An internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation:

  • ISO/IEC/IEEE 29119-1: Concepts & Definitions, published in September 2013

  • ISO/IEC/IEEE 29119-2: Test Processes, published in September 2013

  • ISO/IEC/IEEE 29119-3: Test Documentation, published in September 2013

  • ISO/IEC/IEEE 29119-4: Test Techniques, published in December 2015

  • ISO/IEC/IEEE 29119-5: Keyword-Driven Testing, published in November 2016


Within the context of the Catalogue, ISO/IEC/IEEE 29119 is used to provide a guidance checklist, used to establish suppliers entering a Framework demonstrate quality within their software testing. Context and Risk-based Approach ISO/IEC/IEEE 29119 part 1 states "Software testing is performed as a context-managed process.". It also states "A key premise of this standard is the idea of performing the optimal testing within the given constraints and context using a risk-based approach". These statements stand true for almost any testing approach.

Further Guidance 

http://context-driven-testing.com/

http://softwaretestingstandard.org/

http://www.satisfice.com/tools/htsm.pdf

 

 

Requirements

Testing Requirements

Requirement ID

Requirement Text

Level

Requirement ID

Requirement Text

Level

TEST-1

There will be organisational and project specific test processes underpinned by appropriate artefacts.

Example evidence could include:

  • Test Policy

  • Test Strategy

  • Test Plans

  • Test Specifications

  • Risk identification process as applied to Solution testing 

  • How is testing within development phases established, such as;

    • Unit,

    • Module

    • System

    • System Integration

    • Integration

    • System Integration

    • User Acceptance Testing

  • How testing automation is incorporated into the software build and distribution process, such as continuous build integration

  • Testing Qualifications within supplier organisation

  • Retrospective process improvement

MUST

TEST-2

There will be Test Management Processes covering test planning and design.

Example evidence could include:

  • Test Automation (automated scope as opposed to manual testing automated)

  • Test Regression and process for establishing coverage scope during a specific and subsequent Solution releases

  • Covering both Functional and non-Functional Testing

  • Techniques used - Risk-based, static/dynamic testing, testing heuristics, exploratory testing, smoke testing,...

MUST

TEST-3

There will be Test Management Processes covering test execution and management.

Example evidence could include:

  • Tooling used, explaining how results are captured,

  • Test Execution Management

  • Repeatability

  • Defect/Incident Management

  • Test Environment Management

MUST

TEST-4

There will be Test Management Processes covering test completion and Reporting.

Example evidence could include:

  • Comprehensive test results captured within test tooling

  • Incident management on completion

  • Quality Criteria for exiting testing

MUST