A product team reports that 194 of 200 test cases passed. Does that mean the product is 97% reliable, or that 97% of its users will succeed? Not necessarily. The test cases may repeat easy scenarios while missing one critical failure.
The gold standard of test engineering is evidence that identifies what a system was required to do, which conditions were tested and what those observations justify. This guide includes 21 rigorous controls, changed-context calculations, technical source references and an auditable chain from risk to acceptance.
Quick information: the chain of credible test evidence
| Element | Evidence to retain | Unjustified conclusion |
|---|---|---|
| Requirement | Named condition and acceptance rule | ‘Works well’ is measurable |
| Test design | Scenario, model and risk | More cases automatically mean better coverage |
| Environment | Build, fixture, calibration, data | Lab outcome proves production |
| Result | Observed value, expected value, anomalies | One successful trial means zero risk |
| Acceptance | Reviewed coverage, failures and authority | Green dashboard is release permission |
Worked example: 97% passed does not mean 97% reliable
| Fictional test set | Executed cases | Passed | Pass percentage |
|---|---|---|---|
| Set A | 200 | 194 | 97% |
| Set B | 150 | 149 | 99.33% |
Set B has a higher case pass proportion but might fail a more severe requirement or exercise less coverage. Pass rate is not a measured field reliability probability, and the compared suites differ. Keep the individual failure meanings and requirements visible.
Second example: a defined input interval
| Input | Hypothetical permitted range | Expected result |
|---|---|---|
| 39 | 40 to 80 inclusive | Reject |
| 40 | 40 to 80 inclusive | Accept |
| 80 | 40 to 80 inclusive | Accept |
| 81 | 40 to 80 inclusive | Reject |
Testing values at the boundary distinguishes inclusive from exclusive limits. Separate cases are still needed for missing, decimal or invalid inputs when they are possible. The invented rule has no connection to an actual safety limit.
Navigate all 21 test-engineering controls
Specify what must be proven
Chapters 1–5: Testing begins with a claim somebody needs to trust · Separate verification from validation · Make requirements objectively testable · Trace every test to the requirement and risk · Choose an appropriate test level.
Design meaningful test cases
Chapters 6–10: Define the test basis and oracle · Cover boundaries instead of only typical inputs · Use state-transition tests for multi-step systems · Build the test environment as part of evidence · Measurement uncertainty must be acknowledged.
Measure, automate and investigate
Chapters 11–15: Worked case: pass rate is not proof of reliability · Changed context: fewer failures can conceal greater risk · Test static documents and designs before execution · Performance testing requires meaningful workload · Security and privacy testing need an approved scope.
Assess reliability and suitability
Chapters 16–20: Automation helps repetition, not judgment by itself · Distinguish defect discovery from defect closure · Sampling imposes limits on reliability statements · Validate the human task with representative users · Release gates must distinguish evidence from permission.
Govern and learn
Chapters 21–21: Audit an entire test campaign and keep learning.
- Testing begins with a claim somebody needs to trust
- Separate verification from validation
- Make requirements objectively testable
- Trace every test to the requirement and risk
- Choose an appropriate test level
- Define the test basis and oracle
- Cover boundaries instead of only typical inputs
- Use state-transition tests for multi-step systems
- Build the test environment as part of evidence
- Measurement uncertainty must be acknowledged
- Worked case: pass rate is not proof of reliability
- Changed context: fewer failures can conceal greater risk
- Test static documents and designs before execution
- Performance testing requires meaningful workload
- Security and privacy testing need an approved scope
- Automation helps repetition, not judgment by itself
- Distinguish defect discovery from defect closure
- Sampling imposes limits on reliability statements
- Validate the human task with representative users
- Release gates must distinguish evidence from permission
- Audit an entire test campaign and keep learning
CONTROL 1 OF 21 · All controls
1. Testing begins with a claim somebody needs to trust
Test engineering asks whether an engineered product, component or process behaves as specified, and where it fails under meaningful conditions. A good test is not an elaborate demonstration of a product working once. It is a planned attempt to discover whether a clear claim survives observation, including conditions under which an expected response might fail.
A service that prints a correct receipt in a demonstration may still lose transactions after a power interruption. The gold standard tests both the ordinary task and important failure paths. The relevant claim, operating conditions, consequence of error and intended users come before the choice of instruments or test software. Systems Engineering connects this work to system requirements and operational needs.
CONTROL 2 OF 21 · All controls
2. Separate verification from validation
Verification asks whether an implemented product meets defined requirements or specifications; validation asks whether it serves the intended purpose in the expected environment. A prototype can conform to a poorly chosen specification, making verification evidence necessary but not sufficient to prove user value.
NASA’s systems-engineering guidance on verification and validation explains their distinct aims and how testing, inspection, analysis and demonstration can support them. In a school-access kiosk example, verification might confirm a scan response time; validation asks whether real students and staff can complete the borrowing process without losing records.
CONTROL 3 OF 21 · All controls
3. Make requirements objectively testable
An instruction to build a system that is ‘fast and reliable’ is an aspiration, not a test specification. Specify the transaction, population, load, environment, timing threshold and acceptance criterion. Avoid turning a single observed average into a requirement after test data are collected.
For a fictional library terminal, a proposed requirement could say that at least 95 of 100 valid transactions receive a confirmed acknowledgement within two seconds under a fixed test workload. The exact values are invented and not a product standard. The merit of the requirement is that the input, denominator and expected output can be observed and challenged.
CONTROL 4 OF 21 · All controls
4. Trace every test to the requirement and risk
A traceability matrix links stakeholder needs, requirements, design elements, test cases and result records. It reveals requirements with no verification and tests that do not support any meaningful claim. One requirement may need several tests if it includes normal and unusual conditions.
Record the requirement identifier and version, the responsible test, method, acceptance threshold, tested build and evidence. Do not count an outdated test as current compliance after a material design change. Documentation and Technical Writing reinforce the importance of consistent, auditable labels and instructions.
CONTROL 5 OF 21 · All controls
5. Choose an appropriate test level
Component, integration, system and acceptance testing expose different defects. A unit test can confirm that one function returns the intended value but not that multiple services preserve a transaction during failure. Integration tests reveal interface conflicts; operational validation asks whether the whole activity is usable.
The ISTQB Certified Tester Foundation Level syllabus distinguishes test levels, types, static reviews, design techniques and defect management. Testing strategy should be proportionate to the product’s risk and lifecycle rather than treating a large unit-test count as equivalent to end-to-end coverage.
CONTROL 6 OF 21 · All controls
6. Define the test basis and oracle
A test basis describes the material from which test expectations are derived, such as requirements, models, user journeys, interface contracts or hazard analysis. A test oracle is the rule or trusted information used to decide whether observed behaviour is correct. Without an oracle, a test can produce a number without establishing pass or fail.
If a calculator displays 12 after input, the tester needs the intended computation and allowed rounding before judging it. In data systems, agreement with a flawed old output is not necessarily correctness. ISO/IEC/IEEE 29119-1:2022 discusses the test basis, oracle and the impracticality of exhaustive testing; these distinctions help stop circular verification.
CONTROL 7 OF 21 · All controls
7. Cover boundaries instead of only typical inputs
Many defects appear near thresholds: empty fields, maximum length, zero values, just-before and just-after dates, or numeric extremes. Equivalence partitions reduce redundant cases, while boundary-value analysis targets the edges where errors are often concentrated.
If an invented input accepts integers 40 through 80 inclusive, test 39, 40, 80 and 81 as well as a representative interior value. The expected results are rejection, acceptance, acceptance and rejection. This is a teaching exercise, not an actual product specification. Note whether the field permits decimals, missing values or non-numeric characters; those require separate decisions.
CONTROL 8 OF 21 · All controls
8. Use state-transition tests for multi-step systems
Systems can behave differently depending on prior events. A payment request may be new, pending, confirmed, cancelled or uncertain after a connection failure. Merely testing a happy path misses invalid transitions, duplicated requests and operations resumed from an interrupted state.
Draw the possible states and permitted events before implementing cases. Ask what happens when the same request arrives twice, when a session expires or when a user retries without knowing whether the first action succeeded. A correct response needs a defined contract and safe recovery; test engineers should not invent behaviour that the product owner has not specified.
CONTROL 9 OF 21 · All controls
9. Build the test environment as part of evidence
Testing only has meaning in relation to its environment. Device model, operating system, network, software build, test fixture, temperature, permissions and dependencies can affect results. An environment that differs materially from use needs its limitations disclosed.
Keep the configuration, dataset version, test tool and relevant calibration record. Where destructive actions are possible, use isolated, authorised test systems and disposable data. A successful result in a simulator may support an engineering argument but cannot automatically establish behaviour on a different production configuration.
CONTROL 10 OF 21 · All controls
10. Measurement uncertainty must be acknowledged
A measured figure can vary because of instrument calibration, resolution, environment, procedure and sampling. A reading of 10.00 does not imply perfect accuracy merely because it has two decimal places. The uncertainty budget and traceability should match the decision and the type of measurement.
The US National Institute of Standards and Technology’s Technical Note 1297 provides guidance for evaluating and expressing measurement uncertainty. The method is not replaced by collecting a few readings and guessing a tolerance. Different uncertainty components can be correlated; use competent metrology practice for safety-critical or regulated results.
CONTROL 11 OF 21 · All controls
11. Worked case: pass rate is not proof of reliability
Suppose a fictional test run executes 200 cases, of which 194 pass and six fail. The observed case pass rate is 194/200 = 97%. That percentage describes this collection under its conditions. It is not a 97% probability that the product succeeds for customers, nor does it show that 97% of requirements are verified.
A hundred near-identical low-risk tests can dilute one untested severe hazard in a headline count. Report requirements covered, severity of failures, test categories and the consequences of unexecuted cases. The system might be unsuitable for release even when the numeric pass rate looks high.
CONTROL 12 OF 21 · All controls
12. Changed context: fewer failures can conceal greater risk
A second fictional test pack executes 150 cases and passes 149. Its pass rate is approximately 99.33%, higher than 97%. But imagine its one failed case involves incorrect handling of a critical financial transaction, while the earlier six failures concern nonessential display details.
One cannot rank product safety or readiness using those percentages alone. The case count, scenario selection, severity and coverage have changed. A sound risk-based review reports which requirements and hazards were exercised and the status of critical defects. Arithmetic should provoke questions, not close them.
CONTROL 13 OF 21 · All controls
13. Test static documents and designs before execution
Not all testing requires running software or energising hardware. Requirements reviews, design analysis, code reviews and model inspections can reveal ambiguity or contradiction early. Static evidence does not replace operational measurement, but it can prevent expensive tests of the wrong design.
Review interfaces, units, diagrams and dependencies with people responsible for downstream use. A line drawing with two incompatible dimension conventions is a defect even before manufacturing begins. Document review comments and corrective action, and verify that the revised artifact—not only the proposed edit—was actually approved.
CONTROL 14 OF 21 · All controls
14. Performance testing requires meaningful workload
Response time at one user and response time under heavy concurrent demand answer different questions. Load, stress, endurance and recovery tests expose different risks, including resource saturation, timeouts and gradually accumulating faults. Test engineering should define realistic workload profiles and success criteria.
Record the distribution of observed response times, errors, throughput and resource use; an average can hide failures at peak demand. Maintain distinct tests for ordinary service and beyond-normal limits, with safe authorisation. Do not treat a laboratory stress run as evidence of everyday production reliability without considering differences in environment.
CONTROL 15 OF 21 · All controls
15. Security and privacy testing need an approved scope
Testing may examine authentication, authorisation, information handling, logging and resilience against invalid inputs. But offensive security exercises can affect live customers if attempted without permission. Security testing must follow authorised methods and preserve privacy.
Use a safe test environment, defined scope and controls for sensitive data. Keep meaningful defect evidence without exposing passwords or real records. A high automated security score does not prove every vulnerability is absent. Cybersecurity develops the wider practice and responsible safeguards.
CONTROL 16 OF 21 · All controls
16. Automation helps repetition, not judgment by itself
Automated tests can run consistent scenarios repeatedly and help detect regression. They can also preserve wrong expectations or pass when the wrong system response is asserted. A flaky test may fail inconsistently because of timing, shared state or environmental conditions.
Treat the automation code and test data as maintained engineering artifacts. Review changes, version them and investigate flakiness rather than blindly rerunning until a green result appears. Manual and exploratory tests still matter when the product’s behaviour, user needs or risks are not fully captured by scripted checks.
CONTROL 17 OF 21 · All controls
17. Distinguish defect discovery from defect closure
A detected anomaly needs an identity, reproduction conditions, severity, owner and status. A developer stating ‘fixed’ means something different from a tester reproducing the original case on the corrected build and confirming it no longer fails. Significant changes can create new defects elsewhere.
Retesting checks the particular correction, while regression testing examines potentially affected prior behaviour. Record actual version and evidence for both. Defects should not vanish from reports simply because the schedule is tight. Accepted residual risks require accountable approval, not silent deletion from the defect log.
CONTROL 18 OF 21 · All controls
18. Sampling imposes limits on reliability statements
Even a sequence of successful tests cannot prove that future failures are impossible. Under strong simplified assumptions, zero observed failures in 100 independent comparable Bernoulli trials gives a rough 95% upper bound on an underlying failure probability of approximately 3/100, or 3%, using the common rule-of-three approximation.
Real devices or software requests are often dependent, differently stressed or drawn from a changing population. The approximation cannot be used without those assumptions, and it is not a product certification. A claim of very low field failure probability generally needs an appropriate reliability study with sufficient observations and representative conditions.
CONTROL 19 OF 21 · All controls
19. Validate the human task with representative users
Passing technical requirements may still leave a product confusing or inaccessible. Validation asks whether people can use it to reach the intended outcome in realistic conditions. The test may expose missing labels, unusable error messages or a failure mode not anticipated by the specification.
Use representative scenarios, ethically obtained consent and an appropriate setting. Record actual task completion, dangerous misunderstandings and feedback. Do not substitute a preference poll for observed use when the question is whether a task succeeds. Technical Communication connects the explanation to the user’s ability to act correctly.
CONTROL 20 OF 21 · All controls
20. Release gates must distinguish evidence from permission
A release decision uses technical evidence along with policy, safety, security and contractual requirements. A test report should state unresolved failures, test coverage, exclusions, environment and what risks may remain. It should not silently transform ‘no failures observed in these tests’ into ‘safe for any purpose’.
The responsible authority makes the acceptance decision under applicable governance. Test engineers provide traceable results and independent challenge. For high-stakes products, regulatory approvals and certified specialists may be required beyond a general test plan. Risk Management helps document residual risk but does not override legal duties.
CONTROL 21 OF 21 · All controls
21. Audit an entire test campaign and keep learning
A complete audit follows the claim through specification, strategy, test model, fixture, execution, result, defect and verification of correction. It distinguishes what was never tested from what passed, and can reconstruct every decision using recorded versions.
Use evidence to improve test selection, not merely to increase the volume of cases. The immutable Secondary 1 Mathematics Tutor Clementi reference provides a transferable educational approach: label the quantity, justify each step and test the answer in changed conditions. Its teaching philosophy is useful, but it is not a formal testing standard.
Operational audit: from requirement to independent review
| Claim | What a reviewer needs | Common trap |
|---|---|---|
| System complies | Requirement versions and named tests | Untraced passing tests |
| Results are credible | Real environment, instruments and uncertainty | Precision mistaken for accuracy |
| Risk is controlled | Severity and unresolved defect record | Pass rate hides a critical failure |
| Fix is verified | Specific retest and regression on correct build | Developer says ‘done’ without repeat |
| User task is workable | Representative validation and safety review | Requirements met but unusable |
| Release is justified | Appropriate accountable approval and exclusions | Test score treated as certification |
Frequently asked questions
What is test engineering?
A structured discipline for investigating whether technical products meet defined requirements and work as intended under relevant conditions.
Are verification and validation interchangeable?
No. Verification concerns meeting requirements; validation concerns fitness for intended use.
Is a 99% test pass rate proof of 99% reliability?
No. Test selection, severity, conditions and sampling differ from field reliability.
Can 100 successful tests prove a zero failure rate?
No. Finite samples cannot establish zero future failure risk.
What is a test oracle?
A criterion or trusted source used to determine whether an observed result is correct.
When are automated tests useful?
For repeatable checks under maintained expectations; they still require design, reviews and defect analysis.
What should an audit show?
Requirements, versions, test conditions, measured results, anomalies, corrections, exclusions and accountable acceptance decisions.
Standards and primary verification guidance
Authoritative context: ISO/IEC/IEEE 29119-1:2022 for software testing concepts; ISO/IEC/IEEE 29119-2:2021 for test processes; ISTQB Foundation Level syllabus; NASA distinction between verification and validation; and NIST Technical Note 1297 on measurement uncertainty. These sources have different scopes and do not certify this article or any fictional product. Numerical scenarios are constructed teaching examples.
Related eduKate Gold Standards
Engineering · Systems Engineering · Systems Thinking · Technology · Data Science · Statistics · Risk Management · Cybersecurity · Technical Writing · Technical Communication · Documentation · Evidence-Based Thinking · Education · Information