Choose your reading route
eduKateSG · Why Mathematics? · Software Reliability Growth: Defect Discovery Curves, Failure Intensity, Operational Profiles and Release Decisions
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 8–14
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 15–21
Check assumptions
See where design and calibration matter.Route 4 · Chapters 22–28
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 29–74
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important in software reliability growth? Software does not wear out like a bearing, yet failures can occur when latent faults meet particular inputs and states. Testing discovers defects, fixes change the code, and the operational profile determines how often faults are exposed. Mathematics tracks discovery intensity, residual uncertainty and prediction error.
This guide develops defect discovery curves, nonhomogeneous Poisson growth models, operational profiles and release decisions. It does not claim that mathematics alone makes software safe or that a fitted curve proves defect removal.
Full chapter index
Chapters 1–7 · Understand the question
Chapters 8–14 · Reproduce the example
Chapters 15–21 · Check assumptions
Chapters 22–28 · Explore applications
Chapters 29–74 · Study and verify
- Common misconception: no failures means no bugs
- Common misconception: every fix improves reliability
- Did You Know?
- Practice questions
- FAQ: should one model be mandatory?
- FAQ: can defects per line of code prove quality?
- Reporting checklist
- Useful next reading
- Conclusion
- Coverage is not one number
- Security failures
- Telemetry quality
- User heterogeneity
- Dependencies
- Rollback and recovery
- Change-point detection
- Forecast calibration
- Stopping tests
- Reproducibility
- Independent review
- Ethical communication
- Transfer to learning
- Final learning target
- NHPP count distribution
- Inter-failure times
- Parameter identifiability
- S-shaped models
- Covariate models
- Reliability objective
- Production feedback
- Canary evidence
- Regression suites
- Failure severity profile
- Uncertainty in release decisions
- Post-release calibration
- Documentation
- Worked check
- What students should transfer
- Data leakage
- Issue tracker bias
- Open-source exposure
- Machine learning systems
- Recovery metrics
- Uncertainty intervals
- Decision record
- Closing thought
Define the probability that software performs required functions without failure for a specified time and environment. The answer depends on version, inputs, state, platform and use. A failure count without an operational profile is not a portable property of the code.
A fault is a defect in code or design; an error is an incorrect internal state; a failure is externally observed departure from required behaviour. One fault can cause many failures, and many faults may never be activated in testing. Keep these levels distinct in data and claims.
Testing exposes faults, diagnosis identifies causes, and effective fixes remove or reduce them. Failure intensity may decrease as development progresses. New features, imperfect fixes and changing tests can add faults or alter exposure, so monotone improvement is not guaranteed.
Let N(t) count observed failures by execution time, test cases or another exposure. A software reliability growth model specifies a mean value function m(t)=E[N(t)] and an intensity dm/dt. Define the clock carefully: calendar time mixes staffing and machine use with execution exposure.
Some models assume a finite expected number a of faults that can eventually be exposed under the test process. As testing proceeds, the expected cumulative discoveries approach a. This is a modelling abstraction, not proof that the true fault count is fixed or that all faults are equally detectable.
A common exponential NHPP form is m(t)=a[1-exp(-bt)], where a is expected eventual discoveries and b is a detection-rate parameter. The failure intensity is ab exp(-bt), which declines as the discoverable pool is depleted under model assumptions.
Let a=100 and b=0.02 per test-hour. At 50 hours, expected cumulative discoveries are 100[1-exp(-1)], about 63.21. The instantaneous expected discovery intensity is 2exp(-1), about 0.7358 per hour. These are model expectations, not the exact number of remaining faults.
| Quantity | Value |
|---|---|
| a | 100 |
| b | 0.02/hour |
| Expected discoveries | about 63.21 |
| Discovery intensity | about 0.7358/hour |
| Model residual faults | about 36.79 |
Under the same finite-fault interpretation, expected undiscovered faults are a-m(t)=a exp(-bt), about 36.79 at 50 hours. This quantity is highly model dependent. It should not be reported as a known defect inventory or used without prediction checks.
Another NHPP model uses a logarithmic mean value function and allows failure intensity to decrease with cumulative failures. Different models can fit early data similarly and extrapolate differently. NASA research found that dynamic evaluation and predictive performance matter; no one curve is universally best.
The operational profile is the probability distribution of real inputs, functions and environments. Testing that oversamples easy or rare cases changes observed failure intensity. Weight or design tests to reflect intended use, while separately targeting high-consequence rare scenarios. Reliability always belongs to a profile.
Ten calendar days can contain different test exposure. CPU time, transactions, missions or demands may align better with activation opportunities. Record both development context and operational exposure. A faster test lab can show more failures per day while improving reliability per transaction.
A fix may fail to remove the fault, introduce a new one or change activation probability. Perfect-debugging models can be optimistic. Track fix verification, regressions and reopened defects. Extensions can model imperfect removal, but data must distinguish these outcomes.
Repeated manifestations of one fault can dominate counts. Decide whether the model concerns failure occurrences or unique defects. Both are useful: occurrences describe user burden; unique defects describe discovery and repair. Do not mix them in one curve without a mapping.
A failure intensity treats qualifying events alike, while release decisions may weight safety, security, data loss and inconvenience differently. Keep occurrence probability and consequence separate. One rare catastrophic failure can block release even when the aggregate curve declines.
A new fuzzer, integration stage or user cohort can increase discovery intensity because testing improved, not because code became less reliable. Mark test-regime changes and fit segments or covariates. A rise in findings can be healthy evidence acquisition.
Feature additions and refactoring alter the fault population. A growth curve across major versions may not represent one stable process. Link every failure and fix to commits, builds and environments. Reset, segment or use a richer model when the software boundary changes.
A release with no observed failure over exposure contributes evidence but does not prove zero faults. Production telemetry can be censored by logging, privacy and reporting delays. Define what is observable and how missing failures affect conclusions.
Estimate model parameters from ordered failure times or interval counts using likelihood methods. Small samples and early data create unstable asymptotes. Report intervals and parameter correlation. A fitted a far above observed defects may reflect genuine residual risk or weak identifiability.
Plot cumulative discoveries, inter-failure times and residuals. Compare observed counts with model predictions by interval. A visually smooth cumulative curve can hide dependence or test changes. Use held-out predictive checks because release decisions concern future exposure.
| Layer | Question |
|---|---|
| Operational profile | What use is represented? |
| Failure occurrences | What burden was observed? |
| Unique defects | What causes were discovered? |
| Fix verification | Was the cause removed? |
| Rolling forecast | Does the model predict future exposure? |
Compare several growth models on rolling forecast errors, not only in-sample fit. Preserve the selection rule to avoid choosing the winner after seeing outcomes. If models disagree materially, report the range and collect more representative exposure.
A model may predict near-term failures better than long-term residual faults. Evaluate the horizons that matter for release and support. Avoid extrapolating far beyond the test regime. Prediction intervals should include event randomness and parameter uncertainty.
CHAPTER 22 OF 74 · Explore applications
22. Connection to generic reliability growth
Reliability growth also studies test-fix-test learning. Software models often focus on defect discovery under an operational profile, while hardware power laws may represent recurrent repair events. Do not transfer interpretation automatically.
FRACAS provides versioned reports, causes, fixes and verified closure. Growth curves need that disciplined data to distinguish occurrences, unique faults and effective fixes.
Bayesian reliability can combine prior evidence with current failures. Similarity of software, profile and team matters. Do not count historical test hours as equivalent without justification.
A criterion may require predicted failure intensity below a target, no unresolved critical defects and adequate profile coverage. Prespecify the rule and escalation authority. The curve supports the decision; security, safety, usability and compliance evidence remain necessary.
Operational-profile weighting can under-sample rare high-consequence inputs. Use risk-based testing alongside representative testing. Report the distinction. A strong average reliability does not guarantee performance in a critical corner case.
Calculate the Goel–Okumoto curve for a=100, b=0.02 at 10, 50 and 100 hours. Plot cumulative discoveries and intensity. Then change b and compare. Discuss why the curve may jump when the test strategy changes even if the code does not.
Ask the student what the clock measures, what counts as one failure, and whether tests resemble real use. Encourage a distinction between a bug, a crash and a user-visible failure. These questions connect exponentials, rates and evidence to familiar technology.
CHAPTER 29 OF 74 · Study and verify
29. Common misconception: no failures means no bugs
Zero observed failures means none were seen under recorded exposure and detection. Uncovered paths, rare states and silent errors can remain. Quantify exposure and coverage, then make a bounded claim. Absence of evidence is not a guarantee.
CHAPTER 30 OF 74 · Study and verify
30. Common misconception: every fix improves reliability
A fix can introduce regression, shift a fault or change behaviour. Verify the target and run relevant regression tests. Track reopened defects. Growth should be supported by effective corrective action, not ticket closure alone.
The same software can have different reliability for two users because their operational profiles activate different paths. Reliability is therefore a relationship among code, environment and use, not a number stored inside the program.
1. Distinguish fault, error and failure. 2. Compute m(50). 3. Compute intensity. 4. Interpret residual count carefully. 5. Compare exposure clocks. 6. Define operational profile. 7. Explain imperfect debugging. 8. Design a rolling forecast check. 9. Draft release criteria. 10. Write a cautious conclusion.
No single growth family is reliable for every project. Use models supported by data and process knowledge, compare predictive performance and show sensitivity. Follow applicable organisational or IEEE practices when required.
CHAPTER 34 OF 74 · Study and verify
34. FAQ: can defects per line of code prove quality?
No. Language, generated code, architecture and counting rules differ. NASA lists defect and process indicators as possible evidence, not universal guarantees. Use several measures tied to reliability goals.
Report version, requirements, failure definition, exposure clock, operational profile, test regime, duplicate rule, severity, model equations, parameters, intervals, residuals, rolling forecasts, fixes, regressions, release rule and unresolved limits. Preserve data and code.
Use NASA’s current Software Reliability handbook chapter and the NASA study On the use and performance of software reliability growth models. Consult current IEEE 1633 requirements through authorised access where applicable.
Mathematics matters because software failures emerge from code, inputs, states and testing. Discovery curves and failure intensities can organise evidence when the clock, profile and fixes are explicit. Predictive validation and cautious release governance matter more than a smooth fitted curve.
Statement, branch, path, requirement and state coverage answer different questions. High coverage in one metric does not prove realistic input combinations. Connect coverage to the operational and risk profiles.
Adversarial inputs are not sampled like ordinary use. Treat threat models, abuse cases and vulnerability discovery separately from random operational failures. Do not average them away.
Logging can omit silent corruption, client crashes or privacy-protected events. Measure detection and reporting delay. Reliability estimates inherit telemetry limits.
Different devices, regions and workflows create different profiles. Stratify or model covariates before pooling. A global average can hide a high-risk subgroup.
Distributed services fail through shared databases, networks and deployments. Component failure independence is rarely automatic. Preserve incident links and cascade paths.
A failure may be mitigated by retry, failover or rollback. Define whether the reliability outcome is any internal error, user-visible interruption or unrecovered mission loss. Time to recovery adds availability.
A deployment or test tool can shift discovery intensity abruptly. Mark known changes and use statistical change-point methods cautiously. Post-hoc segmentation can overfit.
Across releases, compare predicted failure counts with realised counts in bins. Retain original predictions. Calibration improves only when definitions and profiles stay comparable.
Ending testing when the curve first falls below a threshold can exploit noise. Prespecify persistence, exposure and independent checks. Repeated looks affect decision error even when the model is unchanged.
Archive failure timestamps, build IDs, profile weights, model code and selection results. A release decision should be reconstructable from the evidence available then.
A second analyst should reproduce parameters and rolling forecasts. Software assurance reviewers should inspect severity, profile coverage and unresolved critical paths.
Do not translate a low predicted failure intensity into guaranteed safety. Users face consequences from rare and systematic failures. State scope and escalation routes.
Practice tests reveal errors under a study profile just as software tests reveal faults under inputs. Students can see why more varied questions improve evidence, though people and code are not identical.
A student should compute a discovery curve, explain operational profiles and separate a fitted decline from proof of residual defect count. That combination is mathematical and critical thinking.
Under an NHPP, failures in an interval have a Poisson distribution with mean m(t2)-m(t1). This supports predictive count ranges, not just expected values. Overdispersion or clustered incidents can reveal that the Poisson assumption is inadequate.
As intensity declines, expected gaps between failures tend to grow, but gaps are noisy and depend on exposure. Plot them against cumulative execution time. A long gap during low testing activity is not evidence of improved reliability.
Early in testing, a and b can trade off: a large eventual fault count with fast detection may resemble a smaller count with slower detection. Report joint uncertainty and avoid firm residual-fault claims until the curve bends sufficiently.
Some models allow discovery intensity to rise initially as testers learn before declining as faults are removed. This can fit staged testing. The shape should reflect process evidence, not merely improve fit. Compare future predictions with exponential models.
Test team size, coverage, input novelty and code churn can change discovery intensity. Prespecified covariates help explain regime changes. Measurement quality and collinearity limit interpretation; association with findings does not prove causal quality.
A target must specify failure definition, mission duration and operational profile. “Five nines” without those conditions is incomplete. High availability from fast recovery is not the same as a low probability of failure.
After release, combine telemetry with exposure by version and profile. Delayed upgrades create several populations. Do not pool a fixed build with rapidly changing canary releases. Preserve rollout percentages and rollback intervals.
A canary release can limit consequence while collecting representative exposure. Selection of users and traffic affects generalisation. Predefine promotion criteria and analyse dependence from shared services. A quiet canary is not proof for all users.
A fixed regression suite supports comparability but can become uninformative as developers optimise to it. Add risk-based and exploratory tests while retaining a stable benchmark. Report which exposure belongs to each suite.
Track critical, major and minor failures separately. A falling total can coexist with an unresolved critical path. Growth models may be fit by severity if counts support it; otherwise maintain qualitative constraints in release governance.
Calculate the probability that future failures exceed an operational threshold under each plausible model. If the decision is unstable, extend representative testing or reduce exposure. Do not hide disagreement by averaging incompatible models without rationale.
Compare predicted and observed counts after every release using the original profile and model version. Analyse systematic optimism or pessimism. Calibration is a programme-level learning process, not a one-release verdict.
Store data extraction, duplicate rules, model candidates, selection criteria, parameterisation and release memo. Future teams need to distinguish genuine process improvement from changed definitions. An auditable chain also supports incident review.
At t=0, the Goel–Okumoto mean is zero and intensity is ab=2 per hour. As t grows, the mean approaches 100 and intensity approaches zero. These limiting checks catch sign and unit errors in code.
The transferable insight is that evidence depends on what is tested and how often real users activate a path. Students should connect exponential curves to coverage, version control and prediction checks rather than treating a fitted decline as proof.
Do not use future failures to choose a model and then report its forecast accuracy on the same interval. Use rolling origins or held-out releases. Preserve every candidate and selection rule.
Issue trackers reflect reporting and triage, not all user failures. Link telemetry and support data carefully, deduplicate incidents and record missing populations. A clean tracker can coexist with silent failures.
Public code may receive uneven scrutiny across modules. Contributor activity is not operational exposure. Model failure activation and fix verification, not popularity alone.
Model and data changes can create failures without code defects. Version training data, model weights and evaluation sets. Operational drift belongs in the profile and monitoring plan.
Pair failure intensity with time to detect and restore when service continuity matters. Fast recovery can improve availability while users still encounter frequent failures. Keep the quantities separate.
Publish predictive intervals for future counts and sensitivity across model families. Parameter precision within one selected curve understates model uncertainty. Explain what new evidence would narrow it.
Archive the release candidate, model snapshot, unresolved defects, profile coverage and approval. Later outcomes should update learning without rewriting what was known at release.
Software reliability growth is a disciplined conversation between testing, fixes and real use. Mathematics makes the evidence auditable; it does not remove the need for engineering judgement.
Extended practice laboratory
Practice case 1: Mean discoveries
Setup: Use a=100,b=0.02,t=50.
Task: Calculate 100(1-exp(-1)).
Checking target: Obtain about 63.21.
Practice case 2: Intensity
Setup: Use ab exp(-bt).
Task: Calculate at 50 hours.
Checking target: Obtain about 0.7358/hour.
Practice case 3: Residual count
Setup: Use a exp(-bt).
Task: Calculate at 50 hours.
Checking target: Obtain about 36.79 under the model.
Practice case 4: Clock
Setup: Compare execution and calendar time.
Task: Choose activation exposure.
Checking target: State the unit.
Practice case 5: Profile
Setup: Assign input probabilities.
Task: Weight test exposure.
Checking target: Keep rare critical cases separate.
Practice case 6: Duplicates
Setup: Separate occurrences from unique defects.
Task: Build two curves.
Checking target: Do not mix outcomes.
Practice case 7: Imperfect fix
Setup: Add one regression.
Task: Update the event ledger.
Checking target: Ticket closure is not verified removal.
Practice case 8: Change point
Setup: Introduce a new fuzzer.
Task: Segment discovery intensity.
Checking target: A rise can reflect better testing.
Practice case 9: Rolling forecast
Setup: Fit on early data.
Task: Predict the next interval.
Checking target: Evaluate out of sample.
Practice case 10: Model comparison
Setup: Compare two NHPP forms.
Task: Use forecast error.
Checking target: Do not choose only by in-sample fit.
Practice case 11: Release rule
Setup: Combine intensity, critical defects and coverage.
Task: Prespecify thresholds.
Checking target: Model output does not authorise release alone.
Practice case 12: Conclusion
Setup: State version, profile, model and uncertainty.
Task: Separate faults from failures.
Checking target: Name validation horizon.