VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

The Gold Standard of Sequential Analysis

Three secondary students review open books and notes together in a classroom.

Sequential analysis studies how to learn and make decisions while observations are still arriving. Its gold standard is an evidence rule that remains valid under the way the data are inspected and the experiment is stopped. Watching a running result is easy; preserving an honest interpretation after choosing when to stop is the central challenge.

The subject matters whenever waiting has a cost: inspecting manufactured items, comparing simulated designs, monitoring a service or deciding whether further measurement is worthwhile. A sequential procedure can sometimes reach a supported conclusion sooner than a fixed-size experiment. It can also remain inconclusive, require many observations or fail when its assumptions do not describe the data. Early stopping is an option within a design, not a promise that every experiment will finish quickly.

This guide explains the observation model, stopping times, likelihood updates, explicit decision boundaries, repeated-look error control, confidence sequences, delayed outcomes, changing conditions and practical reporting. Its original examples use simple binary observations so that the arithmetic and limitations can be checked directly. The emphasis is on understanding what a conclusion means at the moment it is made.

Start with the Gold Standard master for the broader collection, the Education parent for learning that transfers, and the How X Works hub for supporting explanations. The Ranking and Selection root develops the related problem of choosing among noisy alternatives. Sequential analysis asks how conclusions can remain defensible as the observation count itself becomes part of the decision.

CHAPTER 1 OF 14 · Define observations and stopping

1. Specify what arrives and what can be decided

Back to contents

Suppose a test produces observations X₁, X₂ and so on. An observation might indicate whether an item passes inspection, whether a simulated project finishes on time or whether a measured value exceeds a threshold. Before choosing a method, define the unit of observation and the quantity of interest.

For a binary outcome, write X = 1 for success and X = 0 for failure. A simple model assumes independent observations with a constant success probability p. That model is useful for teaching, but neither independence nor constancy follows merely because the data occupy separate spreadsheet rows.

Separate scientific questions from available actions

An experiment might ask whether p equals one value or another. An operational decision might instead ask whether performance is good enough to deploy a system. Those are different questions. A model comparison between p = 0.50 and p = 0.75 does not automatically answer whether p exceeds a service requirement of 0.90.

List the available actions: continue collecting, stop in favour of one specified hypothesis, stop in favour of another, or end without a supported conclusion. Include a maximum budget or deadline if one exists. “Continue until certain” is not a workable plan because statistical evidence rarely provides certainty and some cases are intrinsically difficult to distinguish.

Also define when outcomes become available. An item can be enrolled in an experiment before its eventual result is known. Counting enrolments as completed observations can give a misleading picture of how much information has arrived. Different delays across outcome types can distort the apparent running success rate.

A useful starting statement is: “Each completed observation represents ___; the target is ___; the model assumes ___; we inspect at ___; and our possible decisions are ___.” These details are not administrative extras. They determine the probability calculation to which a stopping rule must apply.

Contents · Next chapter

CHAPTER 2 OF 14 · Define observations and stopping

2. Understand why repeated inspection changes the problem

Back to contents

A fixed-sample test is calibrated for its specified sampling and analysis plan. If an analyst repeatedly applies that test at different sample sizes and stops the first time it gives a desirable result, the overall procedure is different. Its error probability must be assessed for the entire sequence of opportunities to stop.

The same issue appears with ordinary confidence intervals. An interval with 95% coverage at a predetermined time does not automatically provide a 95% guarantee that every interval in a running display covers the parameter. A statement about one time and a statement about all inspected times concern different events.

A simple comparison of events

Imagine four completely independent experiments, each having a 0.05 false-positive probability under its null hypothesis. The probability of at least one false positive is 1 − 0.95⁴, approximately 0.1855. That calculation illustrates the accumulation of opportunities, but it is not the error rate for four nested looks at one accumulating dataset. Nested looks share observations and are dependent.

For dependent looks, the actual joint probability depends on the procedure and model. The independence formula should not be copied simply because there are four analysis times. What remains true is that the desired protection must concern the event of an erroneous conclusion at any permitted look.

Looking at data is not inherently invalid. The problem is using a guarantee derived for one procedure to describe a materially different procedure. Planned group analyses, sequential tests and confidence sequences are ways of making the inspection and stopping decisions part of the inferential design.

An effective repair begins before calculating another p-value. Write down the actual rule used to inspect, continue and stop. If that rule was changed during the experiment, preserve the changes. A running analysis becomes interpretable when its decision process is visible rather than reconstructed afterward around the first favourable result.

Previous chapter · Contents · Next chapter

CHAPTER 3 OF 14 · Define observations and stopping

3. Define a stopping time without looking into the future

Back to contents

A stopping time is a rule for deciding whether to stop using information available up to the current point. If the rule says stop once a running score reaches a fixed boundary, the decision can be made from the observations already received. It does not require knowing later outcomes.

By contrast, “stop at the sample size that eventually produces the most impressive result” uses future knowledge when identifying the stopping point. It is a retrospective selection of a result. A graph can make that selected point look like an ordinary planned endpoint even though the decision required seeing the rest of the path.

Distinguish a data rule from a calendar rule

A fixed calendar deadline can be compatible with a sequential design. The experiment may stop early if an evidence boundary is reached and otherwise end at the deadline. The conclusion at the deadline must follow the rule specified for that event. If no boundary has been reached, a forced declaration of success adds a different decision rule.

Operational stopping and statistical stopping can also differ. An equipment failure may end data collection without establishing either hypothesis. A safety concern may justify suspending an experiment without providing a calibrated statistical conclusion about the original target. Record the reason for stopping rather than treating every ending as evidentially equivalent.

In software, preserve a state containing completed observation count, current evidence score, thresholds, status and termination reason. Once a stop decision is recorded, later observations should not silently overwrite the endpoint. They may support a separately documented follow-up analysis.

This is particularly important when data arrive in batches. A batch can contain observations that would have crossed a boundary partway through if inspected individually. Decide whether inspection occurs after each observation or only after a batch. The procedure is defined by what was actually observable and actionable, not by a convenient reconstruction after all the data are available.

Previous chapter · Contents · Next chapter

CHAPTER 4 OF 14 · Build and check evidence

4. Build a likelihood ratio from first principles

Back to contents

Consider two simple hypotheses for independent binary observations: H₀ says p = 0.50 and H₁ says p = 0.75. Here “simple” means each hypothesis specifies a complete distribution for an observation. The two values are teaching assumptions, not estimates from a real service.

For a success, the probability under H₁ divided by that under H₀ is 0.75/0.50 = 1.5. For a failure, the ratio is 0.25/0.50 = 0.5. A success therefore multiplies the evidence ratio by 1.5; a failure multiplies it by 0.5.

Multiply evidence only under the stated model

After s successes and f failures, the likelihood ratio is L = 1.5ˢ × 0.5ᶠ. It compares the probability of the observed sequence under H₁ with its probability under H₀. Starting before any data, L = 1. Values above one favour H₁ relative to H₀; values below one favour H₀ relative to H₁.

For the sequence success, success, failure, the successive values are 1.5, 2.25 and 1.125. Two successes followed by one failure do not leave evidence of 2.25. The failure must be included. Evidence can reverse direction as new observations arrive.

The final likelihood ratio depends on counts in this independent constant-probability model. The stopping decision can still depend on the path because a boundary might be crossed before the final count is reached. Saving only final totals loses information about when the rule would have stopped.

These multiplications do not apply unchanged when observations are dependent or when the success probabilities change over time. In a more general model, the relevant update uses conditional probabilities given the history. Treating a repeated copy of the same report as an independent observation can multiply the apparent evidence without adding new information. The model supplies the interpretation; arithmetic alone cannot rescue an unsuitable model.

Previous chapter · Contents · Next chapter

CHAPTER 5 OF 14 · Build and check evidence

5. Work through explicit stopping boundaries

Back to contents

For the teaching example, use an upper boundary of 20 and a lower boundary of 0.05. Stop in favour of H₁ when L ≥ 20. Stop in favour of H₀ when L ≤ 0.05. Continue when the ratio is between the two boundaries. If a budget limit arrives first, report the unresolved status rather than inventing a crossing.

Eight consecutive successes give L = 1.5⁸ = 25.62890625. Seven give approximately 17.0859. Thus this path first crosses the upper boundary at observation eight. Five consecutive failures give L = 0.5⁵ = 0.03125, whereas four give 0.0625. That path first crosses the lower boundary at observation five.

A mixed path needs more evidence

Take six successes, then one failure, then four successes. The ratio after the first six is 11.390625. The failure reduces it to 5.6953125. The next three successes bring it to approximately 19.2217, still below 20. The fourth brings it to approximately 28.8325, crossing at observation eleven.

The boundary is not a target to round toward. A ratio of 19.22 does not satisfy L ≥ 20. Likewise, the score does not have to land exactly on 20; discrete updates can overshoot it. Any exact error calculation must respect the actual attainable values and stopping rule.

The upper boundary says that the observed sequence is at least twenty times as likely under the specified H₁ as under H₀. It does not say that H₁ has posterior probability 95%. A posterior statement additionally requires prior odds and an appropriate probabilistic hypothesis framework.

The example deliberately uses conservative reciprocal boundaries with a direct probability argument in the next chapter. Other sequential probability ratio designs use different boundaries and calibrations. Do not swap boundaries between procedures while retaining claims derived for the original design.

Previous chapter · Contents · Next chapter

CHAPTER 6 OF 14 · Build and check evidence

6. Explain the error protection behind the boundaries

Back to contents

Under H₀, the expected one-step multiplier is 0.50 × 1.5 + 0.50 × 0.5 = 1. With independence, the nonnegative likelihood-ratio process has conditional expected next value equal to its current value. In probability language, it is a nonnegative martingale starting at one.

A maximal inequality for this process bounds the probability that it ever reaches 20 under H₀ by 1/20 = 0.05. Thus deciding H₁ at the upper crossing has false-positive probability at most 0.05 under the specified H₀, even though the crossing time is not fixed in advance.

Check the opposite error separately

Under H₁, consider the reciprocal ratio 1/L. Its success multiplier is 2/3 and its failure multiplier is two. The expected multiplier is 0.75 × (2/3) + 0.25 × 2 = 1. The same nonnegative-martingale argument bounds the probability of ever reaching 20 for 1/L by 0.05. That event corresponds to L reaching 0.05, so the probability of incorrectly choosing H₀ under the specified H₁ is also at most 0.05.

These are upper bounds, not statements that both errors equal exactly five percent. Stopping at the opposite boundary can prevent later crossings, and discrete overshoots can make the bounds conservative. The argument protects each endpoint model separately. It does not declare what happens when the true p is some third value.

For a short intuition, imagine stopping a nonnegative fair-value process at its first upper crossing or at a finite horizon. Its expected stopped value remains one under the relevant conditions. On crossing paths it is at least twenty, so those paths cannot have probability greater than one-twentieth. Extending the horizon yields the ever-crossing bound.

This reasoning shows why the boundaries and probability model belong together. A score with the same numerical appearance but no corresponding conditional-expectation property does not inherit the result. Calling a dashboard number “evidence” is insufficient to make repeated stopping valid.

Previous chapter · Contents · Next chapter

CHAPTER 7 OF 14 · Track paths and uncertainty

7. Use logarithms and preserve the whole path

Back to contents

Likelihood products can become extremely large or small. Instead of multiplying ratios indefinitely, store their logarithms. For our binary example, log L = s log(1.5) + f log(0.5). Each success adds approximately 0.405465; each failure adds approximately −0.693147.

The upper log boundary is log(20), approximately 2.995732, and the lower is log(0.05), approximately −2.995732. Compare the accumulated log score directly with these values. This avoids numerical underflow and overflow without changing the mathematical decision rule.

Do not erase unfavourable increments

A restart after every failure would create a different process. If one keeps searching for eight consecutive successes and ignores everything before the latest streak, the earlier likelihood-ratio guarantee no longer describes the resulting rule. A reset is an analytical choice that needs its own justification.

Preserve every completed observation, timestamp, model version and update. Record whether a value was corrected, missing or excluded. If a correction changes a boundary crossing, retain both the original decision record and the corrected analysis. Rewriting history conceals how the operational action was actually taken.

In testing code, compare product and log implementations on short sequences where both are numerically safe. Verify that eight successes stop at eight, five failures stop at five, and the mixed sequence in Chapter 5 stops at eleven. Check that extending a stopped sequence does not silently change the recorded stopping time.

The existing specialist article on sequential evidence, likelihood monitoring and decision boundaries develops these concerns in an intelligence-analysis setting. The statistical root here supplies the mathematical distinction between an accumulating evidence process and a score whose reset or selection rules change whenever the story becomes inconvenient.

Previous chapter · Contents · Next chapter

CHAPTER 8 OF 14 · Track paths and uncertainty

8. Plan a finite number of analyses with an error budget

Back to contents

Not every experiment needs inspection after every observation. A design may permit analysis after four predetermined batches. One transparent method allocates a total error budget α across those looks. For example, allocate 0.01, 0.01, 0.01 and 0.02, which sum to 0.05.

If the probability of an erroneous rejection at look j is at most its allocation under the null, the probability of any erroneous rejection across the four looks is at most the sum, 0.05. This union-bound argument does not require the look events to be independent.

The individual analyses still need valid foundations

The allocation is not itself a complete test. Each analysis must provide the stated error bound under the actual sampling plan. Changing eligibility, outcome definitions or sample sizes in a way not covered by the test can invalidate the pieces before they are added together.

For a simple planned-look design, one can conceptually define all four analyses on a full potential data stream even if collection stops after an earlier rejection. The event of rejecting at any observed look is contained in the union of the four possible rejection events. This explains why early stopping need not defeat a correctly constructed planned-look argument.

Adding a fifth unplanned look with another ordinary five-percent test does not fit inside the original budget. Nor does applying the entire budget separately to many outcomes while reporting only the favourable one. Comparisons across outcomes and opportunities across time both contribute to the overall decision process.

Finite-look designs can be easier to communicate and operate than continuous monitoring. They also impose a tradeoff: a strong result arriving just after a look may wait until the next planned analysis. Choose analysis times that match operational needs and justify the chosen statistical method. Convenience is a valid design consideration, but it should be visible rather than disguised as a mathematical necessity.

Previous chapter · Contents · Next chapter

CHAPTER 9 OF 14 · Track paths and uncertainty

9. Distinguish confidence sequences from ordinary intervals

Back to contents

A confidence sequence is a sequence of intervals designed to cover the target parameter simultaneously over all permitted times with a stated probability. If its assumptions hold, selecting a stopping time does not destroy that simultaneous coverage. This differs from a collection of ordinary intervals that each have coverage only at their own fixed time.

The research of Howard, Ramdas, McAuliffe and Sekhon develops time-uniform confidence sequences under specified conditions. The phrase “time-uniform” concerns the probability guarantee across time. It does not mean that all distributions, data-collection schemes or parameter changes are allowed.

Construct a deliberately conservative example

For independent Bernoulli observations with constant mean p, a standard bounded-observation concentration bound gives P(|p̂_n − p| ≥ r) ≤ 2 exp(−2nr²). Here p̂_n is the sample proportion at fixed n. Allocate α_n = α/[n(n+1)] to time n. These allocations sum to α because 1/[n(n+1)] = 1/n − 1/(n+1).

Set r_n = √{log[2n(n+1)/α]/(2n)}. The fixed-time failure probability is then at most α_n. Summing over all positive integers n bounds the probability of any interval failure by α. Therefore the intervals [p̂_n − r_n, p̂_n + r_n], clipped to [0,1], form a conservative confidence sequence under these assumptions.

This construction makes the accounting visible. It is not presented as an efficient state-of-the-art interval. At α = 0.05, r_100 is approximately 0.254 and r_1000 approximately 0.0936. Even after one hundred observations, the interval can be wide. Simultaneous protection is a stronger demand than coverage at one prespecified time.

A narrower ordinary interval does not necessarily outperform this sequence for the same task because it may offer a different guarantee. Compare methods only after aligning their targets, assumptions and permitted stopping rules. Precision without the required coverage is not an equivalent product.

Previous chapter · Contents · Next chapter

CHAPTER 10 OF 14 · Check assumptions and conclusions

10. Respect dependence, delays and incomplete outcomes

Back to contents

An independent-observation model can fail in several ways. Consecutive items may come from the same production batch. Users may contribute multiple correlated responses. A sensor may retain calibration error across many readings. In each case, the apparent observation count can exceed the amount of independent information.

Duplicating one binary observation ten times illustrates the danger. If the original is a success, multiplying 1.5 ten times treats it as ten independent successes and produces L ≈ 57.665. Yet only one outcome was observed. The numerical boundary crossing is an artefact of duplication, not accumulated evidence.

Define the history used for each update

A valid likelihood update under dependence uses probabilities conditional on the past. If a second report is an exact copy of the first, then after observing the first the copy is certain under both hypotheses. Its conditional likelihood multiplier is one, so it contributes no additional evidence.

Delayed observations raise another problem. Suppose quick outcomes tend to be successes and failures take longer to resolve. Monitoring only completed results early in the experiment can overrepresent successes. A procedure calibrated for an unbiased sequence of complete outcomes does not automatically handle outcome-dependent completion times.

Specify whether analyses wait for cohorts to mature, model the delay mechanism or use another justified approach. Record enrolled, completed and unresolved counts separately. Do not code missing results as failures or remove them as successes merely to simplify the update.

For practical diagnosis, inspect clusters, repeated identifiers, timestamp patterns and completion rates by relevant category. These checks do not prove independence, but they can expose obvious violations. The important repair is to reconnect the sampling model to the actual observation mechanism. Collecting additional rows from the same flawed process may make the unsupported conclusion look stronger without supplying the missing information.

Previous chapter · Contents · Next chapter

CHAPTER 11 OF 14 · Check assumptions and conclusions

11. Separate stable-parameter testing from change detection

Back to contents

Our binary model assumes one constant p for the whole stream. A monitoring problem may instead ask whether a process that was initially stable has changed. Those are different statistical tasks. Evidence accumulated before a change can dominate or dilute evidence afterward, depending on the method.

Suppose a service has p = 0.50 for a long period and later shifts toward 0.75. The likelihood ratio comparing two constant models over the entire history is not automatically a well-designed detector for that transition. Both constant hypotheses misdescribe part of the stream.

A reset changes the question

Starting a fresh analysis after a suspected change may be reasonable, but selecting the restart time after examining the data introduces another decision. Repeatedly trying many restart points and reporting whichever gives the strongest evidence adds opportunities for false alarms. A change-detection design must account for the monitoring and restart mechanism it actually uses.

Also distinguish statistical change from practical harm. A small shift can become detectable with enough information while having little operational importance. Conversely, a severe event can require immediate operational attention before a statistical boundary is reached. Evidence rules support decisions; they do not erase responsibility for interpreting consequences.

The Law of Large Numbers explanation provides an important foundation: averages stabilize under appropriate assumptions, but a large dataset does not automatically represent a single stable process. Combining old and new regimes can estimate a historical mixture rather than today’s performance.

When conditions change, name the target explicitly. Are you estimating the entire-period average, the current rate, a shift size or the time of change? Then select a method designed for that target. A valid answer to one of these questions can be misleading when presented as the answer to another.

Previous chapter · Contents · Next chapter

CHAPTER 12 OF 14 · Check assumptions and conclusions

12. Interpret a stopped result without overclaiming

Back to contents

Crossing the upper boundary in our example supports p = 0.75 relative to p = 0.50 under the stated model. It does not prove that the true p equals 0.75, that every lower value is excluded or that deployment is worthwhile. There may be alternatives, such as p = 0.65, that were never represented by either simple hypothesis.

If the true p is 0.65, both simple hypotheses are false. The procedure still produces a numerical path, but the endpoint-specific error statements from Chapter 6 do not describe this third distribution. A composite hypothesis or estimation procedure is needed for a broader claim about a range of p values.

Report evidence, uncertainty and action separately

State the model comparison, stopping rule, observed stopping time, score at stopping, completed observation count and reason for ending. Then explain the operational decision and the consequences considered. A boundary crossing and a decision to implement are related events, not synonyms.

Estimation after stopping also deserves care. The sample proportion at a data-dependent stopping time can be selected by the stopping rule. An ordinary fixed-n interval applied afterward need not have its advertised coverage. A suitable confidence sequence, a design-specific analysis or fresh independent data can support an appropriately defined estimate.

If the maximum budget is reached without crossing, “inconclusive under this rule” is a legitimate result. It is not evidence that the hypotheses are identical, nor permission to lower the threshold retrospectively. Report what additional evidence or revised question would be useful.

The Ranking and Selection guide makes a parallel distinction between a chosen candidate and a well-supported performance estimate. Sequential analysis adds the same discipline to the timing of decisions: explain what the procedure establishes, what remains unresolved and which assumptions would need revision before the conclusion could travel to a different setting.

Previous chapter · Contents · Next chapter

CHAPTER 13 OF 14 · Transfer and continue learning

13. Transfer checks with worked answers

Back to contents

First exercise: with H₀: p = 0.50 and H₁: p = 0.75, what happens after three successes followed by two failures? The ratio is 1.5³ × 0.5² = 0.84375. It favours H₀ slightly relative to H₁ and lies between the boundaries, so the specified rule continues. A majority of successes does not necessarily favour the higher-probability model.

Second exercise: change H₁ to p = 0.60 while retaining H₀: p = 0.50 and upper boundary 20. A success multiplier is now 1.2, and a failure multiplier is 0.8. Seventeen consecutive successes are needed to cross the upper boundary because 1.2¹⁶ ≈ 18.488 and 1.2¹⁷ ≈ 22.186. The old eight-success rule belongs to a different pair of hypotheses.

Change the error requirement carefully

Third exercise: retain the original hypotheses but require the conservative upper-crossing bound to be at most 0.01. The upper threshold becomes 100. Twelve consecutive successes cross it because 1.5¹¹ ≈ 86.498 and 1.5¹² ≈ 129.746. This changes the upper error protection; it does not by itself specify a new lower boundary or a complete revised design.

Fourth exercise: eight recorded successes come from four independent outcomes, each duplicated once. Does the original ratio justify stopping? No. The independence model is wrong. Under an explicit duplication model, the four unique successes contribute 1.5⁴ = 5.0625, while their exact copies add multipliers of one.

Fifth exercise: an ordinary 95% interval is narrow at the moment an analyst decides to stop. Is it therefore a valid 95% interval for the sequential procedure? Not automatically. Precision and coverage are different properties. The interval must be justified for the selection and stopping mechanism used.

For each exercise, identify which ingredient changed before calculating: hypotheses, boundary, information content or inferential target. That habit is more transferable than memorizing a particular threshold.

Previous chapter · Contents · Next chapter

CHAPTER 14 OF 14 · Transfer and continue learning

14. Learning routes, common questions and sources

Back to contents

How should a beginner practise?

Implement the binary likelihood ratio using both products and logarithms. Reproduce the short paths in this article before generating random streams. Then simulate many complete experiments under p = 0.50 and p = 0.75, recording which boundary is reached, when it is reached and whether a chosen maximum horizon ends the run first. Keep unresolved runs visible in the results.

Does simulation prove the error guarantee?

No. Repeated simulations estimate behaviour for the tested models and settings, with Monte Carlo uncertainty. They are excellent for finding coding errors and understanding sample-size variability. The probability argument establishes the stated bound under its assumptions. A program that appears to behave well in a limited simulation study does not thereby gain a universal guarantee.

Is sequential analysis always more efficient?

No. Efficiency depends on the hypotheses, true distribution, stopping design, cost per observation and target error control. Easy cases can stop early; ambiguous cases may require substantial effort. A fixed maximum duration can also create an unresolved category that must be planned for.

Is a likelihood ratio a posterior probability?

No. A likelihood ratio compares the probability of the observations under specified models. Posterior odds equal prior odds multiplied by a likelihood ratio within an appropriate Bayesian model. Converting the ratio into a posterior probability without specifying prior odds changes the meaning of the number.

Which primary sources support further study?

The U.S. Census Bureau working paper Exact Calculations for Sequential Tests Based on Bernoulli Trials provides a research route into evaluating sequential binary tests. Howard and colleagues’ confidence-sequence paper develops modern time-uniform inference. These sources concern mathematical procedures with stated assumptions; they do not endorse an arbitrary dashboard or stopping rule.

The path calculations and the conservative examples in this guide were derived for teaching. Recheck them when changing hypotheses, independence assumptions, boundaries or the meaning of an observation. Continue through the specialist sequential-evidence article for an applied evidence setting, and return to the Education parent to test understanding: can you explain what remains valid when the stopping moment is chosen from the data?

Previous chapter · Contents · Gold Standard master