How Mathematics Improves The World | Designing an Experiment That Can Actually Answer the Question

Design of experiments, DOE, randomisation, replication, blocking, factorial design, interactions, confounding, statistical power, fractional factorial designs, response surface methodology, central composite designs, Box–Behnken designs and causal inference all belong to one question that appears simple until money, time or lives depend on the answer: how should we choose what to vary, what to hold, what to randomise and what to measure so the experiment can actually distinguish the effect we care about from all the other reasons results change?

A weak experiment can collect a mountain of data and still answer almost nothing. If every high-temperature run happens in the afternoon and every low-temperature run in the morning, temperature is mixed with time of day. If treatment A is given by one operator and treatment B by another, operator skill can masquerade as treatment effect. If we change pressure and speed together, we cannot tell which factor caused the response. Randomisation protects against unknown systematic allocation. Replication measures variability. Blocking absorbs known nuisance structure. Factorial experiments estimate several factors and their interactions efficiently instead of changing one variable at a time.

NIST’s Engineering Statistics Handbook organises DOE around exactly these design choices: set objectives, select factors and levels, choose randomised or blocked comparisons, use full or fractional factorial designs for several factors, and move to response surface designs such as central composite and Box–Behnken designs when curvature and optimisation matter. The Mathematics is not an after-the-fact statistical decoration. Experimental design determines which effects can be estimated independently, which are confounded, how much precision is available, and whether the eventual analysis has any legitimate question to answer.

---

Quick Read

Design of experiments is the mathematical planning of data collection. Before measurements exist, DOE specifies experimental units, treatments, factors, levels, randomisation, replication and blocking so effects of interest can be estimated with known uncertainty and minimal confounding.

A completely randomised design assigns experimental units to treatments using random allocation. A randomised block design groups units that share a nuisance factor—batch, operator, classroom, day, machine—and randomises treatments within each block. NIST’s guidance summarises the principle memorably: block what you can, randomise what you cannot.

A full factorial experiment runs every combination of factor levels, making main effects and interactions estimable. A fractional factorial uses only part of the combinations, saving runs at the price of aliasing some effects. Design resolution describes which classes of effects are confounded. Screening designs identify important factors cheaply. Response surface methodology adds centre and axial or carefully chosen multilevel points so curvature can be estimated and operating conditions improved.

Power calculations connect effect size, variability, sample size and significance threshold. More runs do not automatically create a good experiment: replicating a confounded design only estimates the confounded contrast more precisely. Likewise, a statistically significant result is not automatically practically important or externally generalisable.

One-sentence answer: Mathematics improves the world by designing experiments so scarce observations are arranged to separate competing explanations, estimate important effects efficiently, reveal interactions and uncertainty, and produce evidence capable of answering the question rather than merely producing numbers.

---

1. Data Quality Begins Before Data Exists

Statistics textbooks often begin after a dataset has been collected. Real scientific quality often begins earlier. Which units received which condition? In what order? On which days? With which operators? How many repeats? What response was measured and when?

If the design mixes the factor of interest with nuisance variation, no sophisticated analysis can unmix information that was never separated experimentally. The ambiguity is structural.

DOE therefore treats experimental layout as part of inference. The data matrix is deliberately constructed so certain questions have identifiable answers.

2. Start With the Question, Not the Available Spreadsheet

“Does temperature affect yield?” is one question. “Which combination of temperature, pressure and catalyst maximises yield without increasing defects?” is another. “Which of twenty possible factors deserve further study?” is a screening question.

NIST separates comparative, screening and response-surface objectives because each needs a different design. A two-treatment randomised comparison is excellent for one contrast and wasteful for mapping a curved multivariable process. A screening fraction is efficient for identifying active factors and deliberately too sparse for fitting every high-order interaction.

The design should be chosen for the decision the experiment must support. Data collection without a declared question invites analysis to chase whichever pattern appears afterward.

3. The Experimental Unit Is the Smallest Independently Treated Unit

Suppose a fertiliser treatment is applied to entire fields and ten plants are measured in each field. The experimental unit is the field, not each plant, because treatment was randomised at field level.

Treating the ten plants as ten independent treatment replications is pseudoreplication. It exaggerates sample size and makes uncertainty look smaller than the design supports.

Correct identification of experimental units determines the degrees of freedom for treatment inference. More measurements within a unit improve measurement precision and do not create independent treatment assignments.

4. Factors Are Inputs We Deliberately Vary

Temperature, pressure, dose, algorithm setting, material supplier and teaching method can all be factors. Their chosen settings are levels.

Quantitative factors have numeric levels with meaningful distance, allowing interpolation and response-surface modelling. Qualitative factors such as machine type or catalyst identity have categories without numeric spacing.

Choosing levels is part of the experiment. Too narrow a range can hide the effect. Too wide can enter unsafe or irrelevant regimes. Level selection should reflect scientific meaning, process limits and desired model form.

5. The Response Must Represent the Outcome We Care About

An experiment can be beautifully randomised and measure the wrong response. Manufacturing may optimise mean yield and worsen defect rate. A learning intervention may improve short quiz scores and not long-term retention.

Multiple responses may matter: quality, cost, time, safety and variability. Some are primary, some secondary. Declaring them before data reduces the temptation to promote whichever metric becomes significant.

Measurement systems also add error. A noisy response inflates experimental variance and reduces power even when the treatment effect is real.

6. Treatments Are Specific Combinations of Factor Levels

In a one-factor experiment, treatments can simply be three temperatures. In a factorial experiment with temperature high/low and pressure high/low, four treatment combinations exist: low-low, low-high, high-low, high-high.

The treatment combination is what one experimental unit receives. Analysis later decomposes differences among combinations into main effects and interactions.

This distinction matters because a factor effect is not observed directly. It is estimated from structured contrasts among treatment combinations.

7. Randomisation Breaks Systematic Links With Unknown Nuisance Factors

Suppose treatment A is always run first and B second. Machine temperature drifts upward through the day. The treatment comparison is now entangled with order.

Randomise treatment order and the drift is less likely to align systematically with treatment. Across repeated randomisations, treatment assignment becomes independent of pre-existing nuisance variables under the design mechanism.

Randomisation is not a mystical guarantee that one realised experiment is perfectly balanced on everything. It creates a known assignment mechanism supporting unbiased comparisons and randomisation-based inference.

8. Random Sampling and Random Assignment Solve Different Problems

Random sampling selects units from a population and supports generalisation to that population under sampling assumptions. Random assignment allocates available units to treatments and supports causal comparison inside the experiment.

You can randomly assign volunteers who were not randomly sampled. The treatment contrast can have strong internal causal validity for those participants while population generalisation remains limited.

These two randomisations are often confused. DOE primarily controls treatment assignment; external validity needs a separate argument about who, where and when the units represent.

9. Replication Estimates Variation

Run one observation at treatment A and one at B. If outcomes differ, treatment may have caused the difference—or ordinary unit-to-unit variation may have.

Independent replication supplies information about experimental error and treatment variability. With repeated units at each condition, standard errors and confidence intervals become estimable.

Replication also protects against one anomalous unit dominating the conclusion. It is the experimental source of information about noise.

10. Repeated Measurement Is Not Always Replication

Measure the same specimen ten times. You have ten instrument readings and one experimental unit. The repeats estimate measurement repeatability, not specimen-to-specimen biological or manufacturing variation.

True replication repeats the treatment independently on distinct experimental units or independent runs at the relevant level.

Both repeat measurement and replication are useful. They estimate different variance components. Calling them by the same name can produce falsely narrow confidence intervals.

11. Blocking Removes Known Nuisance Variation From the Error Term

Suppose material batches differ, but batch is not the scientific factor of interest. Put one replicate of every treatment inside each batch and compare treatments within batches.

The block effect absorbs between-batch variation, leaving a smaller residual error for estimating treatment effects. NIST describes blocking as grouping homogeneous units so nuisance factors are held constant while the primary factor varies.

Blocking increases precision when the block factor explains variation. Blocking on something irrelevant can waste degrees of freedom or complicate execution without benefit.

12. “Block What You Can, Randomise What You Cannot”

If operator, day or raw-material lot is known to matter and can be organised, block on it. Unknown nuisance variables remain, so randomise treatment allocation within blocks.

The principle separates predictable structure from unpredictable structure. Blocking models the predictable nuisance; randomisation protects against the rest.

A block is not a group created after looking at outcomes. It is part of the design, chosen because units share a nuisance condition before the treatment comparison is evaluated.

13. Pairing Is Blocking With Blocks of Size Two

Compare two methods on the same subject, matched twins, the same machine before/after, or two very similar experimental units. The pair absorbs baseline heterogeneity.

Analysis focuses on within-pair differences. If pairing is effective, variance of the difference can be far smaller than variance across unrelated units.

Poor pairing can add complexity without reducing variance. Matching should be based on variables genuinely predictive of response.

14. Latin Squares Block Two Nuisance Factors

Suppose treatments must be tested across rows and columns representing two nuisance dimensions—perhaps operator and day. A Latin square arranges treatments so each appears once in each row and once in each column.

This controls two nuisance effects economically when interaction among them and treatment is negligible enough for the additive model to be useful.

Latin squares are elegant and assumption-dependent. They trade ability to estimate certain interactions for nuisance control with few runs.

15. Factorial Experiments Change Several Factors Together on Purpose

One-factor-at-a-time experiments hold everything constant, vary A, reset, then vary B. They feel controlled and miss interactions efficiently.

A factorial experiment runs combinations of A and B. If each has two levels, four runs form a 2² design. Add C and eight runs form 2³.

Because every factor is varied across combinations of the others, main effects are estimated across a broader background and interactions become directly visible.

16. A Main Effect Is an Average Difference Across Other Factors

In a balanced two-level factorial, the main effect of A is the average response at high A minus the average at low A, averaging over all levels of the other factors.

This is more informative than changing A once at one arbitrary setting of B. The effect is measured across the designed experimental region.

But when A interacts strongly with B, an average main effect can be misleading. The interaction tells us the effect of A depends on B.

17. Interaction Means “It Depends” Has Mathematical Structure

Suppose raising temperature increases yield at low pressure and decreases yield at high pressure. The effect of temperature changes sign depending on pressure.

This is an interaction. In a regression model y=β₀+β_A A+β_B B+β_AB AB+ε, the AB term captures how the slope for A changes with B.

Interactions are why OFAT methods can be dangerous. If you vary temperature only at one pressure, you may conclude the wrong direction for another operating condition.

18. Interaction Plots Make Conditional Effects Visible

Plot mean response versus A with separate lines for B levels. Parallel lines suggest little interaction. Nonparallel lines indicate the effect of A changes with B. Crossing lines show reversal.

The plot is descriptive, not a substitute for uncertainty analysis. It reveals the geometry of the fitted means and helps stakeholders understand why one global “best factor” statement is incomplete.

Good DOE makes these conditional effects estimable by design rather than discovering too late that factor combinations were never run.

19. Orthogonality Makes Effect Estimates Independent in the Linear Model

In a balanced coded two-level full factorial, columns for main effects and interactions are orthogonal: their dot products are zero.

This means least-squares estimates of those effects do not borrow information from one another under the model. Removing or adding one orthogonal term does not change another coefficient estimate.

Orthogonality is one reason factorial designs are efficient and interpretable. The design matrix has been engineered to separate effect directions geometrically.

20. Coding Factors as −1 and +1 Simplifies the Algebra

Map low factor level to −1 and high to +1. Interaction AB becomes the product of coded columns. In a full 2ᵏ factorial, all effect columns have simple ±1 structure.

Contrasts become sums and differences. Coefficients relate directly to effects. Orthogonality becomes transparent.

Coding does not change physical settings. It changes the coordinate system in which the design’s mathematical symmetry becomes obvious.

21. A Full 2³ Factorial Estimates Seven Effects

With factors A, B and C at two levels, eight treatment combinations estimate three main effects, three two-factor interactions and one three-factor interaction, plus the intercept.

Each effect is a contrast between four high-sign and four low-sign combinations. Every run contributes information to several effects simultaneously.

This information sharing is why factorial designs can be more efficient than separate OFAT studies despite varying several inputs at once.

22. High-Order Interactions Are Often Smaller and Should Not Be Assumed Zero Blindly

Industrial screening designs often rely on an effect-hierarchy principle: main effects and low-order interactions are more likely to be important than very high-order interactions.

This empirical regularity makes fractional factorials practical because high-order interactions can be sacrificed to estimate lower-order terms with fewer runs.

If the physical process contains strong three- or four-factor interactions, that assumption fails. Domain knowledge and follow-up experiments should challenge hierarchy rather than turn it into dogma.

23. Fractional Factorials Spend Aliasing to Save Runs

A full 2⁶ design needs 64 runs. A one-quarter fraction uses 16. The saving comes because some effect columns become identical or negatives of one another within the chosen subset.

Those effects are aliased: the data estimate their combined contrast, not each effect separately without additional assumptions or runs.

Fractionation is therefore not free data compression. It is a deliberate trade: sacrifice distinguishability among selected high-order effects to obtain efficient information about effects believed more important.

24. Generators Define a Regular Fraction

For a half fraction of a 2³ design, choose C=AB. Then only combinations satisfying I=ABC under coded multiplication are run.

The defining relation I=ABC generates the alias structure: A is aliased with BC, B with AC and C with AB. The design has resolution III.

The algebra lets experimenters know the confounding structure before collecting data. Nothing about aliasing is hidden once the design generator is written.

25. Design Resolution Summarises the Severity of Aliasing

Resolution III designs can confound main effects with two-factor interactions. Resolution IV designs keep main effects clear of two-factor interactions but alias two-factor interactions with one another. Resolution V designs separate main effects and two-factor interactions from each other while aliasing two-factor with three-factor interactions.

Higher resolution generally requires more runs. The right resolution depends on which interactions matter scientifically.

A resolution label is a map of ambiguity, not a quality score detached from the problem.

26. Confounding Is Designed Ambiguity, Not Statistical Confusion

In observational discussions, confounding is often an unwanted mixing of causal effects. In designed experiments, blocking and fractionation can deliberately confound selected effects to save resources.

For example, NIST shows how a blocking effect in a 2³ factorial can be aliased with the three-factor interaction ABC. If ABC is negligible, this is an efficient way to remove batch variation without contaminating main and two-factor effects.

The important difference is knowledge: designed confounding is chosen and documented. Accidental confounding is often discovered too late.

27. Foldover Experiments Can Break Important Aliases

After a fractional factorial identifies possible effects, run a complementary fraction with selected signs reversed. Combining the original and foldover data can separate aliases.

This is efficient sequential experimentation. The first stage screens cheaply. The second stage spends additional runs only where ambiguity matters.

DOE becomes a conversation with the data: design enough to learn what question should be asked next, then augment deliberately.

28. Plackett–Burman Designs Screen Many Factors Economically

When dozens of factors might matter and only a few likely do, screening designs estimate main effects with relatively few runs.

Plackett–Burman designs are non-regular two-level screening designs available for run sizes that are multiples of four. They assume interactions are small enough not to corrupt main-effect interpretation severely.

They are discovery tools, not final mechanistic models. Active factors need confirmation and interaction study later.

29. Screening Is Feature Selection Conducted by Experiment Rather Than Observation

Machine learning often selects predictors from observational data. Screening DOE actively perturbs candidate factors in a balanced structured way so their effects are less correlated.

This can identify which process knobs deserve deeper optimisation. It is especially powerful when there are many controllable inputs and experimental runs are expensive.

The mathematics chooses perturbations that maximise distinguishability, turning scarce experiments into a designed basis for effect estimation.

30. Centre Points Test Whether a Two-Level Model Is Missing Curvature

A two-level factorial estimates a plane with interactions over coded corners. It cannot identify pure quadratic curvature because each factor only takes −1 and +1.

Add replicated centre points where quantitative factors are set midway. Compare mean centre response with the average predicted from factorial corners. A systematic difference suggests curvature.

NIST recommends reserving runs for centre points in screening work for exactly this reason. A few additional runs can tell you whether the linear approximation has stopped being credible.

31. Response Surface Methodology Maps Curvature Near an Operating Region

Once important quantitative factors are identified, the objective often changes from screening to optimisation. We need a model such as y=β₀+Σβᵢxᵢ+Σβᵢⱼxᵢxⱼ+Σβᵢᵢxᵢ²+ε.

The quadratic terms represent curvature; cross-products represent interactions. Contours of the fitted surface reveal ridges, valleys and candidate optima.

Response surface methodology is local. A quadratic approximation can describe a neighbourhood well and fail far outside the experimental region.

32. Central Composite Designs Add Axial Points to a Factorial Core

A central composite design combines factorial or fractional factorial points, centre points and axial star points along each factor axis.

The factorial portion estimates linear and interaction structure. Axial points provide information about quadratic terms. Centre replicates estimate pure error and test curvature.

The axial distance α can be chosen for properties such as rotatability or practical factor limits. Design geometry is tuned to prediction precision.

33. Box–Behnken Designs Avoid Extreme Corners

Box–Behnken designs place points on combinations where one factor is at centre and others vary, plus centre points. They can fit quadratic models without running every factor simultaneously at extreme high or low levels.

This is useful when corner combinations are unsafe or physically implausible. The design sacrifices some geometric regions intentionally.

NIST compares Box–Behnken and central composite options because design selection depends on factor count, allowed region and blocking needs.

34. Rotatability Makes Prediction Variance Depend Mainly on Distance From the Centre

In a rotatable response-surface design, prediction variance is the same for all points equally distant from the design centre.

This isotropy is attractive when no direction in factor space is privileged before the experiment. Central composite designs can choose α to achieve rotatability under standard conditions.

Rotatability is a design-of-information property. The experiment shapes where future predictions will be precise.

35. Optimal Designs Choose Runs by Matrix Criteria

Standard factorial and response-surface designs rely on elegant symmetric regions. Real experiments may have forbidden factor combinations, irregular constraints or expensive settings.

Optimal design algorithms choose a subset of candidate runs to optimise a criterion based on the information matrix XᵀX. D-optimal designs maximise determinant, reducing generalised parameter-variance volume. A-optimal designs minimise average parameter variance. G-optimal designs control worst prediction variance.

DOE can therefore be formulated directly as optimisation over which observations to collect.

36. The Information Matrix Is Where Design Becomes Estimation Precision

For a linear model y=Xβ+ε with independent equal-variance errors, covariance of the least-squares estimate is σ²(XᵀX)⁻¹.

The design matrix X is chosen before data. Its geometry therefore controls coefficient precision after data. Orthogonal columns reduce correlation among estimates; replicated rows estimate error; extreme factor settings create leverage for slopes.

This is the mathematical heart of DOE: arrange X deliberately so the inverse information matrix has the properties needed for the scientific question.

37. Statistical Power Connects Design to the Effect Size We Care About

Power is the probability of rejecting a null hypothesis when a specified alternative effect is true. It depends on effect size, noise, sample size, significance threshold and design.

A study can be too small to detect a practically meaningful effect. It can also be so large that trivial effects become statistically significant.

Power planning begins with the minimum effect worth detecting and credible variability, then asks how many independent experimental units are required under the chosen design.

38. Effect Size Should Be Chosen Before Looking at the Result

If the minimum interesting improvement is 5%, plan around detecting 5%, not “whatever becomes significant”. This anchors sample size to a decision threshold.

Domain experts should define practical importance. Statistical machinery then translates that into required precision.

This separation prevents significance from replacing judgement. Mathematics estimates an effect and uncertainty; stakeholders decide how large an effect needs to be to matter.

39. More Replication Reduces Standard Error at a Square-Root Rate

Under simple independent sampling, standard error often decreases like 1/√n. To halve the standard error, roughly quadruple the independent sample size.

This diminishing return makes variance reduction through blocking or better measurement extremely valuable. If blocking halves residual variance, fewer experimental units may achieve the same precision.

DOE improves power not only by spending more runs but by making each run more informative.

40. Unequal Allocation Can Be Optimal When Groups Have Different Costs or Variances

A simple two-group experiment often uses equal sample sizes. If one treatment is much more expensive or one group has much higher variance, optimal allocation can differ.

Neyman-style allocation ideas place more units where variability is higher when the objective is minimising variance under a fixed total sample. Cost-weighted variants account for per-unit expense.

Balanced designs are powerful defaults and not mathematical commandments. Efficiency depends on the actual variance and cost structure.

41. Cluster Randomisation Needs More Clusters, Not Merely More People

Schools, hospitals, villages or teams may be randomised as clusters because individual assignment is impractical or contamination would occur.

Individuals inside the same cluster resemble one another. The intraclass correlation reduces effective sample size. Adding more people inside one cluster gives less information than adding independent clusters.

Power calculations must reflect cluster-level randomisation and analysis. Treating every student as independently assigned when schools were assigned produces pseudoreplication at scale.

42. Split-Plot Designs Respect Factors That Are Hard to Change

Some factors are expensive to reset. Oven temperature may take hours to change while feed rate changes instantly. Randomising every individual run completely would waste enormous time.

A split-plot design randomises the hard-to-change factor at a whole-plot level and easier factors within each whole plot. This creates two error strata.

Analysis must respect the randomisation structure. The hard-to-change factor has fewer independent replications than the easy factor. Treating all runs as fully randomised understates its uncertainty.

43. Repeated-Measures Designs Model the Same Unit Through Time

Measure the same person or machine at several times or conditions. Observations within a unit are correlated.

Repeated-measures ANOVA, mixed models or generalised estimating equations account for that dependence. Randomisation can concern treatment order within subjects.

Within-unit designs can be powerful because each unit serves as its own control. They also face carryover, learning and period effects that need design counterbalancing.

44. Crossover Designs Trade Between-Subject Variability for Carryover Risk

In a two-period crossover, participants receive A then B or B then A, with randomised sequence. Each participant contributes a within-person treatment comparison.

If treatment effects persist into later periods, carryover confounds the second period. Washout periods and design restrictions attempt to prevent this.

Crossover is excellent when conditions are stable and effects reversible. It is inappropriate when treatment permanently changes the unit.

45. Mixed Models Follow the Design’s Hierarchy

Experiments can contain random blocks, nested units, repeated measures and split plots. Mixed-effects models represent fixed treatment effects alongside random variation from blocks, subjects or batches.

The model should reflect how randomisation occurred. If schools were random blocks and classrooms nested inside schools, the covariance structure is not optional statistical decoration.

Analysis that ignores design hierarchy can produce correct-looking coefficients with wrong standard errors.

46. Nested Factors Are Not Crossed Factors

Operators 1–3 work only on Machine A; operators 4–6 only on Machine B. Operator is nested within machine. The same operator levels do not appear across all machines.

Crossed factors have all or many combinations. Nesting changes which effects can be separated and which experimental units supply replication.

Correct notation prevents an analysis from imagining comparisons the design never performed.

47. Fixed and Random Effects Answer Different Generalisation Questions

If three named machines are the only machines we care about, machine can be treated as fixed. If they are sampled from a larger population of possible machines and we want a variance component, a random-effect model may be appropriate.

The choice concerns the inferential target, not whether a variable “looks random”.

DOE should decide this during planning because the number of levels and replication needed for estimating a variance component differs from comparing specific fixed levels.

48. Measurement-System Analysis Can Precede the Experiment

If measurement noise is large compared with treatment effects, the experiment can fail regardless of randomisation.

Gauge repeatability and reproducibility studies separate equipment repeatability from operator effects. Calibration checks bias and linearity.

Improving the measurement system before the main DOE can produce more power than doubling sample size. The response channel deserves its own experiment.

49. Sequential Experimentation Is Often Better Than One Giant Design

Start with screening. Identify important factors. Add centre points. If curvature appears, augment to a response surface. Move toward the promising region. Confirm at the predicted optimum.

This strategy uses information adaptively without pretending the first design knew everything. NIST explicitly discusses augmenting designs with centre points and foldovers.

Sequential DOE treats experimentation as a learning process rather than one ceremonial data collection event.

50. Steepest Ascent Uses the First-Order Model to Decide Where to Experiment Next

After a factorial experiment fits a local linear model, the gradient vector points in the direction of fastest predicted response increase.

Run experiments along that path until response stops improving. Then centre a new design around the improved region and fit curvature.

The mathematics alternates estimation and movement. Rather than fit a global model over an enormous region, learn locally and navigate the surface.

51. Confirmation Runs Protect Against Optimising Noise

A fitted response surface predicts an optimum. If we declare victory using the same data that selected it, random noise can make the predicted peak look better than reality.

Run fresh confirmation experiments at the proposed settings and at a meaningful baseline. Compare observed response with predicted intervals.

Optimisation without confirmation risks deploying a point chosen partly because of favourable noise.

52. Multiple Responses Require Trade-Offs, Not Separate Optimisation in Isolation

A process should maximise yield, minimise defects and keep energy below a limit. The settings optimising each separately may conflict.

Desirability functions map each response onto a common 0–1 scale and combine them. Multi-objective optimisation can instead display a Pareto frontier.

The mathematics cannot decide how much one quality metric is worth relative to another without stakeholder values. It can show which trade-offs are technically unavoidable.

53. Mixture Experiments Need a Different Geometry Because Components Sum to One

Formulating paint, alloy, food or fuel involves proportions x₁+…+x_q=1. Increasing one component necessarily decreases others.

Ordinary factorial designs that vary components independently leave the feasible simplex. Mixture designs place runs within the simplex and use Scheffé-type polynomial models.

NIST distinguishes mixture objectives from ordinary factor experiments for this reason. The feasible geometry determines the design family.

54. Robust Parameter Design Seeks Performance Insensitive to Noise Factors

Some factors can be controlled in production; others vary with environment, users or raw materials. A robust design chooses control-factor settings that deliver good mean performance and low sensitivity to nuisance noise factors.

Crossed arrays or combined designs vary control and noise factors deliberately. Interactions between them reveal settings where the slope with respect to noise is small.

The objective changes from “find the highest response in the laboratory” to “find settings that remain good when reality fluctuates”.

55. Designed Experiments and Observational Studies Answer Different Causal Questions

Randomised experiments control assignment and can support strong causal contrasts under design assumptions. Observational studies measure naturally occurring exposures and must address confounding through modelling, matching, instruments or other methods.

DOE does not make every experiment ethically or practically possible. You cannot randomise every exposure. But when intervention is possible, design can create causal identification rather than trying to recover it statistically afterward.

Random assignment is therefore not merely a statistical tradition. It is a mechanism for separating treatment from alternative explanations.

---

Advanced Field Manual | Causal Validity, Adaptive Experimentation and Information Design

56. Randomisation Supports an Exact Logic of Comparison

Randomisation is more than a convenient way to balance groups. It creates a known probability mechanism for treatment assignment. Under a sharp null hypothesis that treatment changes no unit’s outcome, the only randomness in the treatment–outcome association comes from the assignment itself.

Randomisation inference asks how extreme the observed treatment contrast would be across all assignments that could have occurred under the design. For small experiments this logic can produce exact tests without assuming normally distributed outcomes.

The important principle is that inference can be grounded in how the experiment was run. A design is not merely a prelude to analysis; the assignment mechanism becomes part of the mathematical justification for the conclusion.

57. Allocation Concealment Protects Randomisation From Human Prediction

A randomisation schedule can be mathematically valid and operationally compromised if the person assigning units can predict the next treatment and selectively influence who receives it.

Allocation concealment prevents foreknowledge of upcoming assignments. Central randomisation, secure software or other controlled procedures keep the random sequence from being manipulated before assignment.

This distinction matters because random generation and random implementation are different jobs. A beautiful probability distribution on paper does not protect the experiment if humans can quietly bypass it.

58. Stratified Randomisation Protects Balance on Variables We Already Know Matter

Pure randomisation can, by chance, place more high-risk or high-baseline units in one treatment when the sample is small. If a few baseline variables are strongly predictive, stratify on them and randomise within strata.

For example, a manufacturing trial can stratify by machine family before assigning process settings. An educational trial can stratify by school and baseline achievement band. The treatment comparison then avoids large accidental imbalances on those known variables.

Too many strata create tiny cells and operational complexity. Stratification should protect genuinely important baseline structure, not encode every available variable.

59. Covariate-Adaptive Randomisation Trades Simplicity for Better Ongoing Balance

Some experiments assign units sequentially and use previous assignments to keep treatment groups balanced across several covariates. Minimisation and biased-coin procedures are examples.

These methods can improve balance in small or heterogeneous samples, especially when simple stratification would create too many cells. Their assignment probabilities are no longer independent identical coin flips, so analysis should respect the actual mechanism.

The design lesson is broader: balance can be engineered dynamically. But every extra adaptation increases the importance of documenting the algorithm precisely enough that inference remains reproducible.

60. Restricted Randomisation Can Protect Against Operationally Bad Sequences

In a long industrial run, simple randomisation could generate ten high-temperature settings in a row. That may be statistically legal and operationally unsafe or inefficient.

Restricted randomisation limits the assignment space—for example, requiring balance inside blocks of runs or preventing excessively long treatment runs. The valid reference distribution for randomisation-based inference is then the restricted design, not every possible permutation.

Restriction is legitimate when declared in advance. Quietly rearranging a random order afterward to make it convenient changes the design without updating the statistical logic.

61. Missing Data Can Be Caused by the Experiment Itself

An experimental unit may drop out because the treatment is difficult, a machine fails under one setting, a specimen breaks, or a participant refuses follow-up. Missingness can therefore contain information about treatment response.

Deleting incomplete cases can bias comparisons when missingness differs by treatment or outcome risk. The design should anticipate likely failure and define what measurements remain possible after discontinuation.

Prevention is often more valuable than sophisticated imputation. Good experiments reduce avoidable missingness, capture reasons and preserve outcome measurement where feasible even when treatment adherence changes.

62. Attrition Is a Threat to the Randomised Comparison When It Is Differential

Randomisation balances treatment groups at assignment. If one group later loses many more units, the observed survivors need not remain comparable.

Report attrition by treatment and reason. Sensitivity analyses can examine how conclusions change under plausible outcomes for missing units. In process experiments, failed runs should be recorded rather than silently rerun until a successful observation appears.

A design cannot guarantee that data remain complete. It can guarantee that missingness is treated as an experimental event rather than erased from the record.

63. Intention-to-Treat Preserves the Original Randomisation Question

When assigned treatment and received treatment differ, one important estimand compares groups according to original assignment. This preserves the randomisation contrast and answers the effect of the treatment policy or assignment strategy.

Per-protocol or as-treated analyses answer different questions and can reintroduce selection because adherence is often related to prognosis, ability or process condition.

The general DOE lesson is to define the estimand before analysis: are we estimating the effect of assignment, the effect among compliant units, or the physical effect of actually receiving a treatment? Different questions require different assumptions.

64. Interference Breaks the Assumption That One Unit’s Treatment Affects Only Itself

A fertiliser applied to one plot may leach into another. Students assigned a learning intervention can share materials with classmates. One network server’s configuration changes load on neighbouring servers.

When treatment spills across units, the usual independent potential-outcome framework needs modification. Cluster randomisation, buffer zones or explicit network-interference designs can reduce or model spillover.

Interference is not merely extra noise. It changes what treatment means because one unit’s outcome depends on other units’ assignments.

65. Contamination Is a Practical Form of Treatment Mixing

Operators trained on a new method may use parts of it during control runs. Teachers can share techniques across classes. Software feature flags can leak when users interact across treatment boundaries.

Contamination reduces treatment contrast and can bias effects toward zero or create hybrid interventions. Designs can randomise at a higher organisational level, separate timing or geography, or measure contamination explicitly.

Randomisation identifies a causal contrast only when the treatment contrast actually exists in practice.

66. Factorial Experiments Can Estimate More Than One Causal Question at Once

In a randomised factorial experiment, factors A and B can both be assigned independently. This allows causal estimation of A averaged over B, B averaged over A and the interaction between them under the design.

This is highly efficient when interventions can be combined and interactions matter. Four groups can answer questions that two separate two-group experiments would answer less efficiently.

The interpretation still depends on the factorial region. The average effect of A across the chosen B levels is not automatically the effect of A in every possible environment.

67. Multiplicity Appears When One Experiment Asks Many Questions

Test twenty unrelated null hypotheses at a 5% threshold and the chance of at least one false positive can become substantial. Factorial experiments, multiple responses, many subgroups and many time points create multiplicity.

Family-wise error procedures, false-discovery-rate methods and hierarchical testing strategies address different objectives. Pre-specifying primary comparisons prevents every exploratory pattern from being presented as confirmatory evidence.

Multiplicity is not a reason to avoid exploration. It is a reason to label exploration honestly and protect the inferential claims that matter most.

68. Pre-Specification Separates Confirmation From Discovery

Before data, write the primary response, factor contrasts, exclusion rules, transformation plans and stopping rule. This does not prevent learning from unexpected patterns.

It prevents the analysis from quietly redefining the original question after seeing the answer. Exploratory results can motivate a new experiment designed to confirm them.

The distinction is methodological memory. A system of science improves when it remembers which hypotheses existed before the data and which were born from the data.

69. Optional Stopping Changes the Reference Distribution

Run ten units, test significance, continue only if p>0.05, test again after five more, and stop the first time p<0.05. The nominal 5% test no longer has a 5% false-positive rate under the usual fixed-sample calculation.

Sequential designs handle interim looks explicitly. Group-sequential boundaries or alpha-spending functions allocate evidence thresholds across analyses.

Stopping can save resources and is not statistically free. The stopping rule belongs in the design just like the randomisation rule.

70. Group-Sequential Designs Allow Early Success or Futility

A long experiment can schedule interim analyses after planned information levels. Strong evidence can justify stopping early for benefit; very weak prospects can justify futility stopping.

Boundaries such as O’Brien–Fleming-type rules are conservative early and closer to conventional thresholds later. Pocock-type rules distribute thresholds differently.

The principle is efficient evidence acquisition: do not spend all planned experimental units when the conclusion is already sufficiently clear under a pre-specified sequential framework.

71. Adaptive Designs Change the Experiment Based on Accumulating Data

An adaptive experiment may drop ineffective treatment arms, change allocation ratios, enrich a subgroup, adjust dose ranges or modify sample size based on interim information.

Adaptation can improve efficiency and complicate inference. Rules must be pre-specified or otherwise accounted for because the data influence which data are collected next.

Adaptive design is not permission to improvise after every result. It is a formal plan in which learning changes later experimental actions while preserving defined error or decision properties.

72. Response-Adaptive Randomisation Has a Different Objective From Balanced Inference

If one treatment appears superior, response-adaptive randomisation can assign more future units to it. This may reduce exposure to poorer conditions during the experiment.

Unequal and data-dependent allocation can reduce information for estimating some contrasts, create time-trend sensitivity and complicate analysis.

The design objective therefore matters. Is the experiment trying to learn the treatment difference as precisely as possible, treat current units as effectively as possible, or balance both?

73. Multi-Armed Bandits Optimise Reward While Experiments Usually Optimise Knowledge

A bandit algorithm balances exploration of uncertain options and exploitation of options that currently look good. Regret—the lost reward from not always choosing the best arm—is a central objective.

A classical experiment is often willing to allocate equally to inferior-looking treatments because precise comparison is the purpose. The best bandit policy for reward can be a poor design for unbiased effect estimation.

These frameworks are neighbours and not synonyms. Before borrowing an adaptive algorithm, decide whether the mission is learning, serving current users, or both.

74. Bayesian Experimental Design Chooses Runs by Expected Information or Decision Value

Bayesian design starts with a prior over unknown parameters or models. A candidate experiment produces a distribution of possible future data. We can score that experiment by expected reduction in posterior uncertainty or expected improvement in a decision.

Expected information gain, predictive entropy reduction and value-of-information criteria turn “which run next?” into an optimisation problem.

This is especially useful when experiments are expensive and sequential. The cost is computational: evaluating future posterior distributions for many candidate designs can be difficult.

75. Computer Experiments Need Space-Filling Designs Rather Than Physical Replication

A deterministic simulation returns the same answer at the same inputs, so ordinary replication does not estimate experimental noise. The challenge is to explore a high-dimensional input region efficiently.

Latin hypercube sampling, maximin distance designs and low-discrepancy sequences spread runs through the space. Surrogate models then interpolate expensive simulations.

Computer experiments show why DOE is bigger than ANOVA. The fundamental job is selecting informative inputs under a finite budget, even when observation noise is zero.

76. Latin Hypercube Sampling Spreads Every Input Across Its Range

Divide each factor’s distribution into n equally probable intervals and sample one value from each interval. Then pair factor values so every interval in every dimension appears once.

This creates good one-dimensional stratification with only n points, much cheaper than a full factorial grid as dimensions grow.

Different Latin hypercubes can have very different multidimensional spacing, so optimisation criteria such as maximin distance can improve the design.

77. Gaussian-Process Surrogates Turn Expensive Simulations Into Probabilistic Maps

A Gaussian process models an unknown response surface with a mean and covariance function. After several simulator runs, it predicts response and uncertainty at untried inputs.

Bayesian optimisation uses an acquisition function to choose the next run, balancing promising predicted response against uncertain regions worth exploring.

The method is a sequential DOE for expensive black-box functions. Every new run is chosen because of what the current surrogate does not yet know.

78. Stochastic Simulation Experiments Need Independent Randomness Managed Deliberately

Queueing, traffic or manufacturing simulations contain random arrivals and service times. Repeated runs at the same factor setting estimate simulation noise.

Random-number seeds determine the realised stochastic path. Independent replications need independent streams. Common random numbers deliberately reuse matched streams across treatment settings so shared randomness cancels in paired differences.

Randomness is therefore part of the design. Seed management can improve precision or quietly create dependence depending on how it is used.

79. Accelerated Life Tests Move Stress Up to Learn About Long-Term Failure Faster

Waiting ten years for every product to fail is impractical. Accelerated testing exposes units to elevated temperature, voltage, load or other stress, then models how failure-time distribution changes with stress.

Stress levels must be chosen high enough to create informative failures and low enough that the failure mechanism remains representative of ordinary use.

The design problem is therefore partly extrapolation. Mathematics can estimate a stress–life relationship, but the physical assumption that the same mechanism persists needs engineering validation.

80. Destructive Testing Makes Experimental Units Scarce by Definition

A crash test destroys a vehicle. Tensile testing breaks a specimen. Some chemical assays consume the sample. Repeated measurement of the same unit is impossible.

Blocking by batch, matched specimens and optimal allocation become especially valuable when every observation consumes expensive material.

Destructive testing makes DOE’s economic purpose obvious: the design should extract the most discriminating information possible from units that cannot be used twice.

Advanced Field Manual | Worked Designs, Failure Diagnosis and Experimental Judgement

81. Worked Example: A 2³ Experiment Can Reveal an Interaction OFAT Misses

A process has three two-level factors: temperature A, pressure B and catalyst concentration C. Eight treatment combinations run the full 2³ factorial.

Suppose raising temperature improves yield strongly when pressure is low and barely changes yield when pressure is high. The average temperature effect is positive, and the AB interaction is also large. An OFAT study changing temperature only at low pressure would exaggerate the general benefit; one conducted only at high pressure might conclude temperature barely matters.

The factorial design answers a richer question because it varies temperature across both pressure settings. Interaction is not a statistical nuisance discovered afterward. It is one of the reasons the design exists.

82. Worked Example: Blocking Turns Day-to-Day Drift Into a Controlled Nuisance

Four treatments must be tested across five days. Ambient humidity and raw-material condition vary by day. Run every treatment once each day and randomise their order within the day.

The block effect captures day-to-day shifts. Treatment comparisons are made within a day and aggregated across days. If day explains substantial response variation, residual error falls and treatment estimates become more precise.

If instead treatment A were run Monday, B Tuesday, C Wednesday and D Thursday, treatment and day would be inseparable. No later regression can reconstruct contrasts that the design never observed within common conditions.

83. Worked Example: A Half Fraction Saves Runs and Creates an Alias

Factors A, B and C each have two levels. A full factorial uses eight runs. Choose the half fraction defined by C=AB and run four combinations.

The defining relation I=ABC implies A is aliased with BC, B with AC and C with AB. If the estimated A contrast is large, the data alone cannot tell whether the cause is A, BC or a combination.

The fraction is efficient only because we are willing to assume two-factor interactions are small enough for main-effect screening. The alias table is the bill for the saved runs.

84. Worked Example: A Foldover Buys Back Information Selectively

The first half fraction suggests A is important, but A is aliased with BC. Rather than repeat a full eight-run experiment blindly, add the complementary foldover fraction.

Combined data separate the previously aliased contrasts. If A remains large while BC is small, the ambiguity is resolved.

This is sequential experimental economy: use a cheap design to discover where uncertainty matters, then spend the next runs specifically to break that uncertainty.

85. Worked Example: Centre Points Detect Curvature With Very Few Extra Runs

A two-factor 2² experiment runs the four corners of a square factor region. The fitted first-order model predicts the response at the centre as the average implied by the plane.

Now run the centre condition several times. If the centre mean differs systematically from the planar prediction beyond pure-error variation, curvature is present.

A handful of centre points can therefore tell the experimenter that the next design should move from screening to response-surface modelling rather than collecting more corner replicates of a model already known to be inadequate.

86. Worked Example: Response Surface Optimisation Needs Confirmation

A central composite design fits a quadratic model for temperature and pressure. The fitted surface predicts maximum yield at 184°C and 2.7 bar.

The optimum is a function of estimated coefficients and therefore noisy. Run independent confirmation trials near 184°C and 2.7 bar, along with a meaningful baseline. If the observed response matches the predicted interval, confidence rises.

If confirmation fails, inspect lack of fit, process drift, unmodelled factors or an optimum outside the region where the quadratic approximation is trustworthy. Optimising a fitted model is not the same as proving the physical process has its true optimum there.

87. Worked Example: Power Depends on Variance as Much as Sample Size

Suppose an improvement of 5 units is practically meaningful. If the response standard deviation is 2, a modest sample can detect it reliably. If standard deviation is 20, many more independent units are required.

Blocking on a strong nuisance factor can reduce residual standard deviation from 20 to 10. That can save more runs than a small increase in total sample size because power depends on the signal-to-noise ratio.

The example explains why DOE is not “do a power calculation after choosing a layout.” Design choices change variance, and variance changes the value of every replication.

88. Worked Example: Split-Plot Analysis Has Two Error Terms

Temperature A is hard to change, so each day uses one temperature. Within each day, feed rate B is randomised across six runs. There are four days at each temperature.

The effective replication for A is based on day-to-day whole plots, not all individual runs. B and the AB interaction use within-day variation more directly.

If analysis treats all forty-eight measurements as equally independent for the temperature effect, its standard error can be drastically too small. The randomisation hierarchy determines the error hierarchy.

89. Worked Example: A Mixture Experiment Cannot Increase Every Ingredient

A formulation contains resin, filler and plasticiser with proportions summing to one. Increasing resin from 30% to 40% necessarily reduces one or both other components.

An ordinary factorial interpretation of “the effect of resin holding all else fixed” is physically impossible. The experiment belongs on a simplex, and mixture-model coefficients describe blends rather than independent coordinate moves.

This is an important design lesson: factor geometry is defined by the feasible world. Statistics should not ask for counterfactual combinations that chemistry or conservation laws forbid.

90. Worked Example: Cluster Randomisation Changes the Effective Sample Size

Twenty schools are randomised, ten per educational approach, with fifty students measured in each. The raw dataset contains 1,000 students.

Students within the same school share teachers, neighbourhoods and culture. If the intraclass correlation is substantial, the effective information for the treatment contrast is much closer to the number of schools than the number of students.

Adding ten more students to each school may contribute less power than adding four additional schools. The design unit and measurement unit are not interchangeable.

91. Failure Mode: One Factor at a Time Misses Interaction

OFAT feels disciplined because only one variable changes. It is inefficient for discovering interactions and can optimise along the wrong path when factors depend on one another.

If temperature helps only at high catalyst concentration, testing temperature while catalyst is fixed low makes the factor look inactive. The experiment has not disproved the effect; it has chosen a slice where the effect is hidden.

Factorial designs deliberately vary combinations so conditional effects become observable.

92. Failure Mode: Time Trend Is Confounded With Treatment Order

Run all controls in the morning and all treatments in the afternoon. Equipment warms, raw material ages or participants tire through the day.

The observed treatment contrast now contains time trend. More replication in the same order makes the biased contrast more precise, not more valid.

Randomised run order or blocking by time period is a design repair. Analysis cannot create the missing crossing of treatment and time after the experiment ends.

93. Failure Mode: Operator Is Perfectly Confounded With Treatment

Operator A always runs method 1; Operator B always runs method 2. Any performance difference can be operator, method or both.

Cross operators with treatments or block by operator so each operator uses each method where feasible. If practical constraints prevent that, the experiment cannot estimate a pure method effect without additional assumptions.

The principle is identifiability. Two explanations that always move together cannot be separated statistically.

94. Failure Mode: Pseudoreplication Creates Imaginary Degrees of Freedom

Apply one coating batch to one panel and measure twenty points. Those twenty points are spatial subsamples of one treated panel, not twenty independent treatment replications.

If the treatment was assigned to panels, panels supply independent treatment information. Within-panel measurements can improve the estimate of each panel’s response and should be modelled as nested or repeated data.

Pseudoreplication makes p-values small by pretending correlated observations contain independent evidence they never had.

95. Failure Mode: The Experiment Is Underpowered and the Null Is Treated as Proof of No Effect

A small noisy experiment finds p=0.31. “No significant difference” becomes “the treatments are the same.” That conclusion can be unsupported.

Inspect confidence intervals and power against effects that matter. If the interval includes large positive and negative effects, the experiment is inconclusive rather than evidence of equivalence.

Equivalence and non-inferiority questions need margins and designs built for those claims. Failure to reject a difference is not automatically proof that differences are negligible.

96. Failure Mode: Too Many Endpoints Turn Chance Into a Discovery Machine

Measure fifty responses, twenty subgroups and several time points, then report only the few p-values below 0.05. Even with no true effects, interesting-looking patterns will appear.

Declare primary outcomes, control multiplicity where confirmatory claims are intended and label the rest exploratory. Replicate surprising findings in new data.

The problem is not curiosity. It is presenting search-generated patterns as though they were pre-specified tests.

97. Failure Mode: Optional Stopping Manufactures Small p-Values

Check significance after every new unit and stop as soon as the threshold is crossed. Under a null effect, repeated opportunities to stop increase false-positive probability.

Use a pre-specified fixed sample or a valid sequential framework that adjusts boundaries. Record all interim looks.

A stopping rule is part of the experiment’s probability mechanism, whether or not it appears in the final spreadsheet.

98. Failure Mode: Measurement Noise Is Larger Than the Effect

The process changes by 1 unit and the measurement system has repeatability standard deviation of 5. Replication can eventually detect the effect and at enormous cost.

Improve calibration, average repeated measurements intelligently or choose a more sensitive response. A gauge R&R study can reveal whether operator and instrument variation dominate.

Sometimes the cheapest way to increase power is not more experimental units. It is a better measurement channel.

99. Failure Mode: The Factor Range Is Too Narrow to Show Anything

Temperature varies from 100.0°C to 100.5°C while ordinary process noise is substantial. No visible response change appears.

The conclusion “temperature does not matter” is too broad. The experiment only tested a half-degree region.

Factor ranges should be wide enough to create scientifically meaningful contrast and narrow enough to remain safe, feasible and within a region where the intended model is reasonable.

100. Failure Mode: Extrapolation Pretends the Model Was Tested Where It Was Not

A response surface is fitted from 150–190°C. The fitted quadratic keeps rising and predicts spectacular performance at 240°C.

The model has no experimental evidence there. Chemistry may change, equipment may fail or the quadratic may bend differently.

DOE supports interpolation inside the designed region far more strongly than uncontrolled extrapolation. New operating regions deserve new experiments.

101. Failure Mode: A High-Order Polynomial Fits Noise Beautifully

With enough polynomial terms, a sparse dataset can be fit almost exactly. The resulting surface oscillates wildly between design points and predicts implausible optima.

Choose model complexity supported by design degrees of freedom and scientific expectation. Use lack-of-fit checks, cross-validation where appropriate and independent confirmation.

A model should summarise repeatable structure, not memorise the experimental run table.

102. Failure Mode: Missingness Becomes Unequal After Treatment

One treatment causes more machine shutdowns or participant withdrawals, and those runs disappear from analysis. The remaining sample can make the treatment look safer or more effective than it was.

Record failures as outcomes when they matter, preserve assignment groups and perform sensitivity analyses. Design data capture so a failed run cannot vanish simply because no final response was measured.

Missing data are often part of treatment behaviour, not clerical inconvenience.

103. Failure Mode: Ignoring the Randomisation Hierarchy Produces the Wrong Standard Error

A split-plot experiment is analysed as if every run were completely randomised. A cluster trial is analysed as if every person were independently assigned.

Point estimates can look sensible while uncertainty is severely understated. The correct error term comes from the level at which treatment was independently randomised.

Always draw the randomisation diagram before choosing the statistical model. The experimental hierarchy should be visible enough that the analysis cannot accidentally erase it.

104. Failure Mode: The “Best” Setting Was Selected From Noise and Never Confirmed

Twenty conditions are tested. One has the largest observed response. It becomes the new production setting.

Selection itself creates optimism because the winner was chosen partly for positive noise. Fresh confirmation separates repeatable advantage from selection luck.

This winner’s-curse effect appears in process optimisation, hyperparameter tuning and scientific discovery alike. Selection changes the distribution of the selected estimate.

105. Primary Mathematics: Good Experiments Begin With Fair Comparisons

Primary students can ask whether two paper aeroplane designs should be tested with the same thrower, same room and multiple attempts. They can randomise which design is thrown first and record distance.

They learn averages, range, tables and graphs while discovering a deeper idea: a fair comparison requires controlling what should stay the same and varying what we intend to study.

The aim is not formal hypothesis testing. It is to teach that evidence quality depends on how observations were produced.

106. Secondary Mathematics: DOE Turns Algebra Into a Plan for Learning

Secondary students already know variables, coordinates, averages, probability, graphs and simple regression. A factorial design turns factors into coded variables and interactions into products.

Randomisation connects probability with fairness of allocation. Blocking connects variance reduction with grouping. Power connects sample size with signal-to-noise ratio.

DOE shows why algebra is useful before data collection: the design matrix is a planned geometry of the questions we want to estimate.

107. Advanced Mathematics: Experimental Design Is Linear Algebra, Probability and Optimisation Working Together

Linear models connect coefficient variance to (XᵀX)⁻¹. Probability describes randomisation, error and power. Combinatorics constructs balanced and fractional designs. Optimisation chooses D-, A- or G-optimal run sets. Bayesian methods maximise expected information.

Mixed models handle hierarchical randomisation. Causal inference formalises treatment contrasts. Gaussian processes and acquisition functions design expensive computer experiments.

Experimental design is therefore not one statistical recipe. It is the mathematics of deciding where evidence should come from.

108. Why This Improves the World

• It makes each experimental run more informative. Factorial and optimal designs extract several effects from scarce observations.
• It separates causes from nuisance variation. Randomisation and blocking prevent known and unknown structure from silently becoming treatment effects.
• It reveals interactions. Systems often behave differently when factors act together than when each is varied alone.
• It reduces waste in research and engineering. Screening, sequential designs and early stopping avoid spending resources on questions already answered or factors shown inactive.
• It supports safer optimisation. Response surfaces and confirmation runs move processes toward better settings without trusting one noisy winner.
• It improves reproducibility. Pre-specified design and analysis make the evidence-generating mechanism inspectable.
• It creates honest uncertainty. The design determines which effects are estimable and which remain aliased or weakly identified.

109. What Mathematics Does Not Do

Randomisation does not guarantee perfect balance in one realised sample. Statistical significance does not guarantee practical importance. A well-designed experiment does not automatically generalise to populations, places or times it did not represent.

DOE cannot identify an effect that was perfectly confounded by the design. More replication cannot repair a missing treatment combination. A response-surface optimum is not trustworthy far outside the region tested. A factorial model cannot make an unsafe factor level ethically acceptable.

And causal interpretation still depends on treatment implementation, interference, missingness, measurement validity and the estimand actually defined. Mathematics creates stronger evidence; it does not turn every experiment into universal truth.

Frequently Asked Questions

What is design of experiments?

Design of experiments is the mathematical planning of controlled data collection so effects of interest can be estimated efficiently while nuisance variation, uncertainty and confounding are managed explicitly.

Why randomise an experiment?

Randomisation creates a known assignment mechanism and prevents treatment from being systematically tied to unknown pre-existing factors, supporting unbiased comparisons and design-based inference.

Why replicate?

Independent replication estimates experimental variability, improves precision and prevents one unusual unit from determining the treatment conclusion.

What is blocking?

Blocking groups similar experimental units or runs according to a known nuisance factor and randomises treatments within blocks so nuisance variation can be separated from the treatment contrast.

Why use a factorial design?

Factorial designs vary several factors in combination, allowing main effects and interactions to be estimated efficiently and avoiding the blind spots of one-factor-at-a-time experimentation.

What is a fractional factorial design?

It uses a carefully chosen subset of full-factorial combinations, reducing run count while intentionally aliasing some effects. Its usefulness depends on which effects are expected to matter.

What does design resolution mean?

Resolution summarises the shortest word in a regular fractional design’s defining relation and therefore which classes of effects can be aliased. Higher resolution protects lower-order effects at the cost of more runs.

What is response surface methodology?

It uses designed experiments and usually quadratic models to describe local curvature in quantitative-factor responses, find promising operating regions and support process optimisation.

What is a central composite design?

A central composite design combines factorial or fractional-factorial points, centre points and axial points so linear, interaction and quadratic effects can be estimated efficiently.

What is a Box–Behnken design?

It is a response-surface design that fits quadratic models using combinations that avoid simultaneous extreme settings of all factors, useful when corner conditions are undesirable.

What is statistical power?

Power is the probability that a statistical test detects a specified effect when that effect is truly present under the model and design assumptions.

Does more data fix a bad design?

No. More replication can estimate a confounded contrast more precisely without separating the causes inside it. Missing combinations, incorrect experimental-unit definitions and systematic treatment–time confounding are design problems, not sample-size problems.

Sources and Further Reading

• NIST/SEMATECH Engineering Statistics Handbook, Choosing an Experimental Design — full and fractional factorials, randomised block designs, Plackett–Burman, response-surface, central composite and Box–Behnken designs.
• NIST/SEMATECH Engineering Statistics Handbook, Selecting an Experimental Design — comparative, screening and response-surface objectives and run-budget guidance.
• NIST/SEMATECH Engineering Statistics Handbook, Randomized Block Designs — nuisance factors, blocking and randomisation.
• NIST/SEMATECH Engineering Statistics Handbook, Blocking Full Factorial Designs — confounding block effects with selected high-order interactions and randomisation within blocks.
• NIST/SEMATECH Engineering Statistics Handbook, Response Surface Designs — second-order modelling and response-surface design principles.
• NIST/SEMATECH Engineering Statistics Handbook, Detailed Table of Contents — broader reference across DOE, process modelling, reliability and measurement-system topics.

Continue Through eduKateSG

Continue with How Mathematics Works. Compare this article with How Investigable Questions Work in School Science: that article owns the classroom question-and-evidence job, while this article owns the mathematical architecture that makes factor effects, interactions and nuisance variation estimable.

Then compare with How Evidence Works. Evidence is the broad epistemic owner; DOE is one specialised machinery for deliberately creating comparisons strong enough to support defined causal or predictive claims.

Final Thought: The Experiment Begins Before the First Measurement

A scientist has a question.

An engineer has six knobs.

A teacher has two methods.

A factory has one week of machine time.

The temptation is to start collecting numbers.

Mathematics asks for one more moment before the first run.

What exactly are we trying to distinguish?

Which variation can we block?

What must be randomised?

Which interactions could reverse the conclusion?

How many independent units are enough?

What will we do if the response curves?

Which uncertainty will remain even after the experiment succeeds?

Then the runs begin.

By the time the first measurement arrives, much of the quality of the final answer has already been decided.

Mathematics improves the world here by designing the path by which evidence will be allowed to become an answer.