Choose your reading route
eduKateSG · Why Mathematics? · Barnard’s Exact Test, Binomial Tables and Unconditional Evidence
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 7–10
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 11–16
Check assumptions
See where design and calibration matter.Route 4 · Chapters 17–19
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 20–22
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when we compare two success rates? Because a difference between fractions is only the beginning of the question. Barnard’s exact test compares the success probabilities of two independent binomial groups while allowing the total number of successes to vary. It constructs a reference from possible outcomes under the sampling design, then protects the comparison against an unknown common success probability.
Imagine two small batches of independently tested components. One batch contains three passes and no failures; the other contains no passes and three failures. The observed difference looks dramatic. Mathematics asks a more careful question: how surprising is this contrast under a common pass probability, and which quantities were fixed by the experiment? A tiny table becomes an opportunity to understand sampling, uncertainty and evidence.
This guide explains the unconditional sample space, the pooled statistic used by SciPy, a hand-checkable exact example and the limits of the conclusion. All examples are synthetic. They describe statistical mechanisms rather than a real product comparison, a school result or a recommendation about treatment. The school connections are enrichment in probability and numeracy, not a claim that a named exact test belongs to a particular Singapore syllabus.
Did You Know? Two legitimate exact tests can return different p-values for the same table because they use different reference experiments. “Exact” describes calibration under a specified model; it does not mean the model was chosen correctly or that the result proves a cause.
Full chapter index
Chapters 1–6 · Understand the question
Chapters 7–10 · Reproduce the example
Chapters 11–16 · Check assumptions
Chapters 17–19 · Explore applications
Chapters 20–22 · Study and verify
Let group A contain n_A independently observed binary outcomes and group B contain n_B. Each outcome is coded success or failure using a definition fixed before analysis. Within a group, the simple binomial model assumes a common success probability. The counts are X_A and X_B, with underlying probabilities p_A and p_B.
The two-sided null is p_A=p_B. The alternative is that these probabilities differ. We estimate each probability by its observed success fraction, but the fractions are noisy because only finitely many independent units were observed. Repeating the experiment would usually produce another pair of counts.
The group sizes are treated as fixed in the reference calculation. The combined number of successes is allowed to vary. This is the central sampling distinction. If the actual design fixed other margins or stopped collecting after a favourable result, the simple two-binomial reference needs reconsideration.
A “success” need not mean something desirable. It can mean a defect, an error or an event of concern. Always state the coding. A positive difference in defect probabilities has a different practical meaning from a positive difference in reliable operation.
CHAPTER 2 OF 22 · Understand the question
2. Read the table before running the test
SciPy’s Barnard interface treats columns as the two binomial experiments. The top row holds success counts and the bottom row failure counts. For three passes in A and none in B, the table is [[3,0],[0,3]]. The first column sums to three and the second also sums to three.
Write the labels beside the numbers. A common mistake is to arrange groups as rows, pass that table to a column-based interface and interpret the sign as though nothing changed. Transposition can change which totals the procedure treats as fixed, so this is more than a cosmetic formatting issue.
Check that every entry is a non-negative integer and that successes plus failures equal the planned group size. Percentages alone are insufficient. A success rate of 60% could represent three of five observations or six hundred of one thousand; those contain very different amounts of information.
Weights and repeated outcomes also require care. A table of weighted pseudo-counts is not automatically a binomial table, and ten tests on one component are not ten independent components. The table should preserve the actual experimental unit.
CHAPTER 3 OF 22 · Understand the question
3. Why a difference of fractions needs calibration
The observed difference is D=X_A/n_A−X_B/n_B. It is a useful descriptive estimate in probability units. Multiplying by one hundred expresses percentage points. A difference of 0.20 is twenty percentage points, not necessarily a twenty-percent relative increase.
The variability of D depends on the group sizes and the underlying probabilities. Near a probability of one-half, a Bernoulli outcome has more variance than near zero or one. The same numerical difference can therefore carry different evidence in different experiments.
A test statistic organises this contrast into a way of ordering possible tables. The p-value then depends on that ordering and the reference probabilities. Keeping these two jobs separate makes the calculation intelligible: first define “at least as extreme”; then ask how much null probability falls there.
Descriptive and inferential statements should remain separate. “All three observed A units passed” is directly known. “Every future A unit will pass” is unsupported. “The common-probability model gives a small tail probability” is an inferential statement whose assumptions should be named.
For the pooled variant, estimate the common success probability from all observations: p̂=(X_A+X_B)/(n_A+n_B). The estimated variance of the difference under this common-probability calculation is p̂(1−p̂)(1/n_A+1/n_B).
Divide D by the square root of that expression. Call the result T. It is dimensionless. The numerator describes the success-fraction contrast, while the denominator provides a scale for ordering tables with different total numbers of successes.
The statistic alone is not the exact test. Comparing T with a standard normal distribution would define an approximation rather than Barnard’s full unconditional tail calculation. The exact procedure visits the finite space of possible success counts and evaluates the selected rejection region under the binomial model.
There is also an unpooled statistic option. That changes the table ordering and may change the p-value. Record which variant was used. The worked example here uses SciPy’s default pooled=True and a two-sided alternative; a report saying only “Barnard” omits a meaningful implementation choice.
CHAPTER 5 OF 22 · Understand the question
5. Enumerate the unconditional sample space
With fixed sizes n_A and n_B, X_A can take n_A+1 values and X_B can take n_B+1. Their combinations give (n_A+1)(n_B+1) possible tables. For three observations in each group, the sample space contains sixteen count pairs.
The pair (a,b) receives probability C(n_A,a)C(n_B,b)p^(a+b)(1−p)^(n_A+n_B−a−b) under a common success probability p. Here C denotes a binomial coefficient. The two counts are independent under the stated model, so their binomial probabilities multiply.
These sixteen count pairs are not equally probable. A middle count can arise through several underlying success/failure sequences, while an all-success count arises through only one. The binomial coefficients preserve that multiplicity. Treating each table as one equally weighted outcome would create the wrong reference.
A useful classroom exercise is to enumerate the underlying binary sequences first, then group them into count tables. Students can see why a compact table needs probability weights even though every underlying sequence is equally likely when p=one-half.
CHAPTER 6 OF 22 · Understand the question
6. The unknown common probability is a nuisance parameter
Under the null, both groups share p, but the null does not tell us its value. It might be 0.1, 0.5 or 0.9. The tail probability of a chosen rejection region changes with p. Substituting a convenient number would fail to protect the test across the whole null model.
Barnard’s unconditional construction takes the supremum of that tail probability over allowable common probabilities. In ordinary language, it asks for the largest null tail probability compatible with equal group probabilities. This produces a conservative protection against not knowing p.
The maximising p is a mathematical ingredient in calibration, not an estimate that the real process has that probability. Nor is it a posterior probability of the null hypothesis. It is the common probability under which the selected extreme region receives its greatest null mass.
This is a good example of constrained mathematical reasoning. We do not remove uncertainty by pretending p is known; we account for it through an optimisation. The guarantee comes from the complete model and correctly evaluated maximum, not from the word “exact” alone.
CHAPTER 7 OF 22 · Reproduce the example
7. Worked example: complete separation in six observations
For table [[3,0],[0,3]], the observed fractions are one and zero. Their difference is one. The pooled estimate is three successes out of six observations, so p̂=0.5. The denominator is √[0.5×0.5×(1/3+1/3)]=√(1/6).
Therefore T=√6≈2.44948974. Under the pooled two-sided ordering, only two count tables are at least this extreme: all A successes with all B failures, and the reverse. Other tables with the same group sizes have a smaller absolute statistic.
The probability of these two outcomes under a common p is 2p³(1−p)³. Since p(1−p)≤one-quarter, the maximum occurs at p=one-half. The maximum tail probability is 2(1/4)³=1/32=0.03125.
SciPy 1.17.0 reproduced statistic 2.449489742783178 and p-value 0.031250000000000014, the tiny numerical difference being floating-point roundoff. At a prespecified five-percent threshold the null is rejected under this design. The result still rests on only six independent observations and does not establish future certainty.
| Outcome | A | B |
|---|---|---|
| Success | 3 | 0 |
| Failure | 0 | 3 |
For the same observed table, Fisher’s two-sided exact test gives p=0.10 using its usual probability ordering. Its reference conditions on the combined number of successes as well as the group sizes. Here that total is three.
Conditional on three successes among six units, there are C(6,3)=20 equally likely allocations under the relevant null model. Only one allocation puts every success in A; another puts every success in B. The two-sided extreme probability is two out of twenty.
Barnard allows the total successes to vary and maximises over p. Fisher conditions on the observed total. Neither calculation is an arithmetic mistake. They describe different reference spaces and can make different threshold decisions. A responsible comparison starts with the design and the chosen inferential plan.
Do not run both and report only the smaller p-value as though the method had been chosen in advance. Exploring alternatives can teach useful distinctions, but confirmatory reporting should state the selection rule. A result is not strengthened by hiding the analyses that were tried.
CHAPTER 9 OF 22 · Reproduce the example
9. One-sided hypotheses must precede the data
For an increasing alternative, the target is p_A>p_B, with the corresponding null allowing p_A≤p_B. A decreasing alternative reverses that direction. The observed sign must be interpreted using the interface’s table orientation and group labels.
Choose the directional question because a prior scientific or practical argument justifies it. Choosing “greater” after observing that A happened to exceed B turns a directional test into an unacknowledged search. The resulting error rate differs from the claimed prespecified procedure.
The two-sided test asks about differences in either direction. Its definition of extremeness is not interchangeable with doubling any one-sided number in every exact-test implementation. Document the actual procedure rather than relying on a universal shortcut that may not apply.
For learning, swap the group columns and predict what should happen. The sign of T should reverse. A symmetric two-sided result should remain the same apart from numerical roundoff. Directional alternatives should be swapped consistently. This simple check catches many coding and interpretation errors.
CHAPTER 10 OF 22 · Reproduce the example
10. Numerical optimisation is part of reproducibility
The finite count space is exact, but evaluating the supremum over p is a numerical task in general. SciPy exposes a sampling-point parameter n for its optimisation strategy. That n is not the number of experimental observations and should not be confused with a group sample size.
The implementation uses a Sobol-based construction and converts the requested number of points to a power of two. Current documentation describes the default as usually sufficient, while acknowledging additional points cost computation. Record the actual setting and software version when numerical reproducibility matters.
Our main example avoids a complicated optimiser because the tail polynomial has a transparent maximum at one-half. A second example, table [[6,2],[4,8]], gives a pooled two-sided statistic about 1.82574186 and p≈0.09487800 with n=128 in the local calculation.
Comparing settings is a numerical stability check, not permission to select the setting that produces significance. If reasonable settings disagree materially, investigate the optimisation and report the limitation. More digits are useful only when the calculation supporting them is stable.
CHAPTER 11 OF 22 · Check assumptions
11. Independence cannot be inferred from a spreadsheet
The binomial model requires independent outcomes within and between groups in the simple analysis. Two separate worksheet tabs do not create independence. Repeated measurements from the same person, device or classroom can share causes and information.
Consider ten trials on each of three components. A component-specific defect may make all ten trials correlated. Entering thirty outcomes as thirty independent components exaggerates information. A hierarchical or clustered analysis may be needed, or the outcome may need to be defined at component level.
For matched before-and-after binary outcomes, the central information concerns changes within a unit. An independent two-binomial comparison discards that matching. A matched method such as McNemar’s test answers a different design question and should preserve the paired record.
Describe the unit explicitly: one independently selected component, one participant or one appropriately randomised experimental unit. If units are clustered, say how that structure was handled. A beautifully reproduced p-value cannot repair a miscount of independent information.
A pass/fail outcome is useful when a meaningful standard exists. But converting a continuous measurement to a binary flag discards how far an observation lies above or below the threshold. Two systems with equal pass fractions can have very different distributions of error.
Fix the threshold before comparing groups and explain why it matters. Selecting a threshold after plotting the data can create a favourable table through searching. If several thresholds are scientifically important, define an analysis that addresses that collection rather than treating one chosen cut as unique.
Retain the original measurements whenever available and appropriate. They help explain borderline cases, measurement uncertainty and the seriousness of failures. A one-millimetre failure and a twenty-millimetre failure both receive the same binary code but may demand different responses.
In education, a pass-rate comparison cannot diagnose the missing prerequisite behind a pupil’s work. The table describes a frequency under a definition. A marked script, explanation and fresh independent attempt provide different information about learning and should remain part of the practical response.
CHAPTER 13 OF 22 · Check assumptions
13. Small samples do not become large through exactness
An exact procedure can calibrate a small-sample test under its model without turning that small dataset into rich evidence about every feature of the populations. Three observations per group leave considerable uncertainty about the actual success probabilities.
The complete-separation example illustrates a model-based rejection, not a claim that effect size is known precisely. Report the observed proportions, sample sizes and an appropriate uncertainty analysis. A p-value describes a tail event; it does not provide a confidence interval automatically.
A study can also fail to reject despite an important observed difference because the sample contains little information. That is not proof of equality. If practical equivalence is the goal, define a tolerance on the probability difference and use a suitable equivalence or precision framework.
Planning should consider the desired effect, error rates, precision, costs and recruitment. An exact test is not a substitute for adequate study design. It supplies one part of a carefully specified route from observations to a limited conclusion.
CHAPTER 14 OF 22 · Check assumptions
14. A zero cell is not a zero population probability
If no failures occur in a group, its observed failure fraction is zero. The population failure probability may still be positive. A finite sample can miss a rare event. This distinction is particularly important whenever uncommon failures carry serious consequences.
Do not add arbitrary pseudo-counts merely because a table contains zero. Different statistical tasks handle zeros differently, and altering the counts changes the observed experiment. An exact table procedure can often work with zero cells, but its assumptions and any degenerate cases still need checking.
An empty group is another matter. If its column total is zero, no observed outcomes estimate that group’s probability. The intended comparison is not supported by that table. Check software warnings and do not interpret a returned number as if both groups had been sampled.
A useful practice question is to distinguish three statements: no observed failures, estimated failure fraction zero, and proven impossibility of failure. Only the first two follow directly from the count. Mathematics helps prevent those statements from collapsing into one misleading sentence.
Build two synthetic batches with known generating success probabilities. Fix group sizes, simulate independent binary outcomes and record the resulting tables. Begin by predicting how often the observed fractions will differ even when the generating probabilities are equal.
Run the prespecified pooled two-sided test on many simulated experiments. Under a chosen common probability, count how often the result rejects at the chosen level. Repeat for several common probabilities. This explores calibration across the nuisance parameter rather than checking only one convenient null point.
Then change the generating probabilities to create a known difference. Compare rejection frequency as group sizes increase. This is a power experiment in the specified simulation model, not a universal promise that more observations always produce a particular p-value in one realised study.
Keep the random seed, generation rule and analysis settings. A classmate should be able to reproduce the demonstration. Report all simulated runs or a declared summary, not only the dramatic tables that make the method look impressive.
CHAPTER 16 OF 22 · Check assumptions
16. A usability comparison that preserves the question
Suppose independent participants try one of two hypothetical interfaces, and success means completing a specified task without assistance within a fixed time. The binary definition, task and timing rule should be the same in both groups.
Random assignment can help support a causal comparison when the broader experiment is appropriate. Without it, prior experience, recruitment and device differences may explain the table. Barnard’s statistic does not identify which causal story generated the success-rate contrast.
A participant who almost finishes and one who cannot begin may both count as failures. Retain completion time, assistance requests and qualitative observations when they matter to design. The binary result answers one operational question and should not crowd out the richer user experience.
This connects mathematics with design and technology careers through a concrete skill: defining a measurable outcome, preserving a fair comparison and explaining uncertainty. Learning the named test alone does not guarantee employment or identify the universally best interface.
Two independent classes might have different proportions above a score threshold. Before testing, ask whether they sat comparable assessments, followed the same scoring rules and came from an appropriate sampling or assignment process. A threshold comparison cannot fix an unequal measurement instrument.
Classes may also be clusters rather than collections of independent pupils. Shared teachers, timetables and prior preparation can create dependence. The simple binomial model may understate uncertainty when that structure is ignored.
Use synthetic data for an enrichment exercise and avoid attaching broad ability labels to children. A success fraction is a summary of one defined outcome in one context. It does not describe all capabilities, future development or the suitability of a learning pathway for an individual.
For parents, the useful follow-up is to inspect what the learner can explain and perform independently. Statistical literacy supports that judgement by identifying which claims a group comparison establishes and which still require evidence from the learner’s actual work.
CHAPTER 18 OF 22 · Explore applications
18. Study the arithmetic before the software
Start with fractions and percentage points. Write the two observed proportions and their difference. Translate the difference into an ordinary sentence with the group names and success definition. This guards against sign errors before any advanced formula appears.
Next study binomial coefficients using short success/failure sequences. Count how many sequences produce one success, two successes or three. Connect those counts with the probabilities of count tables. The sample space becomes visible rather than a hidden computational object.
Then investigate p(1−p). Completing the square gives p(1−p)=1/4−(p−1/2)², so its maximum on [0,1] is one-quarter. This elementary identity supplies the entire nuisance-parameter maximum in our six-observation example.
Finally separate the statistic from its reference. Ask what would change if we used a normal approximation, conditioned on total successes or changed the extreme-region ordering. Students learn that a test is a system of choices and assumptions, not just a formula copied into a calculator.
Create a notebook with three layers: observed counts, descriptive contrasts and inferential settings. Keep the counts unchanged while experimenting with the analysis. This makes it possible to tell a change in data from a change in method.
Compare a two-sided analysis with a genuinely prespecified directional question in a manufactured example. Explain why the tail region changes. Label the exercise exploratory; do not imply that inspecting both directions preserves a one-sided confirmatory claim.
Change group sizes while holding similar observed rates. Record how the possible table space and uncertainty change. Then change a single outcome in a small group. The resulting movement shows why a tiny sample can be sensitive even when its calibration is exact.
End with a conclusion audit. Can every sentence be traced to a count, a stated assumption or a justified inferential calculation? Remove claims about guaranteed future success, personal benefit or causation unless the design and additional evidence actually support them.
Question 1: A has six successes in ten trials and B has two in ten. The estimated probability difference is 0.6−0.2=0.4, or forty percentage points. It is not automatically a forty-percent relative improvement.
Question 2: How many count tables are possible with group sizes three and four? There are four possible A success counts and five B counts, giving twenty tables. Their null probabilities are generally unequal.
Question 3: Why is the maximum of 2p³(1−p)³ equal to 1/32? The product p(1−p) reaches one-quarter at one-half; cubing and multiplying by two gives the result. Check the endpoints, where the tail probability is zero.
Question 4: Can six tests on the same device be treated as six independent devices? Not automatically. Identify the sampling unit and the dependence structure before choosing the reference model.
Question 5: Does p=0.03 mean a three-percent probability that the null is true? No. It is a tail probability under the specified null construction. A probability assigned to the hypothesis would require a different inferential framework and additional ingredients.
What does unconditional mean here?
The reference holds the two group sizes fixed but allows the combined success count to vary. It evaluates possible tables under two independent binomial experiments, then maximises the selected tail probability over the unknown common success probability.
Is this the same as McNemar’s test?
No. McNemar preserves paired binary observations and focuses on discordant changes. Barnard’s simple model compares independent binomial groups. A table that looks rectangular does not make those designs interchangeable.
Is Barnard always the best test?
No method is best for every design, purpose and implementation constraint. The sampling model, target, chosen statistic and calibration matter. Compare methods because their assumptions fit a question, not because one happened to return the smallest p-value.
What should accompany the p-value?
Report counts, group sizes, observed proportions, the probability difference, a suitable uncertainty description, success definition, independence assumptions and chosen variant. Explain whether the study supports description, association or a causal claim.
What does exactness leave unresolved?
Measurement quality, sampling representativeness, dependence, missing outcomes and practical consequence remain unresolved by exact tail calculation alone. The method answers one carefully defined probability question; the surrounding study determines whether that question is useful.
The historical research includes Barnard’s Significance Tests for 2 × 2 Tables. SciPy’s Barnard documentation specifies its current table orientation, variants and numerical controls. The examples here were calculated independently.
For a matched design, continue with McNemar’s discordant-pair comparison. For several matched conditions, Cochran’s Q preserves the repeated-outcome structure. Return to the Mathematics Learning Hub to connect these ideas with probability, algebra and careful mathematical explanation.