A p-value works by assuming a specified statistical model—including the null hypothesis and the sampling or randomisation structure—then asking how often that model would generate a test statistic at least as extreme as the one actually observed. A small p-value means the observed data are relatively incompatible with that model. It does not tell us the probability that the hypothesis is true, the probability that the result happened “by chance”, or how large or important the effect is.
No number in modern research is more familiar than 0.05.
It is printed in journal tables.
It decides whether stars appear beside coefficients.
It separates “significant” from “not significant”.
And because the rule is so familiar, the mechanism underneath it is often forgotten.
The governing question: if the null model and the rest of the statistical assumptions were true, how surprising would data this extreme—or more extreme—be under the procedure we actually used?
Quick Read
QUESTION → NULL MODEL → ALTERNATIVE → DATA → TEST STATISTIC → SAMPLING / RANDOMISATION DISTRIBUTION → TAIL AREA → P-VALUE → EFFECT SIZE + INTERVAL → MULTIPLICITY / DESIGN CHECK → SCIENTIFIC INTERPRETATION
The American Statistical Association’s core guidance remains the right boundary. P-values can indicate how incompatible data are with a specified statistical model. They do not measure the probability that a hypothesis is true, should not be the sole basis for scientific or policy decisions, do not measure effect magnitude or importance, and require full reporting and transparency.
1. A P-Value Begins With a Model
The phrase “the p-value of the data” is incomplete.
A p-value exists relative to a statistical test.
That test contains assumptions about:
- the null hypothesis;
- the data-generating or randomisation process;
- the test statistic;
- the treatment of nuisance parameters;
- the tail or tails considered;
- and often independence, distributional form or asymptotic approximation.
The p-value therefore belongs to a model-data relationship, not to the data alone.
2. The Null Hypothesis Defines the Reference World
Suppose a trial compares two teaching methods.
A simple null hypothesis might say the population mean difference is zero.
The p-value asks what the chosen test statistic would look like if that null world were true and the rest of the test assumptions held.
The null is therefore a reference model, not a fact that is believed until disproved.
3. The Test Statistic Compresses the Data Into a Direction of Disagreement
A t statistic combines a mean difference with its estimated standard error.
A chi-squared statistic summarises discrepancies among observed and expected counts.
A likelihood-ratio statistic compares model fit.
The p-value does not inspect every detail of the data equally.
It measures extremeness through the chosen statistic.
4. “At Least as Extreme” Creates a Tail Area
NIST defines the p-value through the probability, under the null hypothesis, of obtaining a test statistic at least as extreme as the observed one.
For a one-sided test, extremeness may live in one tail.
For a two-sided test, extremeness includes departures in both relevant directions.
Tail definition is part of the test, not something chosen after looking at the sign of the result.
5. A Small P-Value Means “This Would Be Unusual Under the Tested Model”
If p = 0.003, then results at least as extreme as the observed statistic would occur with probability 0.003 under the specified null model and procedure.
That creates tension between model and data.
It does not tell us which assumption failed.
The null effect could be wrong.
The distributional assumptions could be wrong.
The sampling process could be biased.
The model-data disagreement needs scientific interpretation.
6. A Large P-Value Does Not Prove the Null
If p = 0.47, the data are not especially unusual under the null model according to the chosen statistic.
That does not prove the null is true.
The study may be underpowered.
The effect may be small.
The measurement may be noisy.
The test statistic may be insensitive to the kind of departure that actually exists.
7. P Is Not the Probability the Null Hypothesis Is True
This is perhaps the most famous error.
If p = 0.03, it does not mean there is a 3% probability the null is true.
The p-value is calculated assuming the null model.
It does not invert the probability to obtain P(null | data).
That inversion would require additional modelling, such as a Bayesian prior and likelihood.
8. P Is Not the Probability the Result Happened “By Chance”
Chance is not one rival mechanism sitting outside the scientific model.
The p-value describes sampling variation under a specified model.
It does not assign a percentage of the observed effect to chance.
Nor does p = 0.02 mean there is a 2% probability the result was “just random”.
9. P Is Not the Probability of Replication
A p-value of 0.01 does not mean a replication has a 99% chance of succeeding.
Replication depends on the true effect size, design, measurement reliability, sample size, publication selection, context and what “replicate” means.
P-values are not replication forecasts.
10. P Is Not Effect Size
The ASA states this explicitly.
A tiny effect in a massive study can have p < 10-10.
A large effect in a tiny study can have p = 0.08.
The p-value combines effect magnitude with uncertainty and sample information.
For magnitude, see How Effect Sizes Work.
11. Statistical Significance Is a Threshold Decision Applied to the P-Value
If α = 0.05 is chosen in advance, p < 0.05 triggers rejection under the decision rule.
The threshold belongs to a testing procedure.
It does not create a natural scientific cliff at 0.05.
p = 0.049 and p = 0.051 contain almost the same numerical information.
Treating one as discovery and the other as failure exaggerates a conventional boundary.
12. Alpha Is Chosen Before the Data in a Well-Defined Test
NIST recommends deciding in advance how small the p-value must be to reject the null.
Choosing α after seeing the data changes the procedure.
If the threshold moves to accommodate the result, the stated error guarantee no longer corresponds to the actual decision process.
13. Alpha Is Not a Universal Law of Nature
0.05 is conventional.
Different decisions may justify different thresholds.
A high-stakes regulatory claim may require stronger error control.
An exploratory screen may use false-discovery-rate control rather than one universal α.
Error thresholds should reflect the inferential architecture and consequences.
14. One-Sided and Two-Sided P-Values Answer Different Questions
A one-sided test allocates the rejection region to one direction.
A two-sided test protects against departures in either direction.
A one-sided test is not justified merely because the observed result points in the desired direction.
The direction must be part of the pre-data scientific question.
15. Exact Tests and Asymptotic Tests Generate P-Values Differently
Some tests use an exact finite-sample distribution under the null.
Others use large-sample approximations such as normal or chi-squared distributions.
When samples are small, sparse or highly skewed, asymptotic p-values can behave poorly.
The p-value is only as calibrated as the reference distribution used to create it.
16. Randomisation Tests Build the Reference Distribution From the Assignment Mechanism
In a randomised experiment, one can sometimes calculate how unusual the observed difference is across treatment assignments allowed by the design.
The reference distribution comes from the randomisation mechanism rather than a parametric population model.
This shows that p-values are not one monolithic object.
Different procedures create different null reference worlds.
17. Permutation Tests Generalise the Same Intuition
Under exchangeability assumptions, labels can be permuted repeatedly.
The observed statistic is compared with the permutation distribution.
The resulting p-value is tied to the validity of the exchangeability structure.
Resampling does not remove the need for assumptions.
18. Model Misspecification Can Produce Misleading P-Values
Suppose a linear regression assumes independent, constant-variance errors.
The data are strongly clustered by school.
If clustering is ignored, standard errors may be too small and p-values too optimistic.
A p-value can be numerically exact and inferentially wrong because the reference model is wrong.
19. Large Samples Can Make Model Imperfections Highly Significant
With enough data, tiny deviations from a simple null become easy to detect.
This can be scientifically useful.
It can also produce very small p-values for effects too small to matter.
As sample size grows, magnitude and practical meaning become even more important companions to significance.
20. Small Samples Can Produce Large Effects With Modest P-Values
A noisy small experiment may estimate a large difference.
The standard error is also large.
The resulting p-value may not cross 0.05.
Calling the effect absent would confuse lack of resolution with evidence of zero.
See How Statistical Power Works.
21. Confidence Intervals Show More Than the Threshold
For many standard two-sided tests, p < 0.05 corresponds to a 95% confidence interval excluding the null value.
But the interval also shows which effect magnitudes remain compatible with the data and procedure.
See How Confidence Intervals Work.
22. “Significant” and “Not Significant” Are Not Opposites of Effect
Study A estimates a treatment effect of 10 with p = 0.04.
Study B estimates 9 with p = 0.06.
It would be absurd to claim the treatment works in A and does not work in B simply because the two p-values fall on opposite sides of an arbitrary boundary.
Direct comparison of effect estimates and uncertainties is required.
23. “Significant in One Group, Not in Another” Does Not Prove a Group Difference
A treatment is significant in younger participants and not significant in older participants.
The treatment effects may still be nearly identical.
To test whether groups differ, estimate the interaction or direct difference between effects.
Differences in significance are not automatically significant differences.
24. Multiple Testing Changes the Meaning of a Small P-Value
If one planned test produces p = 0.01, the search history is simple.
If 5,000 hypotheses were tested and the smallest p-value was 0.01, the result is not surprising at all under widespread nulls.
See How Multiple Testing Works.
25. Optional Stopping Changes the P-Value Procedure
Collect 30 observations.
Check p.
If p > 0.05, collect ten more.
Repeat until p < 0.05.
The final conventional p-value does not account for all the opportunities to stop.
Valid sequential methods explicitly model repeated looks.
26. HARKing Changes the Evidential Context Around the Same P-Value
A result is discovered after exploring data.
The paper then presents the hypothesis as though it were predicted in advance.
The numerical p-value may be unchanged.
The independence of the hypothesis from the data has changed.
Preregistration preserves that timing.
27. P-Hacking Is Search Over Analysis Paths
Try different exclusions.
Try different covariates.
Try transformations.
Try subgroups.
Stop when one route yields p < 0.05.
The reported p-value now sits at the end of an unreported optimisation process.
The solution is not merely “never analyse flexibly”.
It is to distinguish confirmatory from exploratory work and report the search path transparently.
28. A Very Small P-Value Does Not Rescue a Biased Study
A confounded observational study can produce p < 10-20.
The association may be estimated very precisely.
The causal interpretation can still be wrong.
Sampling uncertainty and systematic bias are different failure modes.
29. P-Values Do Not Diagnose Confounding
An adjusted coefficient with p = 0.001 does not prove all confounders were controlled.
The p-value describes uncertainty conditional on the fitted model and observed variables.
It cannot test the absence of unknown or unmeasured confounding by itself.
30. P-Values Do Not Diagnose Measurement Validity
A study can detect a highly significant difference in app logins.
If the claim is about durable learning, login count may be the wrong outcome.
Statistical incompatibility cannot repair construct mismatch.
31. P-Values and Likelihood Are Related but Different
A p-value asks about tail behaviour of a statistic under a null model.
A likelihood compares how different parameter values support the observed data under a model.
One cannot generally interpret a p-value as a likelihood ratio or Bayes factor.
Different inferential tools answer different questions.
32. P-Values and Bayes Factors Are Not Interchangeable
A Bayes factor compares predictive support for two specified models.
A p-value measures extremeness under one null reference model.
They can disagree because they encode different questions and assumptions.
33. P-Values Can Be Useful When the Question Is Designed for Them
The ASA’s 2021 task-force statement did not call for abandoning p-values.
It emphasised proper use alongside uncertainty, variability, multiplicity, replicability and sound scientific design.
The problem is not the existence of p-values.
The problem is asking one number to do jobs it was never designed to do.
34. Exact P-Values Are Better Than Stars
Reporting p = 0.047 gives more information than reporting “*”.
Reporting p = 0.0007 gives more information than “p < 0.05”.
Exact values help readers see how far the result lies from a decision threshold.
They should still be accompanied by effect estimates and uncertainty.
35. Extremely Small P-Values Should Not Be Reported With Fake Precision
Software may print p = 0.000000000000.
That does not mean the probability is literally zero.
It usually means the value is smaller than the software’s display resolution or numerical threshold.
Report sensible bounds such as p < 0.001 or a more informative calculated value when reliable.
36. A P-Value Is Sensitive to Sample Size
Hold the observed effect roughly constant.
Increase sample size.
The standard error tends to shrink.
The test statistic becomes larger in absolute value.
The p-value often becomes smaller.
This is another reason p-values cannot be effect-size measures.
37. A P-Value Can Be Sensitive to Analytic Representation
Raw outcome or log outcome?
Linear model or robust regression?
Covariate adjustment or not?
These choices can produce different p-values because they change the estimand, standard error or model.
Analytic sensitivity should be explored rather than hidden.
38. Robustness Is Better Shown Across Reasonable Analyses Than by One Lucky P-Value
If a result survives several pre-specified or scientifically justified analyses with similar effect magnitude, confidence increases.
If one model gives p = 0.049 and ten reasonable alternatives give p between 0.2 and 0.8, the threshold-crossing result is fragile.
Robustness is a landscape, not one coordinate.
39. Null Results Need Equivalence Logic When “No Meaningful Effect” Is the Claim
Failing to reject zero does not establish that the effect is negligible.
If the scientific claim is “the two treatments are effectively equivalent”, define a negligible-effect region and test equivalence with enough precision.
The null hypothesis should match the scientific claim.
40. P-Values Can Be Misleading After Model Selection
Select predictors using the same data.
Then report conventional p-values as though the selected model had been fixed in advance.
The ordinary inferential calibration can fail because selection altered the distribution.
Post-selection inference is a distinct problem.
41. P-Values in Machine Learning Benchmarks Can Hide Repeated Search
Two models are compared on a benchmark and a paired test yields p = 0.01.
If the benchmark has been used to tune hundreds of model variants, the final comparison is not independent of the search process.
Fresh holdout data matter more than ceremonial significance testing on an overused benchmark.
42. P-Values in Education Need the Learning Construct Before the Threshold
A new teaching method produces p = 0.002 on immediate recall.
If the claim is durable transfer six months later, the significant result answers the wrong question.
Statistical inference cannot substitute for educational validity.
43. P-Values in Medicine Need Harms, Benefits and Baseline Risk
A treatment has p = 0.0001 for a small biomarker improvement.
The clinical decision still needs absolute benefit, harms, patient values, uncertainty and alternatives.
A p-value can contribute to evidence.
It cannot make the treatment decision alone.
44. The Hostile Test: P = 0.049 Versus P = 0.051
Two nearly identical studies estimate nearly identical effects.
One produces p = 0.049.
The other produces p = 0.051.
Calling the first success and the second failure creates a scientific discontinuity that the evidence does not contain.
45. The Second Hostile Test: P = 0.000001 for a Trivial Effect
Ten million observations show a 0.01-point score increase.
The p-value is microscopic.
The effect is too small to matter educationally.
Statistical incompatibility with exact zero is not practical importance.
46. The Third Hostile Test: P = 0.20 From a Tiny Study
A treatment effect is estimated at +12 points.
The sample is tiny and the confidence interval runs from -8 to +32.
p = 0.20.
The study does not show no effect.
It shows insufficient resolution.
47. The Fourth Hostile Test: One Significant Result From Fifty Outcomes
Fifty outcomes are tested.
One gives p = 0.03.
The other forty-nine are not discussed.
The single p-value cannot be interpreted as though the other opportunities never existed.
48. The Fifth Hostile Test: Significant Association, Invalid Causal Claim
Students who attend tutoring score higher with p < 0.001.
Motivation, prior attainment and parental support differ between groups.
The association is statistically clear.
The causal effect remains unidentified.
49. Primary School: The Seed Is “Surprising Under a Rule”
A child has a fair coin.
Ten heads in a row would be surprising if the coin were fair.
The child does not need formal p-values yet.
The foundational idea is:
If a rule were true, how unusual would this result be?
50. Secondary School: Separate Surprise From Proof
Students can learn that an unusual result under a model counts against that model.
It does not prove one preferred explanation automatically.
Alternative mechanisms, measurement errors and design flaws still need evaluation.
51. JC and University: P-Values Become Operating Characteristics of Tests
At higher levels, learners should reconstruct:
- null and alternative;
- test statistic;
- reference distribution;
- one- or two-sided tail;
- alpha;
- Type I error;
- power;
- multiplicity;
- stopping rule;
- effect estimate;
- confidence interval;
- model assumptions.
The p-value becomes one output of an inference procedure rather than a verdict from the universe.
52. Where P-Values Fit in the eduKateSG “How Works” Landscape
- How Hypotheses Work — the claims being subjected to tests.
- How Statistical Power Works — the probability of detecting specified alternatives.
- How Effect Sizes Work — magnitude that p-values do not measure.
- How Confidence Intervals Work — precision and compatible magnitudes.
- How Multiple Testing Works — what happens when many opportunities for significance exist.
- How Preregistration Works — protects analysis timing and planned hypotheses.
- How Research Validity Works — asks whether the inference answers the intended scientific question.
P-values own one precise canonical role: quantify how extreme the chosen test statistic is relative to its reference distribution under a specified null model and testing procedure.
53. What This Article Does Not Claim
- A p-value is not the probability the null hypothesis is true.
- A p-value is not the probability the result happened by chance.
- A p-value is not an effect size.
- A p-value is not the probability of successful replication.
- p < 0.05 is not a natural scientific law.
- p > 0.05 does not prove no effect.
- A tiny p-value cannot repair confounding, measurement bias or poor design.
- Multiplicity and optional stopping can invalidate naive single-test interpretations.
- Exact reporting is better than binary significance labels, but p-values still need effect size and uncertainty.
- Scientific conclusions should not be based on p-values alone.
54. A Compact P-Value Audit
- What null model is being tested?
- What alternative is relevant?
- What test statistic was chosen?
- Why is that statistic appropriate?
- What reference distribution generates the p-value?
- Is the test exact, asymptotic, permutation-based or randomisation-based?
- Was the test one-sided or two-sided?
- Was direction chosen before seeing the data?
- What alpha was defined?
- Was alpha chosen prospectively?
- What effect size was estimated?
- What confidence interval surrounds it?
- Is the effect practically meaningful?
- Was sample size adequate?
- How many hypotheses were tested?
- Was multiplicity controlled?
- Were repeated interim looks performed?
- Was optional stopping accounted for?
- Were analysis choices pre-specified?
- Could model misspecification or clustering distort the p-value?
- Could confounding or measurement bias make the causal interpretation invalid?
- Would an alternative inferential framework answer the scientific question better?
55. Frequently Asked Questions
What is a p-value?
A p-value is the probability, under a specified null model and testing procedure, of obtaining a test statistic at least as extreme as the observed statistic.
Does p = 0.03 mean there is a 3% chance the null is true?
No. The p-value is calculated assuming the null model. It does not provide the reverse probability that the null hypothesis is true given the data.
Does p > 0.05 mean there is no effect?
No. The study may be compatible with meaningful effects but too imprecise to detect them. Effect estimates, confidence intervals and power are needed to distinguish evidence of negligible effect from inconclusive evidence.
Why is 0.05 used?
It is a historical convention for Type I error control, not a universal boundary of scientific truth. Different contexts can justify different thresholds and error-control strategies.
56. Authoritative Research Corridor
- American Statistical Association Statement on Statistical Significance and P-Values
- ASA President’s Task Force Statement on Statistical Significance and Replicability
- NIST/SEMATECH — Critical Values and P-Values
- CONSORT 2025 Explanation and Elaboration — Effect Estimates, Confidence Intervals and Statistical Interpretation
Final Thought: A P-Value Is a Question Asked of a Model
The p-value is neither hero nor villain.
It is a specialised instrument.
It asks one narrow question:
If this reference model were true, how unusual would evidence at least this extreme be under the procedure we used?
That can be useful.
But the answer still needs the rest of science.
How large is the effect?
How precise is it?
How many tests were tried?
Was the design biased?
Does the outcome mean what we think it means?
Will the result replicate?
The mistake was never that science had a p-value. The mistake was asking one tail probability to carry the full weight of inference.