Statistics is the mathematics of learning from data under uncertainty. It begins after the world has already produced observations and asks what those observations can legitimately tell us about a larger process, population, mechanism or future.
Statistics is not merely tables, charts and averages. It is a system for controlling the distance between what was observed and what is being claimed. Good statistics makes that distance explicit. Bad statistics hides it.
Series route: Mathematics Learning Hub → How Mathematics Works → Statistics. Probability is the mathematical partner that supplies much of the uncertainty machinery: How Mathematics Works | Probability.
1. What statistics is
Statistics studies data: how it is generated, collected, summarised, modelled, compared and used for inference. It asks questions such as:
- What does this dataset describe?
- How variable are the observations?
- Is the sample representative of a larger population?
- How uncertain is an estimate?
- Is an observed difference compatible with chance variation?
- Which model best captures the structure?
- Can an association support a causal claim?
The field therefore links mathematics, evidence and decision-making.
2. Data is produced, not simply found
Every dataset comes from a process: measurement, survey, experiment, transaction log, sensor, observation or administrative system. That process determines what the data can mean.
A spreadsheet may look objective, but the variables were chosen, categories defined, measurements rounded, missing values handled and observations included or excluded through human or machine decisions.
Statistical reasoning therefore begins upstream. Before asking what the numbers show, ask how the numbers came to exist.
3. Population and sample are different objects
A population is the full collection of units or outcomes we care about. A sample is the subset actually observed. Statistics often uses the sample to infer something about the population.
The gap between sample and population is where sampling uncertainty enters. Even a perfectly measured random sample can differ from the population simply because only part of the population was observed.
4. Sampling design controls what generalisation is possible
A large biased sample can be worse than a smaller well-designed sample. If the sampling process systematically overrepresents some groups and underrepresents others, more data may only estimate the wrong target more precisely.
Random sampling helps by giving units known or controlled chances of selection. Stratification can preserve representation across important subgroups. Cluster and multistage designs trade statistical simplicity for practical feasibility.
The design of the sample is therefore part of the mathematics of inference.
5. Descriptive statistics compress data
Raw data can be too large or noisy to inspect directly. Descriptive statistics compress it into summaries such as means, medians, quartiles, variances, ranges and frequencies.
Compression is useful only if the summaries preserve the aspects that matter. A mean can hide multimodality. A range can be dominated by one outlier. Two datasets can share the same mean and variance while having very different shapes.
Good statistics therefore uses summaries as views, not substitutes for the data-generating structure.
6. The mean is a balance point
The arithmetic mean is the sum of observations divided by their number. It is also the value that minimises the sum of squared deviations. This connects the familiar average to optimisation and later to least-squares regression.
The mean is sensitive to extreme values because every observation contributes numerically. That sensitivity can be useful or problematic depending on the question.
7. The median is an order statistic
The median is the central observation after sorting. It is less sensitive to extreme values than the mean and often better represents the centre of skewed distributions.
Choosing between mean and median is not a contest over which is “better.” It is a question about which summary matches the geometry and purpose of the data.
8. Variation is part of the signal
Two groups can have the same mean and very different consistency. Variance and standard deviation measure spread around the mean and therefore describe how tightly observations cluster.
Continue to How Variance Works.
Variation is not always noise to remove. It may be the phenomenon of interest: unequal performance, heterogeneous responses, volatile demand or unstable processes.
9. Distributions preserve shape information
A distribution describes how values are spread across the range of possibilities. Shape can include symmetry, skewness, heavy tails, multiple modes and outliers.
This shape often matters more than a single central summary because real decisions depend on tail events, concentration and asymmetry.
10. Visualisation is statistical reasoning, not decoration
Histograms, box plots, scatterplots, line charts and empirical distributions reveal structure that numerical summaries may hide. But a graph can also mislead through truncated axes, inappropriate scales, aggregation or selective ranges.
The graph is another representation layer. Its job is to make relevant structure visible without distorting the evidence.
11. Inferential statistics moves from sample to population
Inference asks what can be learned about an unobserved population or process from observed data. Point estimates provide a best single estimate. Interval estimates show a range of values compatible with the data and method.
Inference is impossible without assumptions. Sampling assumptions, model form, independence, measurement quality and missing-data mechanisms all influence what the final interval or test means.
12. Standard error measures uncertainty in an estimate
Standard deviation describes variation among observations. Standard error describes variation in an estimator across hypothetical repeated samples under the model.
This distinction is fundamental. A population can be highly variable while its mean is estimated precisely from a very large representative sample.
13. Confidence intervals communicate a procedure, not a magic probability
In a standard frequentist interpretation, a 95% confidence procedure is designed so that across repeated samples generated under the assumptions, 95% of the resulting intervals contain the true parameter.
After one interval is computed, the parameter is not treated as randomly moving in that framework. This is why careful statistical language matters.
14. Hypothesis testing compares data with a null model
A hypothesis test asks how compatible the observed data are with a specified null hypothesis under a testing procedure. A p-value measures how extreme the observed statistic, or something more extreme, would be if the null model were true and the assumptions held.
A p-value is not the probability that the null hypothesis is true. It is also not an effect size, a measure of practical importance or a guarantee of replication.
15. Statistical significance is not practical importance
With enough data, tiny effects can become statistically detectable. With little data, important effects can remain uncertain. Therefore statistical significance and practical significance answer different questions.
Good analysis reports effect sizes, uncertainty and context rather than reducing evidence to a binary significant/not-significant label.
16. Statistical power is detection capability
Power is the probability that a statistical procedure will detect an effect of a specified size under specified conditions. It depends on effect size, variability, sample size, significance threshold and design.
Low power creates more than missed detections. It can also make discovered estimates unstable and exaggerated.
Continue to How Statistical Power Works.
17. Likelihood asks how well parameter values explain observed data
A likelihood function treats the observed data as fixed and compares how strongly different parameter values support that observation under a model.
Maximum likelihood estimation chooses parameter values that maximise that support. This machinery connects probability models to estimation and appears across modern statistics.
Continue to How Likelihood Works.
18. Bayesian statistics updates distributions over unknowns
Bayesian inference begins with a prior distribution, combines it with a likelihood from observed data, and produces a posterior distribution. The posterior represents uncertainty about unknown quantities conditional on the model and evidence.
This framework makes probability statements about parameters natural, but it also requires explicit choices about priors and models.
19. Regression studies relationships between variables
Regression models how an outcome changes with one or more predictors. Linear regression fits a linear conditional relationship. More general regression methods handle nonlinear responses, counts, binary outcomes, survival times and other data types.
Regression coefficients are not automatically causal effects. They describe conditional associations under the model unless causal identification is separately justified.
20. Correlation is not causation
Correlation measures association, not mechanism. Two variables can move together because one causes the other, because the reverse direction holds, because a third factor affects both, because of selection effects or because of chance.
Causal claims require stronger design and reasoning: randomisation, natural experiments, adjustment under defensible assumptions, causal graphs or other identification strategies.
21. Covariance is the bridge into multivariate structure
When variables are analysed together, their co-movement matters. Covariance matrices summarise pairwise linear relationships and become central in multivariate statistics, portfolio theory and machine learning.
Continue to How Covariance Works.
22. Confounding creates false simplicity
A confounder is a variable related to both an exposure and an outcome in a way that can distort the observed association. Ignoring confounding can produce persuasive but wrong stories.
Statistics therefore needs subject knowledge. Pure numerical adjustment cannot decide by itself which variables are causes, consequences, colliders or irrelevant measurements.
23. Missing data is a model problem
Missing values are not just empty cells. Why values are missing determines whether simple deletion is harmless or biased. Missingness related to unobserved outcomes can fundamentally change inference.
Statistical analysis must therefore model not only observed values but sometimes the observation process itself.
24. Outliers can be errors or information
An outlier may come from data-entry error, measurement failure, rare but valid behaviour, a new subgroup or a model that does not fit the tails.
Automatically deleting outliers can erase the phenomenon. Automatically retaining them can distort fragile estimates. The correct response is diagnostic: investigate origin, influence and relevance.
25. Multiple comparisons create false discoveries
If enough hypotheses are tested, some will appear unusual by chance even when all nulls are true. This is why exploratory analysis can generate impressive-looking results that fail to replicate.
Correction methods, preregistration, validation data and replication help control this search problem.
26. Overfitting is learning the sample too well
A model can fit observed data beautifully and predict new data poorly. Overfitting occurs when the model captures noise or sample-specific quirks as if they were stable structure.
Cross-validation, regularisation, holdout sets and simpler models help estimate whether a pattern transfers beyond the data used to discover it.
27. Model checking is part of inference
Statistical models make assumptions about distributions, independence, functional form, variance and data generation. Diagnostics ask whether those assumptions are reasonably compatible with observations.
No model is reality. The aim is a model adequate for the intended task, with limitations understood.
28. A worked mechanism: two classes with the same average
Suppose Class A scores are 68, 69, 70, 71, 72 and Class B scores are 40, 55, 70, 85, 100. Both means are 70.
- The mean alone says the centres match.
- The spreads are radically different.
- Class A is tightly concentrated; Class B is heterogeneous.
- A teaching decision based only on the mean would miss that difference.
This is the core statistical lesson: a summary answers one question and may conceal another.
29. Common statistical failure modes
- Selection bias: the observed sample does not represent the target population.
- Confusing standard deviation with standard error.
- P-value inversion: reading a p-value as the probability a hypothesis is true.
- Significance worship: ignoring effect size and practical importance.
- Correlation-to-causation leap: treating association as mechanism.
- Multiple-testing blindness: searching until something looks significant.
- Overfitting: mistaking sample noise for transferable structure.
- Data-generation blindness: analysing numbers without understanding how they were produced.
30. Statistics and probability run in opposite directions
Probability often says: assume a model, then calculate what data or outcomes are likely. Statistics says: observe data, then infer which models or parameters remain plausible.
This forward/backward pair is one of the central architectures of mathematical inference.
31. Statistics in science
Experiments and observational studies use statistics to estimate effects, quantify uncertainty and distinguish reproducible signals from variation. Scientific evidence is rarely a single number; it is a chain from design through measurement, model, inference and replication.
32. Statistics in policy, medicine and business
Policy uses population data, surveys and causal evaluation. Medicine uses trials, diagnostics and survival analysis. Business uses forecasting, experimentation and customer data. In every case, statistical quality depends as much on design and interpretation as on calculation.
33. Statistics in machine learning
Machine learning shares many statistical foundations: loss functions, probability models, estimation, generalisation, validation and uncertainty. The vocabulary may differ, but the core problem remains learning transferable structure from finite data.
34. Statistics as a machine
A useful machine model is:
Question → Data-Generating Process → Sample/Measurement → Description → Model → Uncertainty Quantification → Inference → Validation/Decision.
The machine fails when the question is vague, the sample is biased, the measurements are poor, the model is inappropriate, uncertainty is hidden, or conclusions extend beyond the design.
35. What statistics mastery looks like
- ask how the data were generated;
- distinguish sample from population;
- read centre, spread and distribution shape together;
- quantify uncertainty around estimates;
- interpret p-values and confidence intervals correctly;
- separate association from causal identification;
- check model assumptions and transfer to new data;
- match statistical claims to the evidence actually collected.
36. Conclusion
Statistics works by converting observations into carefully bounded claims. It compresses data, models variation, estimates hidden quantities, measures uncertainty and tests whether apparent patterns survive contact with sampling and noise.
Its deepest discipline is not calculation. It is epistemic control: knowing exactly how far the evidence allows us to go.
Probability tells us what a model predicts. Statistics tells us what data allow us to infer. Together they form the mathematical architecture of reasoning under uncertainty.
How Mathematics Works | Batch 02
- How Mathematics Works | Probability
- Statistics — this article
- How Mathematics Works | Number Theory
- How Mathematics Works | Mathematical Logic & Foundations
Return to the Mathematics Learning Hub or the How Mathematics Works root.