Statistics is the discipline of learning from incomplete, variable and noisy information. It asks how a sample can tell us something about a population, how uncertainty should be quantified, how patterns can be separated from chance, how models can be checked against data, and how claims should change when evidence is weak, biased or incomplete.
Statistics works because data are not self-explanatory. Every dataset is produced by a process: someone defined what to measure, selected cases, recorded values, handled missingness and chose a model. Statistical reasoning makes those hidden decisions visible so conclusions can be evaluated rather than merely computed.
The shortest useful answer
Statistics works by connecting question → population → sample → measurement → model → uncertainty → inference → checking → decision. The central task is not to eliminate uncertainty, but to represent it honestly enough that evidence can guide action.
- Population: the broader set we want to understand.
- Sample: the observed subset that supplies evidence.
- Variable: a measurable characteristic.
- Distribution: how values are spread and repeated.
- Estimator: a rule for using sample data to estimate an unknown quantity.
- Uncertainty: the range of plausible error around the estimate.
- Model: a simplified representation of the data-generating process.
- Inference: what the data justify us in concluding beyond the observed sample.
1. Statistics begins before the data
A statistical analysis starts with a question. What population matters? Which outcome should represent the phenomenon? What comparison would answer the question? If those choices are vague, even perfect arithmetic can produce an irrelevant result.
Good statistics therefore treats research design as part of analysis. The most sophisticated model cannot recover information that was never measured or repair a sample that systematically excludes the people of interest without strong assumptions.
2. Data are traces of a generating process
Numbers do not appear spontaneously. A test score reflects a learner, a test, instructions, timing and scoring rules. A sales record reflects customers who reached the store, products offered, prices, inventory and recording systems. A medical measurement reflects patient state, instrument performance and measurement procedure.
The data-generating process determines what kinds of conclusions are possible. Statistics becomes stronger when analysts model how the data came into existence rather than treating the spreadsheet as reality itself.
3. Measurement defines what a number means
Variables can be categorical, ordinal, interval-like, counts, proportions, times or continuous quantities. The scale of measurement affects which summaries and models make sense.
A precise number can still measure the wrong construct. Validity asks whether the observed variable actually supports the interpretation we want to make.
4. Distributions reveal structure hidden by averages
The mean compresses many values into one centre, but different distributions can share the same mean. Spread, skewness, multimodality and outliers can change the story completely.
Histograms, box plots, empirical distributions and quantiles often reveal more than a single summary. Statistics asks not only “what is typical?” but “how variable is the system, and where are the unusual cases?”
5. Sampling turns part of the world into evidence about the whole
A sample supports population inference only when the route from population to sample is understood. Random sampling helps because each population member has a known or controlled chance of selection.
Convenience samples can still be useful, but generalisation becomes assumption-heavy. A large biased sample can be more misleading than a smaller well-designed one.
6. Sampling variability is unavoidable
Two random samples from the same population will not produce identical results. This variation is not a mistake; it is the natural consequence of observing only part of the population.
Standard errors estimate how much an estimator would vary across repeated samples under the model. This is one of statistics’ central moves: uncertainty about an unseen population is represented through hypothetical repetition.
7. Probability provides the language of uncertainty
Probability models uncertain events and repeated variability. It can describe frequencies in repeated processes, degrees of belief under a formal model, or both depending on framework.
Conditional probability is especially important because evidence changes what is plausible. The probability of a condition after a positive test depends not only on test accuracy but also on how common the condition was before the test.
A worked way of thinking: base rates
Suppose a rare condition affects 1 in 1,000 people and a test is highly accurate. Even then, false positives among the 999 people without the condition may outnumber true positives. The lesson is general: evidence must be combined with prior prevalence, not interpreted in isolation.
8. Estimation asks how large an effect is
Point estimates give one best-fitting value: a mean difference, regression coefficient, proportion or risk ratio. But a point estimate without uncertainty can create false precision.
Good statistics reports effect sizes with intervals or posterior distributions so readers can see both magnitude and uncertainty.
9. Confidence intervals express sampling uncertainty
A confidence interval is generated by a procedure that, under repeated sampling and its assumptions, captures the true parameter at the stated long-run rate. It is not literally the probability that a fixed parameter lies inside one realised interval under the classical interpretation.
The practical value of intervals is that they show how compatible a range of parameter values is with the data and method.
10. Hypothesis tests compare data with a null model
A hypothesis test asks how surprising the observed data, or something more extreme, would be if a specified null model were true. The p-value is one summary of that calculation.
A small p-value does not measure the probability that the null hypothesis is true. Nor does it measure practical importance. Statistical significance and substantive importance are different questions.
11. Errors come in more than one form
Decision procedures can produce false positives and false negatives. Tightening a threshold to reduce one type of error can increase the other.
The appropriate balance depends on consequences. Screening, safety systems, fraud detection and scientific discovery may tolerate different error trade-offs.
12. Statistical power is the chance to detect a real effect under specified conditions
Power depends on effect size, sample size, noise, measurement quality and decision threshold. Low-powered studies can miss real effects and can also exaggerate the magnitude of effects that happen to reach significance.
Planning sample size before data collection is therefore part of scientific design, not an administrative detail.
13. Regression describes conditional relationships
Regression models how an outcome changes with one or more predictors. Linear regression estimates expected differences associated with predictor changes while holding included variables fixed.
The coefficients are only as meaningful as the model and design. Adding more variables does not automatically create causality. Poorly chosen controls can even introduce bias.
14. Correlation summarises association, not mechanism
Correlation measures how two variables move together, usually linearly. A strong correlation can arise from causation, reverse causation, a shared cause, selection or chance.
Statistics strengthens causal reasoning when the design explains why competing pathways are blocked, not when the correlation coefficient is merely large.
15. Causal inference requires a counterfactual
The causal effect of a treatment is the difference between what happened with the treatment and what would have happened to the same unit without it. The second outcome is unobservable.
Randomised experiments estimate this missing counterfactual by creating comparable groups. Observational methods attempt to recover comparable conditions through design and modelling.
16. Confounding mixes causes together
A confounder influences both the exposure and the outcome, creating an association that may not reflect the causal effect of interest. For example, exercise and health may be confounded by age, prior health, income or other factors.
Identifying confounders is a causal reasoning task, not merely a software setting.
17. Bayesian statistics updates uncertainty directly
Bayesian inference combines a prior distribution with a likelihood to produce a posterior distribution. The posterior represents uncertainty about model parameters after observing the data.
Bayesian methods make prior assumptions explicit and support direct probability statements about parameters, but results still depend on model quality and prior choice.
18. Prediction and explanation are different jobs
A model can predict accurately without providing a causal explanation. Conversely, a scientifically meaningful causal model may not maximise short-term predictive accuracy.
Statistics improves when the job is stated clearly: estimate a parameter, test a theory, predict future outcomes, classify cases or support a decision.
19. Overfitting mistakes noise for structure
A flexible model can fit observed data extremely well by learning accidental quirks. Performance then collapses on new data.
Cross-validation, regularisation and held-out testing estimate whether the learned pattern generalises beyond the training sample.
20. Model assumptions define the operating envelope
Models assume things about functional form, independence, error distribution, measurement and sampling. Violations do not always destroy an analysis, but they change how trustworthy the result is.
Residual plots, posterior predictive checks, sensitivity analysis and alternative specifications help test whether conclusions depend dangerously on one assumption.
21. Missing data are part of the mechanism
Data can be missing completely at random, related to observed variables, or related to unobserved values themselves. These cases have different implications.
Dropping incomplete records can create bias when missingness is systematic. Imputation can help, but only under assumptions that should be stated.
22. Multiple testing increases false discoveries
If hundreds of hypotheses are tested, some small p-values will appear by chance. Correction procedures and preregistration help control this search problem.
The deeper lesson is that the number of questions asked of the data matters. Selective reporting can create apparently convincing evidence from noise.
23. Reproducibility separates computation from credibility
A reproducible analysis can be rerun from data and code to obtain the same result. This is essential but not sufficient. The same reproducible pipeline can faithfully reproduce a biased design.
Credibility therefore requires both computational reproducibility and defensible measurement, sampling and causal logic.
24. Statistical graphics are reasoning tools
Plots reveal distribution, outliers, nonlinear relationships, group differences and time patterns that summary statistics can hide. Good graphics reduce the cognitive cost of seeing structure.
But visualisation can also mislead through truncated axes, selective ranges, excessive smoothing or inappropriate scales. The graphic is part of the statistical argument.
25. Common misconceptions
- “A large sample removes all bias.” Large samples reduce random error, not systematic bias.
- “Statistically significant means important.” Small effects can be highly significant in large samples.
- “A p-value is the probability the null is true.” It is calculated assuming the null model.
- “Correlation proves nothing.” Correlation is useful evidence, but causal interpretation requires additional design and assumptions.
- “More variables always improve a regression.” Irrelevant or post-treatment controls can worsen inference.
- “A model that fits the data well must be true.” Many different models can fit the same data.
- “Uncertainty is weakness.” Quantified uncertainty is a strength because it reveals the limits of the evidence.
26. How to solve a statistics problem
- State the question and target population.
- Define the estimand: what quantity actually answers the question?
- Inspect how the sample was obtained.
- Check how each variable was measured.
- Explore the distribution before modelling.
- Choose a model that matches the data type and job.
- Estimate effect size and uncertainty.
- Check assumptions and residual patterns.
- Separate association from causation.
- Test sensitivity to reasonable alternative choices.
- Validate predictions on new data when prediction is the goal.
- Report limitations and the conditions under which the conclusion may fail.
27. A compact map of the discipline
- Descriptive statistics: summarising distributions and relationships.
- Probability: mathematical models of uncertainty.
- Statistical inference: estimation and testing from samples.
- Regression and multivariate analysis: conditional relationships among variables.
- Experimental design: creating informative comparisons.
- Causal inference: estimating effects under explicit identification assumptions.
- Bayesian statistics: updating uncertainty through probability models.
- Time series: dependence and forecasting through time.
- Survey sampling: population inference from designed samples.
- Statistical learning: prediction, regularisation and generalisation.
The deeper answer: statistics works by making the unseen visible through uncertainty
We rarely observe the whole population, every future outcome or the counterfactual world in which a decision was not made. Statistics is the machinery that connects limited observations to disciplined claims about those unseen quantities.
The field is strongest when it resists false certainty. A good statistical answer does not merely provide a number. It explains where the number came from, what assumptions hold it up, how uncertain it is, how it was checked and what new evidence would change the conclusion.
Continue: How X Works Hub — the eduKateSG map of connected explanations across subjects.