Likelihood works by taking a probability model that describes how possible data would arise under different parameter values, then—after the actual data are observed—holding those data fixed and comparing how strongly different parameter values support that observed dataset. The likelihood function is therefore a function of the unknown parameters for fixed data. Maximum likelihood chooses the parameter values that maximise that function, while likelihood ratios compare relative support between parameter values or models.
Likelihood begins with a small conceptual reversal.
Before data arrive, a probability model asks:
If the parameter were θ, what data might I see?
After data arrive, likelihood asks:
Given the data I actually saw, which parameter values make those observations most supported by this model?
The formula may be mathematically the same expression.
The variable that is allowed to move has changed.
Quick Read
PROBABILITY MODEL → OBSERVED DATA → HOLD DATA FIXED → VARY PARAMETERS → LIKELIHOOD FUNCTION → LOG-LIKELIHOOD → MAXIMUM → MLE → CURVATURE / INFORMATION → STANDARD ERROR / PROFILE → LIKELIHOOD RATIOS → MODEL CHECKS → INTERPRETATION
NIST describes maximum likelihood estimation as beginning with a likelihood function for the sample data and choosing the unknown parameter values that maximise that sample likelihood. The method is widely applicable, including to censored data, and has important large-sample properties.
1. Probability and Likelihood Use the Same Model in Different Directions
Suppose a coin has probability θ of heads.
Before flipping, probability asks how likely different sequences are for a fixed θ.
After observing seven heads in ten flips, likelihood holds that result fixed and asks how the support varies as θ moves from 0 to 1.
The direction of inquiry changed.
2. Likelihood Is Not a Probability Distribution Over Parameters by Itself
The likelihood for θ need not integrate or sum to one over θ.
Its absolute scale is arbitrary up to multiplication by a positive constant that does not depend on θ.
What matters for ordinary likelihood inference is relative support across parameter values.
To obtain a probability distribution over parameters, Bayesian inference combines likelihood with a prior and normalises the result.
3. The Likelihood Function Is Indexed by a Model
The same data can generate different likelihoods under different statistical models.
A normal model, Poisson model and negative-binomial model encode different data-generating assumptions.
Likelihood cannot escape model specification.
A beautifully peaked likelihood under the wrong model can still support the wrong scientific representation.
4. Independent Observations Multiply Their Likelihood Contributions
If observations are conditionally independent given θ, the joint likelihood is the product of their individual probability densities or mass functions.
Each new observation contributes another factor.
This multiplication is why likelihood can become extremely small numerically even when the model is reasonable.
5. Log-Likelihood Turns Products Into Sums
Take logarithms.
Products become sums.
The parameter value that maximises likelihood also maximises log-likelihood because log is monotonic.
Log-likelihood is numerically more stable and easier to differentiate, which is why most optimisation software works on it.
6. Maximum Likelihood Chooses the Best-Supported Parameter Under the Model
The maximum likelihood estimate, or MLE, is the parameter value that maximises L(θ|data).
For a Bernoulli model, the MLE of θ is the observed proportion of successes.
For a normal model with known variance, the MLE of the mean is the sample mean.
Different familiar estimators can therefore emerge from one general principle.
7. Maximum Likelihood Is a Model-Fitting Principle, Not a Truth Detector
The MLE identifies the parameter value that best supports the observed data among the values allowed by the model.
If every model in the candidate family is wrong, the MLE chooses the least-wrong member according to the likelihood geometry.
Optimisation within a model is not validation of the model.
8. In Normal Regression, Least Squares and Maximum Likelihood Meet
Assume independent normal errors with constant variance.
Maximising the likelihood with respect to regression coefficients is equivalent to minimising the sum of squared residuals.
This connects least-squares regression to likelihood theory.
9. Different Error Models Produce Different Loss Functions
Normal errors imply squared-error fitting.
Laplace errors connect to absolute-error fitting.
Poisson likelihood produces the deviance structure used for count regression.
The loss function is often the negative log-likelihood of a probabilistic model.
Optimisation geometry and probability assumptions are two views of the same machinery.
10. The Shape of the Likelihood Contains Uncertainty Information
A sharply peaked likelihood means nearby parameter values lose support quickly.
A broad flat likelihood means many parameter values fit the observed data similarly.
Uncertainty is therefore not only where the maximum sits.
It is also how the support falls away from the maximum.
11. Curvature Near the Maximum Measures Local Information
The second derivative of the log-likelihood describes local curvature.
Strong negative curvature means a narrow peak.
The inverse curvature is closely related to the variance of the MLE in regular large-sample settings.
Geometry becomes statistical precision.
12. Fisher Information Formalises Expected Curvature
Fisher information measures how strongly the data distribution changes with the parameter.
More information generally means smaller estimator variance.
Independent observations often contribute information additively.
This is one mathematical reason larger samples tend to sharpen likelihoods.
13. The Score Function Is the Slope of the Log-Likelihood
The score is the derivative of log-likelihood with respect to a parameter.
At a smooth interior maximum, the score is zero.
Score equations therefore provide one route to calculating MLEs.
They also underlie score tests and influence-function theory.
14. MLEs Have Powerful Large-Sample Properties Under Regularity Conditions
NIST highlights the desirable properties of maximum likelihood estimators.
Under regular conditions, MLEs are consistent, asymptotically normal and efficient in important senses.
These are asymptotic guarantees.
Small samples, boundaries, non-identifiability or model misspecification can break the simple approximation.
15. Consistency Means Convergence to the Model’s Target, Not Necessarily Reality
If the model is correctly specified and regularity conditions hold, the MLE tends toward the true parameter as sample size grows.
If the model is misspecified, the estimator can instead converge precisely to a pseudo-true parameter that best approximates the data-generating distribution inside the wrong family.
Consistency cannot rescue the wrong target.
16. The Invariance Property Makes Transformations Convenient
If θ̂ is the MLE of θ and g is a one-to-one transformation, then g(θ̂) is the MLE of g(θ).
Estimate log-odds and exponentiate to obtain the MLE of the odds.
This property makes likelihood especially convenient across parameterisations.
17. Parameterisation Changes Coordinates, Not the Underlying Fit
A normal model can be parameterised by variance σ² or standard deviation σ.
The likelihood surface changes coordinates, but the fitted distribution can represent the same model.
Interpretation should focus on scientifically meaningful parameter functions, not arbitrary coordinates.
18. Nuisance Parameters Are Needed for the Model but Not the Main Scientific Target
A study may care about a treatment coefficient while the model also contains intercepts, variance parameters and baseline covariates.
These nuisance parameters affect the likelihood and uncertainty around the target.
Inference must account for them even when they are not scientifically interesting.
19. Profile Likelihood Removes Nuisance Parameters by Re-Optimising Them
Fix the parameter of interest at one candidate value.
Maximise the likelihood over all nuisance parameters.
Repeat across candidate values.
The resulting profile likelihood shows support for the target parameter after giving nuisance parameters their best possible values at each point.
20. Profile-Likelihood Intervals Can Be Asymmetric
Wald intervals often use estimate ± constant × standard error and are therefore symmetric on the chosen scale.
Likelihood surfaces can be skewed.
Profile-likelihood intervals follow that asymmetry and can behave better near parameter boundaries or in nonlinear models.
21. A Likelihood Ratio Compares Relative Support
If L(θ₁) is twice L(θ₂), then θ₁ has twice the likelihood support of θ₂ for the observed data under the model.
This is a relative statement.
It does not mean θ₁ has twice the probability of being true unless a Bayesian framework supplies priors and posterior normalisation.
22. Likelihood-Ratio Tests Compare Nested Models
Fit a restricted model under the null.
Fit a larger model under the alternative.
Compare twice the difference in maximised log-likelihoods.
Under regular conditions, the statistic has an asymptotic chi-squared reference distribution based on the difference in parameter dimension.
23. Wald, Score and Likelihood-Ratio Tests Ask Related Questions Differently
The Wald test asks how far the unrestricted estimate lies from the null relative to its standard error.
The score test examines the slope of the likelihood at the null.
The likelihood-ratio test compares maximised fit under null and alternative.
They often agree in large samples and can differ substantially in small or irregular problems.
24. Boundary Parameters Break Ordinary Chi-Squared Rules
Testing whether a variance component equals zero places the null on the boundary because variance cannot be negative.
Standard likelihood-ratio asymptotics may fail.
Mixture distributions, simulation or specialised theory may be required.
Regularity conditions are part of the method, not fine print.
25. Identifiability Means Distinct Parameter Values Produce Distinct Observable Distributions
If two parameter combinations generate exactly the same data distribution, the likelihood cannot distinguish them no matter how much data are collected.
The model is non-identifiable.
Optimisation problems in such models often have ridges or multiple equivalent maxima.
26. Weak Identification Produces Broad or Flat Likelihoods
The parameter may technically be identifiable but only weakly informed by the observed data.
The likelihood then changes little across a large region.
A point estimate can still exist, but treating the maximum as precise hides the weak information state.
27. Mixture Models Can Have Multiple Local Maxima
A mixture of two normal distributions must estimate component means, variances and weights.
The likelihood can contain several peaks and label-switching symmetries.
An optimiser can stop at a local maximum rather than the global one.
Multiple starting values and diagnostic inspection matter.
28. The EM Algorithm Handles Latent or Missing Components
Expectation-Maximisation alternates between estimating expected sufficient information for latent variables and maximising the resulting expected complete-data log-likelihood.
It is widely used in mixture models and missing-data problems.
EM increases likelihood each iteration under standard conditions but can converge slowly and to local maxima.
29. Censoring Changes the Likelihood Contribution
If a component survives past 1,000 hours but its exact failure time is unknown, the observation does not contribute an ordinary density at one time.
It contributes the probability of surviving beyond the censoring time.
NIST highlights maximum likelihood’s ability to handle censored reliability data naturally.
30. Truncation Is Different From Censoring
With censoring, we know a unit exists but only partially observe its outcome.
With truncation, units outside a range may never enter the dataset.
The likelihood must condition on being observed under the truncation rule.
Selection belongs inside the probability model.
31. Missing Data Can Be Handled by Integrating Over Unobserved Values
Likelihood methods can sum or integrate over missing values rather than fill each one with a deterministic guess.
Under assumptions such as MAR and correct model specification, full-information likelihood can use incomplete cases efficiently.
32. Latent-Variable Models Create Likelihoods by Integrating Hidden States
Factor analysis, hidden Markov models and random-effects models contain unobserved variables.
The observed-data likelihood averages over those latent states according to the model.
Hidden structure becomes inferentially usable only because the probability model specifies how hidden and observed variables connect.
33. Conditional Likelihood Can Remove Nuisance Parameters Structurally
Some models allow conditioning on sufficient statistics that eliminate nuisance parameters from the likelihood.
Conditional logistic regression is a classic example.
This is different from profiling, which optimises nuisance parameters rather than conditioning them away.
34. Partial Likelihood Uses Only the Likelihood Information Needed for Selected Parameters
The Cox proportional-hazards model estimates regression coefficients without specifying the baseline hazard parametrically.
Its partial likelihood extracts information about relative hazards from the ordering of event times and risk sets.
Not every useful likelihood-like object is a full joint likelihood for all observed data.
35. Quasi-Likelihood Uses Mean–Variance Structure Without a Full Distribution
Sometimes researchers trust the conditional mean and variance relation more than a complete probability distribution.
Quasi-likelihood and estimating-equation methods can estimate parameters under weaker distributional commitments.
The price is that some full likelihood tools, such as ordinary likelihood ratios, are no longer available in the same way.
36. Composite Likelihood Approximates Complex Joint Models
When the full likelihood is computationally impossible, analysts may multiply lower-dimensional marginal or conditional likelihood components.
Composite likelihood can estimate useful parameters but requires adjusted uncertainty because component contributions are not a true independent joint likelihood.
37. Penalised Likelihood Adds a Complexity Cost
Maximise log-likelihood minus a penalty.
Ridge penalties shrink coefficients smoothly.
Lasso penalties can set coefficients exactly to zero.
The result no longer maximises pure sample likelihood; it balances fit against a structural preference for smaller or sparser parameter values.
38. Penalised Likelihood Has a Bayesian Interpretation Under Certain Priors
Ridge regression corresponds to a Gaussian prior on coefficients in a maximum-a-posteriori interpretation.
Lasso corresponds to a Laplace prior.
The same optimisation can therefore be described as penalised likelihood or posterior mode estimation under a chosen prior.
39. The Likelihood Principle Focuses on the Observed Likelihood Function
Under the likelihood principle, two experiments producing proportional likelihood functions contain the same evidence about θ.
This differs from some frequentist procedures whose p-values and stopping-rule properties can depend on unobserved possible data.
The philosophical distinction explains why inferential frameworks can use the same likelihood differently.
40. Stopping Rules Matter Differently Across Inferential Frameworks
In ordinary fixed-sample frequentist testing, repeatedly checking p-values without adjustment alters Type I error.
A pure likelihood ratio for the observed data may be unchanged by some stopping-rule descriptions if the final likelihood is proportional.
Bayesian updating also has its own conditions.
“Stopping does not matter” is therefore framework-specific, not a universal statistical rule.
41. AIC Uses Maximised Likelihood With a Complexity Penalty for Prediction
Akaike’s Information Criterion is commonly written:
AIC = −2 log L̂ + 2k
The first term rewards fit.
The second penalises parameter count.
AIC targets expected predictive information loss under assumptions; it is not a probability that one model is true.
42. BIC Uses a Different Penalty and a Different Asymptotic Motivation
Bayesian Information Criterion penalises complexity roughly by k log n.
Its motivation relates to large-sample approximations to marginal likelihood under particular priors and model-selection assumptions.
AIC and BIC can select different models because they optimise different objectives.
43. A Larger Maximised Likelihood Does Not Automatically Mean a Better Model
Add enough parameters and in-sample likelihood usually improves.
A saturated model can fit the observed data almost perfectly.
Generalisation can become worse.
Model comparison needs complexity control, validation and scientific interpretability.
44. Non-Nested Likelihoods Need Careful Comparison
An ordinary likelihood-ratio test assumes nested models under standard use.
Comparing two unrelated non-nested models requires different theory, information criteria, predictive validation or specialised tests.
“Higher likelihood wins” is incomplete when model dimensions differ.
45. Absolute Likelihood Values Are Usually Not Interpretable Across Arbitrary Data Representations
For continuous data, likelihood uses densities, which can exceed one and depend on units.
A single raw likelihood number therefore has little standalone meaning.
Relative likelihoods within the same model and data representation are the meaningful object.
46. Likelihood Ratios Can Be Compelling Without Providing Posterior Probabilities
A parameter value may be 100 times more supported by likelihood than another.
That is strong relative evidence under the model.
But posterior odds also depend on prior odds.
Likelihood tells us what the data contribute; Bayesian inference adds what was believed before the data.
47. Likelihood Can Be Extremely Sharp Around a Biased Scientific Target
A million-person observational dataset estimates a confounded regression coefficient.
The likelihood is extraordinarily narrow.
The parameter is precisely estimated inside the statistical model.
The causal interpretation can still be wrong.
Likelihood precision is not causal validity.
48. Likelihood Does Not Repair Invalid Measurement
A model can estimate a latent engagement score with exquisite likelihood precision.
If engagement is not the construct the decision requires, the inference remains misaligned.
Likelihood works inside the representation given to it.
It does not certify what the variables mean.
49. Likelihood in Machine Learning Is Often Called Negative Log-Loss
Classification systems trained with cross-entropy are often maximising Bernoulli or categorical likelihood, equivalently minimising negative log-likelihood.
This connects classical statistical estimation to modern neural-network training.
The optimiser may be new.
The probabilistic scoring principle is old.
50. Proper Scoring Rules Reward Calibrated Probabilistic Predictions
Log-loss is a strictly proper scoring rule: in expectation, a forecaster minimises expected log-loss by reporting their true predictive distribution.
This gives likelihood-based prediction a decision-theoretic interpretation.
Good probabilistic forecasts should be sharp when justified and calibrated against outcomes.
51. Maximum Likelihood Can Overfit High-Dimensional Models
With many parameters relative to observations, the model can chase noise.
Some logistic regressions even experience complete separation, where coefficients diverge toward infinity while the likelihood approaches its supremum.
Penalisation, priors or exact methods can stabilise these cases.
52. Separation Shows That “Maximum” May Live at Infinity
If one predictor perfectly separates successes from failures, ordinary logistic MLEs may not be finite.
Software can print enormous coefficients and standard errors or fail to converge.
The issue is geometric information in the data, not merely a software bug.
53. Convergence Messages Are Scientific Warnings
An optimiser that fails to converge may indicate poor scaling, multiple modes, weak identification, a boundary solution or an impossible model.
Suppressing the warning and reporting the coefficient table does not solve the inferential problem.
Numerical diagnostics are part of model diagnostics.
54. Likelihood and Bayesian Inference Fit Together Exactly at the Data Interface
Bayes’ theorem combines:
posterior ∝ prior × likelihood
The likelihood carries the information contributed by the observed data under the model.
The next canonical owner is How Bayesian Inference Works.
55. Likelihood and P-Values Are Not the Same Evidence Measure
A p-value asks how extreme the observed statistic is under a null reference distribution.
A likelihood ratio compares support for parameter values or models given the observed data.
The same dataset can yield a small p-value and a likelihood surface whose scientifically meaningful interpretation remains uncertain.
56. Likelihood and Confidence Intervals Connect Through Asymptotic Theory
Wald intervals use local normal approximations around the MLE.
Profile-likelihood intervals use relative likelihood shape more directly.
Both sit inside statistical inference but make different approximation choices.
57. The Hostile Test: Highest Likelihood Model Is Grossly Overfit
A model with 100 parameters fits 100 observations almost perfectly.
Its maximised likelihood beats a ten-parameter model.
On new data, the complex model collapses.
In-sample likelihood rewards fit; generalisation requires a complexity or validation layer.
58. The Second Hostile Test: Narrow Likelihood, Wrong Causal Claim
A huge observational study finds a precise association between tutoring attendance and scores.
The likelihood sharply excludes zero.
Motivation remains unmeasured.
The statistical parameter is precise.
The causal effect is not identified.
59. The Third Hostile Test: Local Maximum Mistaken for the Maximum
A mixture model is fit once from one starting point.
The optimiser converges.
Another starting point finds a much higher log-likelihood.
“Converged” meant the algorithm stopped at a stationary point, not that the global solution was established.
60. The Fourth Hostile Test: Likelihood Interpreted as Posterior Probability
L(θ₁) is nine times L(θ₂).
The report says θ₁ has a 90% probability of being true.
That conclusion does not follow.
Posterior probability also requires the prior distribution and normalisation across parameter space.
61. The Fifth Hostile Test: Perfect Fit to an Invalid Measurement
A model predicts daily app opens almost perfectly.
The research claim is durable learning.
The likelihood is excellent.
The scientific representation is wrong.
Statistical fit cannot repair construct validity.
62. Primary School: Likelihood Begins as “Which Rule Makes What I Saw Less Surprising?”
A bag contains mostly red or mostly blue marbles.
A child draws eight red and two blue.
Which bag composition better fits what was observed?
The child need not calculate likelihood yet.
After seeing evidence, compare which possible rules would make that evidence more expected.
63. Secondary School: Hold the Data Still and Move the Model
Students can contrast probability and likelihood explicitly.
Probability fixes the model and imagines different data.
Likelihood fixes the observed data and compares different model parameters.
The same formula becomes a different question.
64. JC and University: Likelihood Becomes the Geometry of Parameter Support
At higher levels, learners should reconstruct:
- probability model;
- observed data;
- likelihood;
- log-likelihood;
- score;
- information;
- MLE;
- nuisance parameters;
- profile likelihood;
- likelihood ratios;
- regularity conditions;
- identifiability;
- model checking.
65. Where Likelihood Fits in the eduKateSG “How Works” Landscape
- How Probability Works — probability distributions before the data are fixed.
- How Statistical Inference Works — the umbrella estimation and uncertainty system.
- How Regression Works — many fitted models whose coefficients can be estimated by likelihood.
- How P-Values Work — a different null-reference evidence mechanism.
- How Confidence Intervals Work — uncertainty around parameter estimates.
- How Missing Data Works — full-information likelihood and latent missing values.
Likelihood owns one precise canonical job: measure the relative support that observed data give to candidate parameter values within a specified probability model, and provide the fitting geometry from which maximum-likelihood estimates and many inferential tools are built.
66. What This Article Does Not Claim
- Likelihood is not automatically a probability distribution over parameters.
- The MLE is not the probability-most-likely “truth” outside the assumed model.
- A higher in-sample likelihood does not automatically mean better generalisation.
- Likelihood precision does not remove confounding or measurement bias.
- Regular large-sample properties need regularity conditions.
- Boundary, separation and non-identification problems can invalidate standard approximations.
- Local optimiser convergence does not guarantee the global maximum.
- A likelihood ratio is not automatically a posterior odds ratio.
- AIC and BIC optimise different criteria and are not probabilities that models are true.
- Statistical fit remains subordinate to scientific model validity.
67. A Compact Likelihood Audit
- What probability model defines the likelihood?
- Which data are held fixed?
- Which parameters vary?
- Are observations conditionally independent?
- What is the log-likelihood?
- Is the maximum finite and unique?
- Did the optimiser converge reliably from multiple starting values?
- Are parameters identifiable?
- Is the likelihood sharply or weakly concentrated?
- What nuisance parameters exist?
- Would profile likelihood be useful?
- Are standard-error approximations supported by sample size and regularity?
- Are any parameters on boundaries?
- Is censoring or truncation represented correctly?
- How are missing or latent variables integrated?
- Is penalisation being used?
- What objective does the penalty encode?
- Are models nested if a likelihood-ratio test is used?
- Is model complexity accounted for?
- Has predictive performance been validated?
- Is likelihood being confused with posterior probability?
- Does the statistical model represent the scientific question?
68. Frequently Asked Questions
What is likelihood?
Likelihood is a function of model parameters for fixed observed data. It describes how strongly different parameter values support the observed dataset under the specified probability model.
What is maximum likelihood estimation?
Maximum likelihood estimation chooses the parameter values that maximise the likelihood of the observed data under the model.
Is likelihood the probability that a parameter is true?
No. Likelihood is not generally normalised over parameter values. Bayesian inference combines likelihood with a prior to obtain a posterior probability distribution over parameters.
Why use log-likelihood?
Log-likelihood converts products of observation probabilities into sums, improves numerical stability and makes differentiation and optimisation easier while preserving the location of the maximum.
69. Authoritative Research Corridor
- NIST/SEMATECH — Maximum Likelihood Estimation
- NIST/SEMATECH — Maximum Likelihood
- NIST/SEMATECH e-Handbook of Statistical Methods
- eduKateSG — How Statistical Inference Works
- eduKateSG — How Regression Works
Final Thought: Likelihood Turns Evidence Into a Landscape
A point estimate gives us a summit.
Likelihood gives us the mountain around it.
Is the peak sharp?
Is the ridge broad?
Are there several peaks?
Does the landscape disappear into a boundary?
Those shapes tell us how the observed data distinguish possibilities inside the model.
They still do not tell us whether the model itself is the right map of the world.
Likelihood tells us which parameter values the observed evidence supports inside a model. Scientific judgement begins again when we ask whether that model deserves to be the world in which support is measured.