Did you know? An AI system predicting whether an event will happen needs a different kind of output from a model predicting a number. A school practice app might want to estimate whether a student will answer the next question correctly. A simple forecasting task might want a yes-or-no answer about whether a parcel will be late. In each case, the useful first result is often a probability, not merely a label.
Quick answer: choose a clearly defined outcome → prepare labelled examples → calculate a weighted score → pass it through a sigmoid function → train against real labels → check probability quality → choose a decision threshold appropriate to the task. This is the central idea of logistic regression, a standard machine-learning method for binary classification.
Although the word regression appears in its name, binary logistic regression is commonly used as a classifier. It can return an estimated probability of a specified class. That estimate is not a guarantee that an event will happen.
Logistic regression in 30 seconds
- Input features: measurable information supplied to the model, such as count, duration or a suitably encoded category.
- Binary target: an outcome coded as 0 or 1, with the positive class explicitly defined.
- Weights: learned coefficients describing how features affect the model’s log-odds score.
- Intercept: a baseline score before feature contributions are added.
- Sigmoid: an S-shaped function mapping the score to a number between 0 and 1.
- Threshold: a separate rule that turns a probability estimate into a yes-or-no decision.
- Evaluation: a check on unfamiliar examples, including calibration and mistakes in each class.
Why ordinary linear regression is not enough
In our guide to linear regression, a straight-line model estimates a continuous quantity. Its predictions can extend below zero or above one. That is acceptable for some numerical tasks but unsuitable as an unconstrained probability output.
Logistic regression still starts with a linear score. The crucial extra step is to transform that score through the sigmoid function so the final value stays between zero and one. A model can still be wrong or poorly calibrated; keeping its output in the right numerical range does not make it truthful.
Step 1: Define exactly what yes means
Imagine a fictional revision app. Its target is 1 if the student answers one selected question correctly without a hint, and 0 otherwise. That target is narrower and more measurable than ‘student success’ or ‘intelligence’.
The app might observe prior performance on that topic, time since last practice and whether prerequisite exercises were completed. It must not quietly substitute sensitive personal attributes or assume a correlation proves why a student learns well.
A classifier is only as meaningful as the event being labelled. If the definition changes halfway through the dataset, the resulting model learns from incompatible examples.
Step 2: Convert information into features
Suppose we use a single numerical feature, x, representing the number of earlier practice items answered correctly in a short review. Real projects may use many features, but one is sufficient to see the mechanism.
Each feature needs a consistent definition. A value of 5 should mean the same thing in training and in real use. Missing values, different time windows and inconsistent coding can turn a mathematically correct formula into a misleading system.
Step 3: Build a weighted score
For one feature, write the score as z = b + wx, where b is the intercept and w is the coefficient. With several features, add one weighted term for each input.
This score is not yet a probability. It is the log-odds of the modelled outcome. Positive scores correspond to probabilities above 0.5; negative scores correspond to probabilities below 0.5, under the basic binary sigmoid formulation.
The sign of a weight shows how a one-unit feature increase changes the log-odds while the other features are held fixed. It does not by itself prove that changing the feature would cause a change in the real outcome.
Step 4: Turn the score into a probability
The sigmoid formula is p = 1 / (1 + exp(−z)). Here exp means the exponential function, and p is the model’s estimated probability of the positive class.
For an easy, deliberately invented example, let b = −1, w = 0.5, and let a fictional learner have x = 4. The weighted score is z = −1 + 0.5 × 4 = 1. The sigmoid produces p ≈ 0.731. The model therefore assigns about 73.1% probability to its defined positive outcome.
That does not mean the student will certainly answer correctly, nor that a real school’s students have been measured. All numbers in this worked example are teaching values, not data from eduKateSG students.
Google’s sigmoid explanation demonstrates the same mathematical conversion and explains why the formula produces a value between zero and one.
Step 5: Learn the weights from labelled examples
Training begins with examples that include both input features and known outcomes. A candidate set of weights produces predicted probabilities; the training procedure then adjusts those weights to make the observed labels more plausible under the chosen model.
A widely used training objective is log loss, also called binary cross-entropy. Correct, confident predictions receive smaller loss than confidently incorrect ones. In one positive example, predicting 0.8 gives loss about 0.223, whereas predicting 0.2 gives loss about 1.609.
Training is not memorising one spreadsheet row at a time. The model searches for a set of coefficients that performs well across the training observations, often while controlling model complexity through regularisation.
Step 6: Regularise to avoid fitting noise
A model can learn accidental quirks of the training data. L1 or L2 regularisation discourages excessively complex weight patterns in different ways. Regularisation settings should be selected through a proper validation procedure.
A small training loss is not enough. The purpose is to perform sensibly on new cases, not to produce perfect numbers for old examples.
Step 7: Separate probability from decision
Suppose the model returns 0.731. If the operational threshold is 0.5, the system labels the example ‘likely yes’. If it is 0.8, the very same example is labelled ‘not above threshold’. The model score did not change; the decision rule did.
A threshold should reflect the consequences of false positives and false negatives, not simply default to 0.5 because that is convenient. In a revision tool, a false prediction may select the wrong next exercise. In consequential settings, additional expert review and safeguards are required.
Step 8: Recognise class imbalance
Imagine 100 examples in which only five belong to the positive class. A lazy system that always predicts ‘no’ achieves 95% accuracy but identifies none of the five positive examples.
This is why useful evaluation may require precision, recall, a confusion matrix, balanced metrics and appropriate threshold analysis. The correct choice depends on the question being answered.
Step 9: Check whether the probabilities are reliable
A probability forecast should be judged differently from a binary label. If a system outputs 0.7 on many genuinely comparable cases, roughly seven in ten should be positive over a suitable large sample for the forecasts to be well calibrated at that level.
A classifier can rank cases usefully yet have poorly calibrated probabilities. Calibration checks and log loss or Brier score can reveal defects that raw accuracy hides.
For model selection and scoring, consult the scikit-learn evaluation guide. Good testing keeps the final test set separate from training and repeated tuning.
Step 10: Prevent hidden data leakage
Do not let future results appear among the features used to predict the past. Do not scale or impute a combined dataset before dividing it into training and evaluation groups if that lets test-set information influence training.
For repeated measurements from one learner or customer, random row splitting can allow near-duplicate observations to appear on both sides of the split. Choose a split that matches how the system will actually be used.
When logistic regression is useful
- Binary forecasts: estimating the chance of a precisely defined event.
- Text features: simple document classification using suitable encoded word patterns.
- Learning tools: testing whether a narrow activity outcome is associated with observable practice features, without reducing a child to a permanent label.
- Baseline comparisons: providing an interpretable reference against which more complex models can be evaluated.
A logistic model may be simpler, faster and easier to audit than an unnecessarily deep model. But it cannot automatically capture arbitrary nonlinear relationships unless the right feature transformations are designed.
Common mistakes and how to repair them
- Mistake: ‘Logistic regression predicts a continuous amount.’ Repair: binary logistic regression predicts a positive-class probability, unlike ordinary linear regression.
- Mistake: ‘73% probability means 73% certainty for this individual.’ Repair: it is an estimated frequency model with sampling and modelling uncertainty.
- Mistake: ‘A high accuracy figure proves success.’ Repair: inspect class imbalance, error costs, calibration and out-of-sample performance.
- Mistake: ‘A positive feature weight proves causation.’ Repair: association and causal influence are different questions.
- Mistake: ‘Choose a threshold first and ignore the probability.’ Repair: evaluate the probability forecast and then justify the operational threshold.
Try it in one minute
- Calculate z when b = −2, w = 0.5 and x = 4. You get z = 0.
- Use sigmoid(0) = 0.5, so the modelled positive-class probability is 50%.
- Change x from 4 to 6. Now z = 1 and the probability becomes about 73.1%.
- Ask which part changed: the feature value and therefore the model score, not the learned coefficient.
- Ask a second question: does a changed mathematical estimate prove any real child improved? No; real-world evidence would be needed.
Frequently asked questions
Is logistic regression an AI algorithm?
Yes. It is a long-established statistical classification method commonly used in machine learning and as a baseline for AI systems.
Why is it called regression if it classifies?
The model uses a regression-style linear combination to estimate log-odds, which the sigmoid transforms into a probability. The resulting target is categorical.
Is a logistic model always more accurate than k-nearest neighbours?
No. Their inductive assumptions differ, and the best choice depends on data quality, feature structure, class balance and honest validation.
What happens if the model predicts 0.5?
That means the model assigns equal estimated probabilities to the two binary classes. A tie-breaking or abstention rule is an operational decision.
Can students understand this without advanced AI programming?
Yes. Start with a weighted sum, see how the sigmoid converts it into a probability, and only then study optimisation and evaluation.
The one-minute takeaway
Logistic regression learns coefficients for a weighted input score, transforms that score into a 0-to-1 estimate and can apply a threshold to classify an outcome. Its real quality depends on trustworthy labels, fair data preparation, held-out testing and well-calibrated probabilities—not just a convincing-looking percentage.
Trusted reading and connected articles
- Google Machine Learning Crash Course: Logistic Regression — probabilities, sigmoid, loss and regularisation
- Google: Calculating Probability With the Sigmoid — worked score-to-probability method
- scikit-learn: Logistic Regression — mathematical model and implementation cautions
- scikit-learn: Model Evaluation — classification metrics and probability scoring
- How AI Uses Linear Regression to Make Predictions — continuous-number predictions and the key distinction
- How AI Classifies Data With k-Nearest Neighbours — a distance-based alternative
- How AI Is Tested Before Release — why held-out evaluation matters
- How AI Works: Complete Overview — the wider explainer index
- Secondary 1 Mathematics Tutor Clementi — learning numerical relationships from first principles