Did you know? A computer does not always need to learn a complicated equation or a deep neural network to classify a new example. It can sometimes look for familiar examples that are numerically close and use their labels to make a prediction.
This is the idea behind k-nearest neighbours (kNN), an important machine-learning method. The letter k tells us how many nearby training examples the classifier considers.
Quick answer: represent each example with useful numerical features → measure distances from the new example → choose the k closest training examples → combine their labels or values → check the prediction on unseen cases. This can be surprisingly effective, but everything depends on whether the chosen distance really captures similarity for the task.
Imagine a fictional flower collection. A new flower has a certain petal width and petal length. If its measurements resemble several verified examples of one species, a kNN classifier may predict that species. It has not proved the flower’s biological identity; it has found a plausible classification based on a particular measurement space.
k-Nearest Neighbours in 30 seconds
- Training example: a past observation with relevant features and, for supervised classification, a known label.
- Feature vector: numerical coordinates representing one example.
- Distance metric: a rule for measuring the difference between examples.
- Nearest neighbour: an example with a relatively small calculated distance.
- k: the number of neighbours used for a prediction.
- Majority vote: selecting the class represented by most of the chosen neighbours.
- Distance weighting: giving closer examples more influence than farther ones.
For numerical predictions rather than categories, kNN regression can average or otherwise combine neighbours’ target values. The method is therefore useful for understanding both classification and regression.
A fictional flower classifier
Imagine two features: petal width and petal length, measured consistently. Our verified training examples include three flowers of Species A and several of Species B.
A new flower arrives. Instead of reading a textbook paragraph about each species, our simple model asks which training flower measurements lie closest to the new point in two-dimensional space.
If the three closest verified flowers include two of Species A and one of Species B, an ordinary k = 3 majority-vote classifier chooses Species A. The prediction could still be wrong because measurements overlap or the flower belongs to an unrepresented species.
Step 1: Decide what counts as a useful feature
Choosing the right features matters at least as much as choosing the algorithm. For flowers, petal and sepal measurements might be useful. For text passages, raw document length may be a poor measure of whether two questions ask the same thing.
Suppose an application is meant to classify library books. It should not treat the book’s random database row number as though a nearby number means a nearby topic.
A model can calculate distance perfectly in a badly designed feature space. That calculation is mathematically correct but practically useless.
Step 2: Put the examples into a numerical space
Each item is represented by one or more numbers. In our fictional flower case, one point could be (2, 2) and another (4, 2).
The numbers need consistent meanings and scales. The first coordinate might stand for petal width in a defined unit and the second for petal length in the same or another properly recorded unit.
If a field has missing or incorrect measurements, the comparison can become misleading. Appropriate cleaning and imputation, when justified, belong to the modelling workflow.
Step 3: Measure distance with Euclidean geometry
A common distance metric is Euclidean distance, familiar from the Pythagorean theorem.
For points (x₁, y₁) and (x₂, y₂), the distance is the square root of (x₂ − x₁)² + (y₂ − y₁)².
Suppose the new example is (3, 3) and a verified Species A flower is (2, 2). The distance is √[(3 − 2)² + (3 − 2)²] = √2 ≈ 1.41.
A second Species A example at (4, 2) is also √2 away. A Species B example at (3, 5) is 2 units away.
These are invented coordinates. The purpose is to see the process, not to draw conclusions about actual flower species.
Step 4: Choose the closest k examples
For k = 3, our three examples at distances 1.41, 1.41 and 2 form the selected neighbourhood. Two are Species A; one is Species B.
A simple majority vote predicts Species A. Notice that the algorithm did not infer a biological law. It looked at the labels associated with the three closest points under our measurement choice.
If the new observation were farther from every known example, the same procedure might still select three neighbours and return a confident-looking category. That is a reason to check whether an input is unfamiliar instead of forcing every observation into a known class.
Step 5: Why k = 1 can be fragile
With k = 1, the model copies the label of the single closest training example. This can work well when the features are informative and the closest item is trustworthy.
But one incorrectly labelled or noisy training example can change a prediction. If a flower was mistakenly labelled Species A, a new flower near it may receive the wrong category.
Choosing more neighbours can reduce the effect of one isolated bad example. The trade-off is that a large neighbourhood may include many examples from a different nearby region, smoothing away useful local boundaries.
Step 6: Why increasing k is not always better
The official scikit-learn nearest-neighbours guide explains that the best k depends on the data. Larger values may reduce sensitivity to noise but can blur the distinction between nearby categories.
Imagine a tiny island of one plant species surrounded by many examples of another. A very large k may overrule that local group simply because the wider dataset contains more of the other species.
Select k using development data and an appropriate evaluation method. The final independent test set should not become a repeated tool for finding the most flattering parameter value.
Step 7: What if neighbours disagree?
For k = 4, an example could have two neighbours from Species A and two from Species B. Under equal voting, this creates a tie.
A software implementation needs an explicit way to resolve ties or report uncertainty. Its exact behaviour can depend on distance order, labels and library rules.
Choosing an odd k can reduce some simple binary ties, but it does not solve every ambiguity, especially with more than two possible classes.
A reliable application should explain that the result may depend on how a difficult tie is handled rather than presenting every classification as inevitable.
Step 8: Weight closer neighbours more strongly
Not every kNN system gives each neighbour exactly one vote. A distance-weighted classifier can give greater influence to nearby examples.
Imagine one Species A example is extremely close, while two Species B examples are only just inside the selected neighbourhood but considerably farther away. Uniform voting would give B the majority, but distance weighting could favour A.
The correct behaviour depends on the chosen weighting formula and task. The scikit-learn KNeighborsClassifier reference documents both uniform and inverse-distance weighting options.
The existence of weights does not turn the output into a verified probability of truth. They are part of a classification procedure.
Step 9: Feature scaling can change the answer
Suppose one feature measures petal width in millimetres, typically ranging from 1 to 10, while another feature is a count ranging from 0 to 100,000.
Without suitable scaling, the second feature can dominate Euclidean distance simply because its numbers are larger. An apparently close neighbour might be very different on the smaller-scale variable.
Standardisation or another task-appropriate transformation can help. But do not standardise blindly: some physical scales carry meaningful relative importance.
Fit the scaling transformation on training data, then apply it consistently to unseen inputs. Otherwise, test information can leak into model development.
A distance calculation is only meaningful when the coordinate system and feature preparation are meaningful.
Step 10: What about Manhattan distance?
Euclidean distance is not the only metric. Manhattan distance adds the absolute coordinate differences instead of squaring and taking a square root.
Between (2, 2) and (5, 6), Manhattan distance is |5 − 2| + |6 − 2| = 3 + 4 = 7. Euclidean distance is √(3² + 4²) = 5.
The two metrics rank some example pairs differently. The better metric depends on the feature representation and actual task, not on a universal hierarchy.
Text embeddings may use cosine similarity or other measures. A nearest-neighbour system should always specify what its numerical closeness means.
Step 11: Why kNN is called instance-based learning
Some machine-learning models train parameters that represent a general function. kNN often relies directly on a stored set of training examples or a structure built to search them efficiently.
That is why the method is sometimes called instance-based learning or lazy learning. It usually does less heavy parameter fitting before prediction than a deep neural-network training procedure.
The trade-off is that prediction may require searching through many reference points. Storage and search efficiency become important as the dataset grows.
Step 12: Can kNN predict a number?
Yes. In kNN regression, the neighbours have numeric target values rather than category labels.
Imagine three fictitious similar delivery cases with observed times of 10, 12 and 14 minutes. A uniform average of those three targets is (10 + 12 + 14) ÷ 3 = 12 minutes.
That is a simple kNN regression prediction for an appropriately defined setup, not a real travel-time estimate. The distance metric, relevance of the examples and future conditions determine whether the average is useful.
Step 13: Why high dimensions are difficult
Suppose each example has a thousand numerical features instead of two. In many high-dimensional spaces, distances between points become less informative under naive metrics, making it harder to identify a truly meaningful local neighbourhood.
This broad difficulty is often called part of the curse of dimensionality. The problem is not that a computer cannot subtract a thousand numbers; it is that distance may no longer align well with the intended similarities.
Feature selection, appropriate representations and careful evaluation become increasingly important. Adding more irrelevant dimensions is not automatically helpful.
Step 14: Speeding up neighbour search
A naive exact search could compare one query with every stored training record. For large datasets, that may require substantial computation.
Spatial structures such as KD trees or ball trees can accelerate suitable nearest-neighbour searches, depending on dimensionality and data geometry. Other applications use approximate nearest-neighbour indexing for very large collections.
Approximate search may trade a small amount of neighbour accuracy for speed. That can be useful for retrieval but should be evaluated against the needs of the application.
Step 15: kNN versus k-means clustering
The names sound related but answer different questions.
kNN commonly uses labelled neighbours to predict a category or numeric target for a new point. k-means is an unsupervised clustering method that organises examples around learned cluster centres.
The meaning of k differs: in kNN it commonly means the number of neighbours; in k-means it means the number of clusters. Do not confuse them merely because both use distances.
See How AI Finds Patterns Without Labels for the grouping approach.
Step 16: kNN versus embedding search
A semantic search application may use learned embeddings so sentences with related meanings become nearby vectors. A nearest-neighbour index can then retrieve passages close to a user’s query.
This is related to kNN mathematically, but retrieving similar text is not automatically supervised class prediction. The embedding model and index determine what “nearby” represents.
The embedding guide and search reranking guide explain the wider information-retrieval pipeline.
Step 17: Measure errors on a meaningful test
For classification, evaluate precision, recall and overall accuracy where appropriate. Examine which categories and operating conditions produce mistakes.
If one class is rare, the classifier may achieve high overall accuracy while repeatedly missing it. A few examples of underrepresented categories deserve special attention.
For regression, measure numerical errors using the task’s natural units. In either case, a beautiful two-dimensional plot is not enough to demonstrate trustworthy behaviour on new data.
Step 18: Data privacy matters when storing examples
Because instance-based methods can retain training records, a poorly designed service may expose sensitive information if it returns raw neighbours or personal attributes.
Use authorised data, access controls, retention limits and appropriate de-identification procedures. A nearest-neighbour search result should not automatically reveal another person’s private record.
For an educational demonstration, synthetic numbers and fictional categories are sufficient. There is no reason to upload real student identities merely to illustrate Euclidean distance.
Try it: classify a point on graph paper
Plot Species A points at (2,2) and (4,2), plus a Species B point at (3,5). Mark the new point at (3,3).
Calculate the three distances: √2, √2 and 2. Under k = 3, the majority predicts A. Now add a new verified Species B point at (3,3.1) and see how k = 1 changes.
This illustrates why a prediction depends on the selected neighbourhood and the quality of the examples.
Five mistakes worth avoiding
- “The nearest point is always the correct category.” Noise, overlap and mislabels remain possible.
- “A large k must improve every result.” Large neighbourhoods can erase useful local patterns.
- “Distance automatically equals meaning.” Features and scaling define the geometry.
- “kNN and k-means do the same thing.” Supervised prediction differs from clustering.
- “Stored neighbours have no privacy implications.” Reference examples may contain sensitive data.
Frequently asked questions
What does k stand for in k-nearest neighbours?
It is the number of nearby training examples considered for the prediction.
Does kNN require labelled data?
For supervised kNN classification, yes. Unsupervised nearest-neighbour search can operate without category labels.
Is kNN a neural network?
No. It is an instance-based machine-learning method and need not use neural-network training.
Can kNN predict continuous numbers?
Yes. kNN regression combines target values from nearby examples.
Is the closest example proof of identity?
No. Similar measurements or embeddings do not establish that two records refer to the same real-world person.
Your one-minute takeaway
kNN makes predictions from the labels or values of nearby reference examples. The method is intuitive and often provides a strong baseline, but its reliability depends on meaningful features, appropriate distance metrics, k selection and independent tests. Numerically close does not automatically mean factually identical.
Trusted reading and connected articles
- scikit-learn: Nearest Neighbors — supervised classification, regression and distance metrics.
- scikit-learn: KNeighborsClassifier — k, voting and distance weighting.
- scikit-learn: Preprocessing Data — feature scaling and preparation.
- How AI Finds Patterns Without Labels — contrast with clustering.
- How Embeddings Find Similar Meanings — vector geometry for text.
- How AI Reranks Search Results — what happens after similarity search.
- How AI Finds Duplicate Names and Records — why similarity is not identity.
- How AI Works: Complete Overview — further reading.