Machine learning fundamentalsMiddleLesson 1124 min read

Nearest neighbours

To label a new example, find the most similar examples you already know and let them vote. Simple, strong, and very sensitive to scale.

✦ Ask the tutor 🧪 Lab · 15 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Nearest neighbours

To label a new example, find the most similar examples you already know and let them vote. Simple, strong, and very sensitive to scale.

1/5

In simple words

A new student joins your class. You want to guess which cricket team she supports. You look at the five students who live closest to her house, and three of them support the same team. So you guess that team. That is nearest neighbours: look at who is closest, and go with the majority.

The k-nearest neighbours method (k-NN) does almost no work during training: it just stores the examples. To predict a new row, it measures the distance to every stored example, takes the k closest ones, and lets them vote.

Python: nearest neighbours by handpython

import numpy as np

# Made-up fruits: (weight in grams, sweetness 1-10) and their kind
points = np.array([[120, 7], [130, 8], [115, 6], [300, 3], [320, 2], [290, 4]])
kinds  = np.array(["guava", "guava", "guava", "papaya", "papaya", "papaya"])

new = np.array([140, 6])
dist = np.sqrt(((points - new) ** 2).sum(axis=1))   # straight-line distance
order = np.argsort(dist)

k = 3
print("Distances:", dist.round(1))
print("The", k, "nearest:", kinds[order[:k]])
values, counts = np.unique(kinds[order[:k]], return_counts=True)
print("Vote:", values[np.argmax(counts)])

Choosing k

k	What happens
1	Copies the single closest example. Very sensitive to noise and odd examples.
Medium (for example 5 to 15)	Smooths out noise. Often a good range.
Very large	Ignores local detail and drifts toward the most common label overall.

Choose k by testing several values on data that is not the test set. The lesson on train, validation and test sets shows the right way.

The big trap: scale

Look at the fruit code. Weight is in the hundreds; sweetness is between 1 and 10. The distance is almost all weight. Sweetness barely counts. If one feature is measured in large units, it decides who the "neighbours" are, even if it is the less useful feature.

Danger

For k-NN, always scale your features, for example with StandardScaler in a pipeline. Without scaling, the model may quietly ignore most of your features. In this lesson's lab you will see how big the difference can be.

Where k-NN fits

Weak points

Slow to predict when there are very many examples, since it compares with all of them
Struggles when there are very many features
Needs scaling and a sensible distance

Strong points

Very simple to explain: "these similar cases had this label"
No training time
Adding new examples is instant

Note

The same idea powers similarity search in modern AI. Search systems turn text into vectors (embeddings) and find the nearest ones. The lesson Embeddings and similarity search covers it.

Try this. In the fruit code, divide weight by 100 before measuring distance (points / [100, 1] and new / [100, 1]). Do the nearest neighbours change?

Watch and read more

scikit-learn: Nearest neighborsscikit-learn · reading

Lab

The same 5-nearest-neighbour model on the wine data, with and without scaling, and the gap between them.

~15 min

The problem

Run the starter. It fits k-NN with k = 5 on the raw wine features and prints the test accuracy.
Build scaled_model: a pipeline of StandardScaler then KNeighborsClassifier(n_neighbors=5). Fit it on the training rows.
Set scaled_acc to its test accuracy.
Print both accuracies. Then look at X_train.max(axis=0): which feature has by far the largest values, and why does that hurt the unscaled model?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

Output

No output. Add print(...) to see values.

You are done when

scaled_model is a pipeline with StandardScaler and 5 neighbours
It is fitted on the training rows only
The scaled accuracy is clearly higher than the raw one
The learner names the large-valued feature (proline) and explains that it dominates the distance

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why must the scaler be fitted on the training rows only, and how does the pipeline make sure of that?Reveal

The scaler learns each feature's average and spread. If it saw the test rows, some test information would shape the training, and the test score would be a bit too kind. A pipeline's fit only sees the data you pass it (the training rows) and then reuses those learned numbers when you call score or predict on new rows.

Q2Is scaling always the right fix? When might it hurt?Reveal

Scaling gives every feature an equal say. If some features are pure noise, giving them an equal say can hurt, because they now affect the distance as much as useful ones. Then you may need to drop noisy features, or use a model that picks features itself, such as a forest.

Please sign in to continue.

Questions people ask

What distance does scikit-learn use?

By default, the ordinary straight-line (Euclidean) distance. Other distances are available through the metric setting.

What if the vote is a tie?

Using an odd k helps for two groups. scikit-learn has a fixed rule for breaking ties, and you can also weight closer neighbours more with weights="distance".

Is k-NN used in real products?

The idea is everywhere: "customers similar to you also bought", and vector search in AI apps. Large systems use fast approximate search instead of comparing with every example.

Can k-NN do regression?

Yes. KNeighborsRegressor predicts the average label of the k nearest examples.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.