Machine learning fundamentalsMiddleLesson 1135 min read

Overfitting and how to spot it

A model that memorises its training examples looks perfect in practice and fails on new data. Learn to see the gap and to control it.

✦ Ask the tutor 🧪 Lab · 25 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Overfitting and how to spot it

A model that memorises its training examples looks perfect in practice and fails on new data. Learn to see the gap and to control it.

1/3

In simple words

Two students prepare for a maths exam. One memorises the answers to last year's paper, word for word. The other learns how to solve the problems. On last year's paper, both score full marks. On this year's new paper, only the second one does well. The first student "overfitted" to last year's paper.

Overfitting means the model has learned the training examples too closely, including their random noise, so it does worse on new data. Underfitting is the opposite: the model is too simple to catch the real pattern, so it does badly everywhere.

Seeing it with curves

Here we make 12 noisy made-up points from a gentle curve and fit three polynomials: a straight line, a curve with degree 3, and a very wiggly curve with degree 9. Then we check each on 100 new points from the same source.

Python: simple, right, and too flexiblepython

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(3)
def truth(x):
    return np.sin(x)                         # the hidden pattern

x_train = np.sort(rng.uniform(0, 6, 12))
y_train = truth(x_train) + rng.normal(0, 0.25, 12)
x_new = rng.uniform(x_train.min(), x_train.max(), 100)   # new points inside the same range
y_new = truth(x_new) + rng.normal(0, 0.25, 100)

grid = np.linspace(0, 6, 200)
plt.scatter(x_train, y_train, color="black", label="training points")
for degree in [1, 3, 9]:
    coefs = np.polyfit(x_train, y_train, degree)
    train_err = np.mean(np.abs(np.polyval(coefs, x_train) - y_train))
    new_err = np.mean(np.abs(np.polyval(coefs, x_new) - y_new))
    print("degree", degree, " training error", round(train_err, 3), " new-data error", round(new_err, 3))
    plt.plot(grid, np.polyval(coefs, grid), label="degree " + str(degree))
plt.ylim(-2, 2); plt.legend()

Run it. The degree-9 curve has the smallest training error, because it bends to pass near almost every point. But its error on new data is much larger than its training error. The degree-3 curve does well on both. The straight line is poor on both: it underfits.

The tell-tale sign: the gap

Training score	Score on unseen data	Diagnosis
Poor	Poor	Underfitting: model too simple, or features not useful
Good	Good, close to training	A healthy fit
Excellent	Much worse	Overfitting: the model memorised noise

You cannot see overfitting from the training score alone. You must always compare with data the model has not seen.

Ways to reduce overfitting

Use a simpler model: lower polynomial degree, smaller max_depth for trees, larger k for nearest neighbours.
Get more data: with more examples, noise averages out and memorising gets harder.
Regularise: many models have a setting that punishes extreme weights. In scikit-learn's logistic regression it is C (smaller C means stronger regularisation); Ridge is linear regression with it built in.
Average many models: random forests reduce the overfitting of single trees.
Remove useless features: every extra feature is another place to find fake patterns.

Real example

A small shop owner in Pune builds a model to predict daily sales from 40 columns, including the weather, the day, and the colour of the shirt she wore. On last month's data it is nearly perfect. Next month it is poor. With only 30 days of data and 40 columns, the model found "patterns" that were pure chance, like the shirt colour. With fewer, sensible features, the model is less perfect on the past and better on the future.

Note

Large language models face a version of this too: a model can do well on a benchmark because similar questions were in its training data. See Contamination: when the test leaked.

Try this. In the code, change 12 training points to 60. Does the degree-9 curve still do much worse on new data?

Watch and read more

scikit-learn: Underfitting vs. overfitting (example)scikit-learn · reading

Lab

A table of training and validation accuracy for tree depths 1 to 12 on noisy made-up data, and the depth you would choose.

~25 min

The problem

The starter makes a noisy two-group dataset with make_moons (made-up points, fixed seed) and splits off a validation set, used for choosing settings.

For each depth from 1 to 12, fit DecisionTreeClassifier(max_depth=depth, random_state=0) on the training rows.
Store results[depth] = (train_acc, val_acc).
Set best_depth to the depth with the highest validation accuracy (if there is a tie, the smallest such depth).
Set deep_gap to training accuracy minus validation accuracy for depth 12.
Look at the printed table: where does the gap start to grow?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

Output

No output. Add print(...) to see values.

You are done when

results has depths 1 to 12, each with training and validation accuracy
Training accuracy keeps rising with depth while validation accuracy peaks early and then falls
best_depth is chosen with the validation set, not the training set
deep_gap is clearly larger than the gap at the best depth

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Several depths have the same best validation accuracy. Why prefer the smallest?Reveal

When two models do equally well on unseen data, the simpler one is usually safer: it has fewer ways to have memorised noise, and it is easier to explain. This preference for simpler models is an old rule of thumb in science and statistics.

Q2The validation set has only 120 points. How much should you trust a 1-point difference in accuracy?Reveal

Not much. One point here is 1 or 2 more correct predictions out of 120, which can easily happen by chance. Use cross-validation, or a larger validation set, before treating small differences as real.

Please sign in to continue.

Questions people ask

Is a 100% training score always a bad sign?

Not always, but it is a reason to look closely. Some models, like a deep tree, can reach 100% on training data by memorising. What matters is the score on unseen data and the gap between the two.

How big a gap is too big?

There is no fixed number. Compare models: if a simpler model has a similar unseen score with a smaller gap, prefer it. Also consider how noisy the task is.

Can a model overfit to the test set?

Yes, if you try many models and keep the one with the best test score. You slowly tune to that test set. The next lesson shows how a validation set prevents this.

Is underfitting fixed by adding more data?

Usually not. An underfitting model is too simple to use what it has. Try a more flexible model or better features.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.