Machine learning fundamentalsMiddleLesson 1145 min read

Train, validation and test sets

Use one part of the data to learn, one to choose, and one, touched only once at the end, to report. Plus cross-validation for small datasets.

✦ Ask the tutor🧪 Lab · 25 min📝 Lesson test
Lesson in motion

In 60 seconds

Train, validation and test sets

Use one part of the data to learn, one to choose, and one, touched only once at the end, to report. Plus cross-validation for small datasets.

1/4
In simple words
A teacher gives you practice worksheets to learn from. Then a mock exam to see where you stand and what to revise. Then the final exam, with new questions nobody has seen. If the final exam questions leaked, your marks would not show what you really know. The test set is the final exam: keep it sealed.
Each part of the data has one job:
SetJobHow often you look
Training setThe model learns its weights from itAll the time
Validation setYou compare models and choose settings like depth or kMany times
Test setYou report the final score, once, on data untouched by any choiceOnce, at the end
Why not choose settings using the test set? Because every choice you make by looking at the test score fits the model a little to that test set. After many tries, the test score is too optimistic: the same problem as overfitting, one level up.
Python: a three-way splitpython
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)

# First keep 20% aside as the sealed test set
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
# Then split the rest into training and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, random_state=42, stratify=y_rest)

print("train:", len(X_train), " validation:", len(X_val), " test:", len(X_test))

Cross-validation: better use of small data

With little data, a single validation set is small and its score is noisy. k-fold cross-validation splits the training data into k parts (folds). It trains k times, each time holding out a different fold for validation, and averages the scores. Every training row gets used for validation once.
  1. 1

    Split

    Divide the training data into, say, 5 folds.
  2. 2

    Rotate

    Train on 4 folds, score on the 5th. Repeat so each fold is held out once.
  3. 3

    Average

    The mean of the 5 scores is your validation score. The spread shows how steady it is.
  4. 4

    Finish

    Retrain the chosen model on all the training data, then score once on the test set.
In scikit-learn, cross_val_score(model, X_train, y_train, cv=5) does the rotation for you. If the model is a pipeline with a scaler, the scaler is refitted inside each fold, so nothing leaks.

Common ways the test set leaks

  • Preparing all data before splitting: fitting a scaler, filling missing values or picking features using all rows. Split first, or use a pipeline.
  • Duplicates across sets: the same example (or a near copy) in training and test. Remove duplicates before splitting.
  • Time mix-ups: for data that changes over time, such as prices, train on the past and test on the future. A random split lets the model "see the future".
  • Groups split apart: several rows from the same person or the same farm in both sets. The model recognises the person, not the pattern. Keep each group in one set (scikit-learn has GroupKFold).
  • Repeated peeking: "just one more look" at the test score after each change.
Watch out
If you have already used the test set to make choices, say so when you report the number. Honest reporting matters more than a high score. The lesson Benchmarks and their limits shows the same problem at the scale of whole research fields.
Try this. Change random_state=42 in the code to other numbers. The sizes stay the same, but the rows change. Why does that matter for small datasets?

Watch and read more

scikit-learn: Cross-validationscikit-learn · reading

Lab

A k-nearest-neighbours model whose k is chosen by 5-fold cross-validation on the training data, then scored once on a sealed test set.

~25 min

The problem

  1. Run the starter. It seals 20% of the breast cancer data as the test set.
  2. For each k in candidates, compute the mean 5-fold cross-validation accuracy on the training rows only, using a pipeline of StandardScaler and KNeighborsClassifier(n_neighbors=k). Store it in cv_scores[k].
  3. Set best_k to the k with the highest mean score.
  4. Build final_model with that k, fit it on all the training rows, and set final_test_acc to its test accuracy. This is the only time you touch the test set.
  5. Print the table of k and scores, then the final number.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1The CV scores for k = 5, 7 and 9 differ by less than 0.01. What should you make of that?Reveal

The differences are within the noise of cross-validation on about 450 rows. Any of those k values is a fair choice. You could look at the spread of fold scores, prefer a simpler or more stable option, or accept the top one while not claiming it is truly better.

Q2Your final test accuracy is a bit lower than the best CV score. Is something wrong?Reveal

Not necessarily. Picking the best of eight CV scores favours whichever k got a little lucky, so the best CV score tends to be slightly optimistic. The test set also has only 114 rows. A small drop is normal. A large drop would be a reason to check for leakage or differences between the sets.

Please sign in to continue.

Questions people ask

What split sizes should I use?

Common choices are 60/20/20 or 70/15/15 for training, validation and test. With cross-validation, keep about 20% as the test set and cross-validate on the rest. With very large data, smaller shares for validation and test are enough.

How many folds?

5 or 10 are common. More folds use more data for training each time, but take longer.

After choosing the model, should I retrain on training plus validation data?

Often yes: once the settings are fixed, retraining on more data usually helps. Then score once on the test set. Never add the test set to training before you have your final number.

What if my test score is much worse than my validation score?

It may be chance (small test set), or your choices overfitted the validation data, or the test data differs from the rest. Report it honestly and investigate. Do not quietly keep tuning on the test set.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress