Machine learning fundamentalsMiddleLesson 1155 min read

Measuring a classifier: precision and recall

Accuracy can look great while the model misses every case that matters. The confusion matrix, precision and recall show what really happened.

✦ Ask the tutor 🧪 Lab · 25 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Measuring a classifier: precision and recall

Accuracy can look great while the model misses every case that matters. The confusion matrix, precision and recall show what really happened.

1/3

In simple words

Imagine a smoke alarm that never rings. In a year with no fire, it is "right" every single day. But the one day there is a fire, it stays quiet. Being right most days is not enough; it must catch the rare day that matters. Precision and recall help us check this.

Suppose 2 in every 100 transactions are fraud. A "model" that always says "not fraud" is right 98% of the time. Its accuracy is 98%, and it catches zero fraud. When one group is rare, accuracy hides the mistakes that matter.

The confusion matrix

For two groups, every prediction lands in one of four boxes. We call the rare, important group "positive".

	Predicted: negative	Predicted: positive
Actually negative	True negative (TN): correctly left alone	False positive (FP): a false alarm
Actually positive	False negative (FN): a missed case	True positive (TP): correctly caught

Precision and recall

Precision = TP / (TP + FP). Of all the alarms raised, how many were real? Low precision means many false alarms.
Recall = TP / (TP + FN). Of all the real cases, how many did we catch? Low recall means many missed cases.
F1 combines the two into one number (their harmonic mean). It is only high when both are high.

Python: count the boxes by hand and with scikit-learnpython

from sklearn.metrics import confusion_matrix, precision_score, recall_score, accuracy_score

# 1 = positive (the rare case we care about), 0 = negative. Made-up labels.
actual    = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]
predicted = [0, 0, 0, 0, 0, 1, 0, 1, 1, 0]

tn, fp, fn, tp = confusion_matrix(actual, predicted).ravel()
print("TN", tn, " FP", fp, " FN", fn, " TP", tp)
print("accuracy ", accuracy_score(actual, predicted))
print("precision", tp / (tp + fp), "=", precision_score(actual, predicted))
print("recall   ", tp / (tp + fn), "=", recall_score(actual, predicted))

Which one matters more?

Situation	Worse mistake	Lean towards
Screening for a serious disease, followed by a proper test	Missing a sick person (FN)	Higher recall
Flagging bank transactions for a quick check	Missing fraud (FN), within a limit of checks staff can handle	Recall, with precision kept workable
Spam filter	A real email hidden in spam (FP)	Higher precision
Auto-blocking an account with no human review	Blocking an innocent person (FP)	Higher precision, or add human review

You usually trade one for the other by moving the threshold. Lower threshold: more alarms, higher recall, lower precision. Higher threshold: fewer alarms, higher precision, lower recall. The right point depends on the real cost of each mistake, which is a decision for people, not the model.

Watch out

A model used to screen for disease should support doctors, not replace them. The numbers here are about teaching measurement, not medical practice.

Other habits for rare classes

Split with stratify=y so the rare class appears in every set.
Always report the confusion matrix next to any single number.
Compare against the "always say negative" baseline.
Some models accept class_weight="balanced", which makes mistakes on the rare class count more in training.

Try this. In the code, change the last prediction to 1. Which boxes change, and what happens to precision and recall?

Watch and read more

scikit-learn: Metrics and scoringscikit-learn · reading

Lab

A made-up fraud dataset where a do-nothing model looks 98% accurate, a real model measured with precision and recall, and the effect of a lower threshold.

~25 min

The problem

The starter generates made-up transactions with a fixed seed. About 2% are fraud. The numbers are invented for practice and do not describe any real bank.

Run it. It trains a scaled logistic regression and prints accuracy.
For the do-nothing model that always predicts 0, set dumb_acc and dumb_recall on the test rows.
For model, set lr_precision and lr_recall on the test rows.
Make low_pred: predict 1 when the model's fraud probability is at least 0.2. Set low_precision and low_recall.
Print the confusion matrix for both thresholds. In two sentences, say which threshold you would pick if each flagged transaction gets a quick human check, and why.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix

# Made-up transactions. Invented numbers, fixed seed.
rng = np.random.default_rng(7)
n = 4000
is_fraud = (rng.random(n) < 0.02).astype(int)
amount = rng.lognormal(6, 1, n) * np.where(is_fraud, 3, 1)
night = (rng.random(n) < np.where(is_fraud, 0.6, 0.1)).astype(int)
new_device = (rng.random(n) < np.where(is_fraud, 0.5, 0.05)).astype(int)
km_away = rng.exponential(np.where(is_fraud, 300, 20))
X = np.column_stack([np.log(amount), night, new_device, np.log1p(km_away)])
y = is_fraud

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
pred = model.predict(X_test)
print("Fraud cases in test set:", y_test.sum(), "of", len(y_test))
print("Model accuracy:", round(accuracy_score(y_test, pred), 4))

# TODO 1: the do-nothing model predicts 0 for every test row
dumb_acc, dumb_recall = None, None

# TODO 2: precision and recall of model at the default threshold
lr_precision, lr_recall = None, None

# TODO 3: predict 1 when the fraud probability is at least 0.2
low_pred = None
low_precision, low_recall = None, None

print("Do-nothing:", dumb_acc, dumb_recall)
print("Default threshold:", lr_precision, lr_recall)
print("Threshold 0.2:", low_precision, low_recall)

Output

No output. Add print(...) to see values.

You are done when

The do-nothing model shows high accuracy and zero recall
The model's precision and recall are computed on the test rows
The 0.2 threshold raises recall and changes precision, visible in the two confusion matrices
The written choice links the threshold to the cost of a human check versus a missed fraud

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why is choosing the 0.2 threshold by looking at these test results not fully honest, and what would you do instead?Reveal

Any threshold chosen by looking at the test results is tuned to that test set, so the reported numbers become optimistic. Choose the threshold on a validation set, or with cross-validated probabilities on the training data, then apply it once to the test set.

Q2The test set has only a few dozen fraud cases. What does that mean for the recall number?Reveal

Each fraud case changes recall by several percentage points, so the number is rough. Report the counts (for example 20 of 24 caught), not just the percentage, and be careful about small differences between models. More fraud examples, or repeated splits, give steadier numbers.

Please sign in to continue.

Questions people ask

Which class should be "positive"?

By habit, the rare class you care about catching: fraud, disease, defect. Precision and recall are then about that class. scikit-learn treats label 1 as positive by default.

Is there a single best measure?

No. Choose the measure that matches the cost of each mistake in your situation, and always show the confusion matrix so others can see the full picture.

What is a precision-recall curve?

A chart of precision against recall at every possible threshold. It shows the whole trade-off at once, and helps you pick a threshold. scikit-learn has precision_recall_curve.

Can I choose the threshold on the test set?

No, that is a choice like any other. Choose it with validation data or cross-validation, then report the test result once.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.