Machine learning fundamentalsMiddleLesson 1105 min read

Classification: sorting into groups

Logistic regression predicts which group a row belongs to, and how sure it is. Learn probabilities, thresholds and why scaling features helps.

✦ Ask the tutor🧪 Lab · 20 min📝 Lesson test
Lesson in motion

In 60 seconds

Classification: sorting into groups

Logistic regression predicts which group a row belongs to, and how sure it is. Learn probabilities, thresholds and why scaling features helps.

1/4
In simple words
A weather app does not just say "rain" or "no rain". It says "70% chance of rain". Then you decide: at 70% you carry an umbrella. Logistic regression works the same way. It gives a chance between 0 and 1, and then we pick a cut-off to turn that chance into a yes or no.
In classification, the label is a group, not a number on a scale: spam or not spam, pass or fail, species A, B or C. With two groups it is called binary classification. We usually write the groups as 0 and 1.

How logistic regression works

  1. 1

    Score

    Like linear regression, it multiplies each feature by a weight and adds them up. The score can be any number.
  2. 2

    Squash

    The sigmoid function turns the score into a probability between 0 and 1. Big positive scores become close to 1, big negative scores close to 0.
  3. 3

    Decide

    If the probability is at least the threshold (0.5 by default), predict 1. Otherwise predict 0.
Despite its name, logistic regression is used for classification. The name comes from the maths of the squashing step.
Python: the sigmoid, and a tiny made-up examplepython
import numpy as np
from sklearn.linear_model import LogisticRegression

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in [-4, -1, 0, 1, 4]:
    print("score", z, "-> probability", round(sigmoid(z), 3))

# Made-up data: hours of practice and whether a learner passed a driving test.
hours = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]])
passed = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])

model = LogisticRegression().fit(hours, passed)
for h in [2, 5, 9]:
    p = model.predict_proba([[h]])[0, 1]
    print(h, "hours: chance of passing", round(p, 2), "-> predict", model.predict([[h]])[0])
predict_proba gives the probability of each group. Column 1 is the chance of group 1. predict applies the 0.5 threshold for you.

Choosing the threshold

0.5 is only a default. If missing a positive case is very costly, lower the threshold so the model says "yes" more often. If false alarms are costly, raise it. The lesson on precision and recall shows how to choose.

Scale your features

Features often have very different ranges. One might be between 0 and 1, another in the thousands. Many models, including logistic regression, train better when every feature is put on a similar scale. StandardScaler shifts each feature to average 0 and spread 1.
Put the scaler and the model in a pipeline. The pipeline learns the scaling from the training rows only, and applies the same scaling to any new rows. This avoids a quiet form of leakage.
Python: a pipeline with scalingpython
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
print("Test accuracy:", round(model.score(X_test, y_test), 3))
print("Probabilities for the first test wine:", model.predict_proba(X_test[:1]).round(3))
Logistic regression also handles more than two groups, as the three wine growers show. stratify=y keeps the share of each group the same in the training and test rows.
Watch out
A probability from a model is not a guarantee. "0.9" means the model's score is high, not that nine out of ten such cases will surely be positive. Check whether the probabilities match reality on test data before you rely on them.
Try this. In the first code, change the 6-hour learner to passed (1). How do the chances change?

Watch and read more

scikit-learn: Pipelines and composite estimatorsscikit-learn · reading

Lab

A scaled logistic regression that sorts breast-tumour measurements into two groups, scored on a held-out test set.

~20 min

The problem

The breast cancer dataset in scikit-learn has 569 rows of measurements from tumour images, labelled 0 (malignant) or 1 (benign). It is a classic teaching dataset. Nothing here is medical advice, and real diagnosis is done by doctors with many more checks.

  1. Run the starter. It splits the data with stratify=y.
  2. Build model as a pipeline: StandardScaler then LogisticRegression. Fit it on the training rows.
  3. Set test_acc to the accuracy on the test rows.
  4. Set probs to the predicted probability of group 1 for the first five test rows.
  5. Print them. Which of the five is the model least sure about?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your model is 95% accurate on this test set. Why would that still not be enough to use it for real patients?Reveal

The test set is small and comes from one old collection of images, so real clinics may differ. Accuracy also hides which mistakes were made: missing a malignant tumour is far worse than a false alarm. A real medical tool needs careful clinical testing, approval under the rules of each country, and doctors in charge of decisions.

Q2Try the model without the scaler. What changes and why?Reveal

You may see a ConvergenceWarning and slightly different accuracy. The features have very different ranges, so the training steps struggle to settle. Scaling puts them on a similar footing. The exact numbers depend on the split and the library version, so compare for yourself.

Please sign in to continue.

Questions people ask

Why not use linear regression with 0 and 1 as the label?

A straight line can predict values below 0 or above 1, which make no sense as chances. The sigmoid in logistic regression keeps the output between 0 and 1.

What does accuracy mean?

The share of predictions that were right. It is easy to understand, but can mislead when one group is rare. The lesson on precision and recall explains why.

Do I always need to scale?

Not for every model. Decision trees do not care about scale. Logistic regression, nearest neighbours and k-means usually do better with scaled features. A pipeline with a scaler is a safe habit.

What is a ConvergenceWarning?

It means the training steps stopped before settling on the best weights. Scaling the features usually fixes it. You can also raise max_iter.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress