Machine learning fundamentalsMiddleLesson 1115 min read

Decision trees and forests

A decision tree asks a chain of yes/no questions. A random forest asks many slightly different trees and takes a vote.

✦ Ask the tutor🧪 Lab · 20 min📝 Lesson test
Lesson in motion

In 60 seconds

Decision trees and forests

A decision tree asks a chain of yes/no questions. A random forest asks many slightly different trees and takes a vote.

1/5
In simple words
Remember the game "twenty questions"? "Is it an animal? Yes. Does it live in water? No. Does it have stripes? Yes. It's a tiger!" A decision tree plays this game with data. Each question splits the examples into smaller groups until each group is mostly one kind.
A decision tree is a flowchart the computer builds from examples. At each step it picks the feature and the cut-off that best separate the labels, for example "petal length below 2.45 cm?". Then it repeats inside each branch. The ends are called leaves, and each leaf gives a prediction.
Python: print a small tree as rulespython
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tree.fit(iris.data, iris.target)

print(export_text(tree, feature_names=list(iris.feature_names)))
print("Classes:", list(iris.target_names))
The printed rules are easy to read: you can follow any flower down the tree and see why it got its label. This is the big strength of a small tree.

Strengths and weaknesses of one tree

Weaknesses
  • A deep tree can memorise the training data, noise and all
  • Small changes in the data can give a very different tree
  • Its boundaries are made of steps, not smooth curves
Strengths
  • Easy to explain when it is small
  • No need to scale features
  • Handles mixed features and curved patterns

Random forests: many trees vote

A random forest builds many trees. Each tree trains on a random sample of the rows, and at each split it only looks at a random subset of the features. So the trees make different mistakes. For a prediction, all trees vote and the majority wins. The random mistakes tend to cancel out.
  1. 1

    Sample

    Each tree gets its own random sample of the training rows (with repeats).
  2. 2

    Grow

    Each tree grows, choosing each split from a random subset of features.
  3. 3

    Vote

    For a new row, every tree predicts; the forest takes the majority.
Forests are often a strong first choice for tables of numbers. The price is that you cannot read a hundred trees the way you can read one.

Feature importance, with care

Forests report feature_importances_: roughly, how much each feature helped the splits. It is a useful hint about what the model relies on.
Watch out
Feature importance is not cause and effect. It also can be misleading when two features carry the same information (they split the credit), and it tends to favour features with many different values. Treat it as a clue for further checking, not a final answer.
SettingWhat it controls
max_depthHow many questions deep a tree may go. Smaller means simpler.
min_samples_leafThe fewest rows allowed in a leaf. Larger means smoother.
n_estimatorsHow many trees in a forest. More is steadier but slower.
random_stateFixes the randomness so results repeat.
Try this. Change max_depth=2 to max_depth=4 in the code. Are the extra rules still easy to follow?

Watch and read more

scikit-learn: Decision treesscikit-learn · reading

Lab

One decision tree and one random forest on the wine data, compared on the same test rows, plus the forest's top feature.

~20 min

The problem

  1. Run the starter. It loads the wine data as a table and splits it.
  2. Fit a DecisionTreeClassifier(random_state=0) named tree and a RandomForestClassifier(n_estimators=100, random_state=0) named forest, both on the training rows.
  3. Set tree_acc and forest_acc to their test accuracies.
  4. Set top_feature to the name of the feature with the highest forest.feature_importances_.
  5. Print all three. Then change the forest's random_state to 1 and 2. Does the top feature stay the same?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1The forest scores higher than the tree on this split. What would you need before claiming the forest is better in general?Reveal

More than one split. The test set here has only 45 wines, so one or two mistakes change the accuracy a lot. Repeat with several splits or use cross-validation, and see if the forest wins most of the time.

Q2If the top feature changes when you change random_state, what does that tell you?Reveal

That the importance scores of the top features are close, and the ranking is not stable. Several features may carry similar information. Do not build a story on the single top feature; look at the group of important features and check them with other methods.

Please sign in to continue.

Questions people ask

Does a tree need scaled features?

No. A tree only asks "is this feature above this value?", so the unit does not matter.

How many trees should a forest have?

A hundred is a common starting point and the scikit-learn default. More trees rarely hurt accuracy, but they cost time and memory. Stop adding when test results stop improving.

Can trees do regression?

Yes. DecisionTreeRegressor and RandomForestRegressor predict numbers. Each leaf predicts the average label of its training rows.

What about gradient boosting?

Boosting also combines many trees, but builds them one after another, each fixing the errors of the ones before. It is often very accurate on tables. scikit-learn has HistGradientBoostingClassifier. Try it after you are comfortable with forests.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress