Machine learning fundamentalsMiddleLesson 1186 min read

Project: predict crop yield from rainfall

An end-to-end project on made-up data: explore, split, set a baseline, try a straight line and a curve, and report the result honestly.

✦ Ask the tutor 🧪 Lab · 40 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Project: predict crop yield from rainfall

An end-to-end project on made-up data: explore, split, set a baseline, try a straight line and a curve, and report the result honestly.

1/5

In simple words

Farmers know that a crop needs rain, but too much rain can drown it. If you drew rain on one side and harvest on the other, you would not get a straight line; you would get a hill. In this project you teach a computer to find that hill from examples, and then check honestly how good its guesses are.

Danger

All the data in this lesson is made up by a formula with a fixed random seed. It is not real rainfall or yield data for any place or crop, and the numbers must not be used for any farming decision. Real yields depend on soil, seed, irrigation, pests, timing and much more.

The project plan

1
Frame the question
Predict yield per field (made-up tonnes per hectare) from seasonal rainfall (made-up mm).
2
Look at the data
Plot it. Check ranges, missing values and odd points.
3
Split
Keep a test set aside before any modelling.
4
Baseline
Predict the training average for every field.
5
Models
Try a straight line, then a curve. Choose using the training data only.
6
Test once
Score the chosen model on the test set.
7
Report honestly
Numbers, baseline, limits and what the data is.

Look before you model

Python: make and plot the made-up datapython

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# MADE-UP data from a formula with noise. Not real measurements.
rng = np.random.default_rng(42)
n = 200
rain_mm = rng.uniform(300, 1500, n)
yield_t = 1.0 + 0.006 * rain_mm - 0.0000032 * rain_mm ** 2 + rng.normal(0, 0.35, n)
fields = pd.DataFrame({"rain_mm": rain_mm, "yield_t_per_ha": yield_t})

print(fields.describe().round(2))
print("Missing values:", int(fields.isna().sum().sum()))
plt.scatter(fields["rain_mm"], fields["yield_t_per_ha"], s=12)
plt.xlabel("seasonal rainfall (mm, made up)")
plt.ylabel("yield (t/ha, made up)")
plt.title("Made-up data for practice")

The plot shows a hill: yield rises with rain, peaks, then falls. That already tells us a straight line will struggle.

Letting a linear model bend

A straight line has the form a × rain + b. If we also give it rain² as a second feature, it becomes a × rain + c × rain² + b, which can make a hill. scikit-learn's PolynomialFeatures(degree=2) adds the squared column for you. The model is still "linear" in its weights, but its shape can curve.

Model	What it can draw	Risk
Baseline (average)	A flat line	Ignores rainfall completely
Linear regression	A straight sloped line	Cannot follow the hill
Degree-2 polynomial	One smooth hill or valley	Fine here; wrong if the real shape differs
Random forest	Steps that follow the data	Can be jumpy with little data

Report honestly

State that the data is made up, and how it was made.
Give the test error and the baseline error, in the label's units.
Say how you chose the model, and that the test set was used once.
List what the model ignores: here, everything except rainfall.
Do not claim cause and effect, or that it works anywhere else.

Real example

A student team builds a yield model on real data from their district. It does well on their test set. But all the data comes from three years with normal rains. In a drought year, the model would be guessing far outside what it has seen. Their honest report says: "Tested only for rainfall between X and Y mm. Not tested in drought years." That one sentence can save someone from a bad decision.

Try this. In the code, change the noise 0.35 to 0.1, then to 1.0. How clear is the hill each time?

Watch and read more

scikit-learn: Polynomial featuresscikit-learn · reading

Lab

A complete, honest mini-project on made-up data: test set sealed, baseline, straight line, curve chosen by cross-validation, one final test score.

~40 min

The problem

The starter makes the made-up rainfall and yield data and seals a test set. Do the rest:

Set baseline_mae: the test error when you predict the training average for every test field.
Compare two candidates with 5-fold cross-validation on the training rows only: LinearRegression() and a pipeline of PolynomialFeatures(degree=2) then LinearRegression(). Use scoring="neg_mean_absolute_error". Store the mean CV MAE (as a positive number) in cv_mae["line"] and cv_mae["curve"].
Pick the candidate with the lower CV MAE, fit it on all the training rows, and call it best_model.
Only now, set best_mae to its test MAE.
Print a three-line report: the data is made up; baseline versus model test error; the rainfall range the model was tested on.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_absolute_error

# MADE-UP data from a formula with noise. Not real measurements.
rng = np.random.default_rng(42)
n = 200
rain_mm = rng.uniform(300, 1500, n)
yield_t = 1.0 + 0.006 * rain_mm - 0.0000032 * rain_mm ** 2 + rng.normal(0, 0.35, n)

X = rain_mm.reshape(-1, 1)
X_train, X_test, y_train, y_test = train_test_split(X, yield_t, test_size=0.25, random_state=0)
print("Training fields:", len(X_train), " Test fields (sealed):", len(X_test))

# TODO 1: baseline test MAE (predict the training average)
baseline_mae = None

# TODO 2: mean 5-fold CV MAE on the training rows for each candidate
cv_mae = {}

# TODO 3: the better candidate, fitted on all training rows
best_model = None

# TODO 4: test MAE of best_model (the only use of the test set)
best_mae = None

print("CV MAE:", cv_mae)
print("Baseline test MAE:", baseline_mae, " Model test MAE:", best_mae)

Output

No output. Add print(...) to see values.

You are done when

The test set is used only for the baseline and the one final score
Both candidates are compared by cross-validation on the training rows
The curve is chosen and clearly beats the baseline; the straight line barely does
The report says the data is made up and gives the tested rainfall range

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You know the true formula here, because you made the data. How does that help you judge your method, and why can you never do this with real data?Reveal

You can check whether the model found the real shape: the fitted curve should peak near where the formula peaks (a little under 950 mm here). That tells you the method works when the pattern is known. With real data the true rule is unknown, so you can only judge by honest tests on unseen data, which is why the habits in this track matter.

Q2A friend suggests adding degree 10 to the candidates because "more flexible is better". What would you expect, and how would the process protect you?Reveal

A degree-10 curve can wiggle to fit noise, especially near the edges of the data, so it would likely overfit (and large powers of rainfall can cause numerical trouble without scaling). Because you choose by cross-validation on the training rows, a model that overfits should show a worse CV error and would not be chosen. The sealed test set gives a final honest check either way.

Please sign in to continue.

Questions people ask

Why not use real crop data?

Real data is the goal, but it needs care: permission, cleaning and many more features. Made-up data lets you learn the steps when you already know the true pattern, so you can see if your model finds it. When you move to real data, check the source and its licence, and expect more mess.

How do I choose between the straight line and the curve without touching the test set?

Use cross-validation on the training rows. Compare the mean error of each model, pick the better one, and only then score it once on the test set.

What if the model is used outside the range of the data?

Predictions outside the range seen in training are guesses. A degree-2 curve keeps falling (or rising) forever beyond the data. Always state the range the model was trained and tested on.

Is lower error always better?

Lower test error is better if the test is fair. But also consider simplicity, how stable the result is across splits, and whether people can understand and check the model.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Project: predict crop yield from rainfall

Project: predict crop yield from rainfall

The project plan

Frame the question

Look at the data

Split

Baseline

Models

Test once

Report honestly

Look before you model

Letting a linear model bend

Report honestly

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask