Math Vale

📊 Statistics Steppe · Statistics

Correlation & Regression

Use a fitted line to predict and to measure how far each point misses it, read the strength and direction of a relationship from the correlation coefficient r, and keep correlation apart from causation.

In short

  • A fitted line is a rule for predicting: substitute an x and read off the predicted y, remembering that the answer is a trend value, not a measurement.
  • In y = mx + b the slope is a rate in up-units per across-unit, and the intercept is the prediction when x is 0.
  • A residual is actual minus predicted, so a positive residual means the reading sits above the line and a negative one means below.
  • The least-squares line is the line that makes the total of the squared residuals as small as possible, which is why squared residuals are the way to compare two candidate lines.
  • r runs from -1 to 1: its sign gives the direction and its distance from 0 gives the strength, so -0.9 is stronger than 0.6.
  • A correlation, however strong, can be explained by a lurking variable, by reverse causation or by chance in a small sample.
  • Constant first differences mean linear, constant ratios mean exponential and constant second differences mean quadratic — run the three checks in that order and stop at the first one that settles.

A line you can predict with, inside the data and outside it

A scatter plot shows what happened. A fitted line — also called a regression line or a line of best fit — turns that into a rule you can use: give it an x, and it hands back a predicted y.

The rule is written the same way every line is written:

y = 2.5x + 40

Here x might be days since planting and y the height of a sapling in cm. To predict the height at 20 days, substitute:

y = 2.5 times 20 + 40 = 50 + 40 = 90 cm

Two numbers carry all the meaning, and both are read in context.

  • The slope 2.5 is a rate: each extra day goes with about 2.5 cm more height. Its units are the up-units per one across-unit, cm per day.
  • The intercept 40 is the prediction at x = 0: about 40 cm on the day of planting.

The word "about" is doing real work. A fitted line describes a trend across many readings, not a promise about any single one.

Predicting at an x inside the range of the readings is called interpolation. That is what the line was built for, and it is usually safe.

Predicting at an x outside that range is extrapolation, and it rests on an assumption nobody checked: that the same straight-line pattern carries on into ground where no readings were taken.

Suppose saplings were measured from day 2 to day 20 and the model is y = 2.5x + 40.

  • At day 15 the prediction is 77.5 cm. That is interpolation — reasonable.
  • At day 500 the prediction is 1290 cm, a sapling thirteen metres tall. The arithmetic is perfect and the answer is nonsense, because saplings stop growing and the line does not.

A strong r does not rescue an extrapolation. r describes how tightly the readings you have hug the line; it says nothing about x values you never observed. Extrapolating is allowed when you have a reason to expect the pattern to continue — but the warning always travels with the number.

Residuals: the misses, with their signs

No line goes through every point. The residual of a reading is how far it missed:

residual = actual - predicted

That order matters, because the sign is the message. A positive residual means the reading sits above the line; a negative one means it sits below.

Suppose the line is y = 4x + 10 and a plot with 6 rows gave a harvest of 30 kg.

  • Predicted: 4 times 6 + 10 = 34 kg
  • Residual: 30 - 34 = -4 kg, so that plot came in 4 kg under the trend.

Residuals are also how you compare two candidate lines. Add up the squares of the residuals — the sum of squared residuals — and the smaller total belongs to the closer line. Squaring is deliberate: it stops a miss of +5 cancelling a miss of -5, and it makes one large miss count for far more than several small ones.

The least-squares line is simply the line that wins that contest against every other line. You will not be asked to find it by hand here; what matters is knowing what it is chosen to do.

r: one number for direction and strength

The correlation coefficient r measures how close a set of points comes to lying on one straight line. It always sits between -1 and 1.

  • The sign gives the direction. Positive: the two rise together. Negative: one rises as the other falls.
  • The size, |r|, gives the strength. Near 1 means the points hug a line; near 0 means no straight-line drift at all.

A rough reading, good enough for most work:

  • |r| above about 0.8 — strong
  • 0.5 to 0.8 — moderate
  • 0.2 to 0.5 — weak
  • below about 0.15 — almost nothing linear

So r = -0.85 is a strong negative relationship, and r = 0.1 is almost nothing at all. Notice that -0.9 is stronger than 0.6: strength is distance from 0, and the minus sign only reports which way the line leans.

Be clear about what r does not tell you. It says nothing about how steep the line is. It measures straight-line fit only, so a beautiful curve — points that rise then fall — can have r close to 0. And a single outlier can drag r a long way, which is why you look at the plot as well as the number.

Which model does the table want?

A line is only one shape. Before fitting anything you have to decide what shape to fit, and a table of readings taken at evenly spaced x values will tell you — if you ask it in the right order.

1. First differences. Subtract each reading from the next.

x: 0 1 2 3 4 y: 5 8 11 14 17 first differences: 3, 3, 3, 3

Constant first differences mean the same amount is added at every step, so the model is linear: y = 3x + 5. The common difference is the slope, and the reading at x = 0 is the intercept.

2. Ratios. If the differences keep changing, divide each reading by the one before it.

y: 2 6 18 54 162 first differences: 4, 12, 36, 108 (not constant) ratios: 3, 3, 3, 3

Constant ratios mean each reading is the one before multiplied by a fixed factor, so the model is exponential: y = 2 * 3x. The ratio is the base, and the reading at x = 0 is the starting amount.

3. Second differences. If neither of those settles, take the differences of the differences.

y: 4 7 14 25 40 first differences: 3, 7, 11, 15 (not constant) ratios: 1.75, 2, 1.79, 1.6 (not constant) second differences: 4, 4, 4

Constant second differences mean the model is quadratic. The second difference is 2a, so here a = 2, and the readings supply the rest: y = 2x2 + x + 4.

Stop at the first check that comes out constant — there is nothing to gain by going further. And the order matters for a reason: over two or three readings a quadratic and an exponential both simply "grow faster and faster", and only the third check tells them apart.

Which model to fit is not only arithmetic. A quantity that grows by a fixed amount each period is linear; one that grows by a fixed percent is exponential; one whose rate of change itself climbs steadily is quadratic. Where the table and the story disagree, look again at the readings — one of them is being misread.

Correlation is not causation

Two quantities moving together is evidence that they move together. On its own it is not evidence that one produces the other.

Before accepting a cause, work through three alternatives.

  • A lurking variable — some third thing moves both. Iced-fruit sales and sunburn cases rise and fall together all summer, and neither causes the other: hot sunny weather drives both.
  • Reverse causation — the arrow points the other way. Cottages that burn the most firewood report the coldest rooms. Firewood does not cool a house; cold houses are why the firewood is burned.
  • Coincidence — with only a handful of readings, a large r turns up by chance surprisingly often. Five weeks of data showing r = 0.91 between grain prices and the length of a winning horse's name is a fluke, not a finding.

What does support a causal claim is an experiment: assign the treatment yourself, at random, so that lurking variables are spread evenly between the groups. Observed data, however tidy, can only ever suggest where to look.

This idea carries into the rest of statistics — samples, bias, experimental design — and it is the most useful thing here outside a mathematics lesson.

Worked examples

Example 1

A fitted line for a market survey is y = 6x + 25, where x is the stalls open and y is the coins taken. Predict the coins taken with 14 stalls open, then find the residual for a day when 14 stalls took 121 coins.

  1. Substitute x = 14 into the model: y = 6 times 14 + 25.
  2. 6 times 14 = 84, and 84 + 25 = 109, so the prediction is 109 coins.
  3. Residual = actual - predicted = 121 - 109 = 12.
  4. The residual is positive, so that day sat 12 coins above the trend line.

Example 2

Three readings are (2, 15), (4, 22), (6, 32). The line y = 4x + 8 is proposed. Work out the sum of the squared residuals.

  1. Predictions: at x = 2, 4 times 2 + 8 = 16; at x = 4, 24; at x = 6, 32.
  2. Residuals (actual - predicted): 15 - 16 = -1, 22 - 24 = -2, 32 - 32 = 0.
  3. Square each one: (-1)2 = 1, (-2)2 = 4, 02 = 0.
  4. Sum of squared residuals = 1 + 4 + 0 = 5. A rival line beats this only by scoring lower than 5.

Example 3

Four surveys report r = 0.6, r = -0.9, r = 0.1 and r = -0.4. Which shows the strongest linear relationship, and which the weakest?

  1. Strength is distance from 0, so drop the signs first: 0.6, 0.9, 0.1, 0.4.
  2. The largest of those is 0.9, which came from r = -0.9.
  3. So r = -0.9 is the strongest — the minus sign reports a downward direction, not a weak one.
  4. The smallest size is 0.1, so r = 0.1 is the weakest: almost no straight-line relationship at all.

Example 4

Readings taken at x = 0, 1, 2, 3, 4 are y = 3, 5, 11, 21, 35. Which model fits, and what does it predict at x = 8?

  1. First differences: 5 - 3 = 2, 11 - 5 = 6, 21 - 11 = 10, 35 - 21 = 14. They are not constant, so the model is not linear.
  2. Ratios: 5/3 = 1.67, 11/5 = 2.2, 21/11 = 1.91, 35/21 = 1.67. They are not constant either, so the model is not exponential.
  3. Second differences: 6 - 2 = 4, 10 - 6 = 4, 14 - 10 = 4. Constant, so the model is quadratic.
  4. The second difference is 2a, so 2a = 4 and a = 2. The reading at x = 0 gives c = 3, and at x = 1, 2 + b + 3 = 5 gives b = 0. The model is y = 2x2 + 3.
  5. At x = 8: 2 * 64 + 3 = 128 + 3 = 131. Note how far that sits above what a line through the first two readings would have predicted.

Practice problems, with solutions

Three problems of increasing difficulty, each with the full working. In the game these are generated fresh every time; these three are fixed so this page always shows the same ones.

Problem 1

Difficulty 1 of 5

Well-wardens noted the days since the rains and the water still standing in the cistern. The line of best fit is y = 2x + 5, where x is the days since the rains (days) and y is the water left in the cistern (litres). Predict the water left in the cistern at x = 4 days.

Answer: 13 litres

  1. The model is y = 2x + 5.
  2. Substitute x = 4: y = 2 times 4 + 5
  3. y = 8 + 5 = 13
  4. So the prediction is about 13 litres — a value read off a trend, not a measurement.

Problem 2

Difficulty 3 of 5

Well-wardens noted the days since the rains and the water still standing in the cistern. The line of best fit is y = 4x + 36, where x is the days since the rains (days) and y is the water left in the cistern (litres). One reading was 6 days with a water left in the cistern of 66 litres. Find the residual for that reading: actual minus predicted.

Answer: 6 litres

  1. Predicted: y = 4 times 6 + 36 = 60
  2. Residual = actual - predicted = 66 - 60 = 6
  3. The residual is positive, so this reading sits 6 litres ABOVE the line.

Problem 3

Difficulty 4 of 5

Well-wardens noted the days since the rains and the water still standing in the cistern. The line of best fit is y = -10x + 220, where x is the days since the rains (days) and y is the water left in the cistern (litres). What does the slope tell you?

  1. each extra day goes with about 10 litres less water left
  2. each extra litre of water left goes with about 10 fewer days
  3. each extra day goes with about 220 litres more water left
  4. each extra day goes with about 10 litres more water left

Answer: A. each extra day goes with about 10 litres less water left

  1. In y = mx + b, m is the slope: the change in y for a one-unit rise in x.
  2. Here m = -10, so each extra day matches a fall of about 10 litres.
  3. The word "about" matters: the line is a trend, so this is an average rate, not a promise for any one reading.
  4. The intercept 220 answers a different question — the prediction at x = 0.

Common mistakes

  • Subtracting the wrong way round: writing predicted minus actual, which flips the sign of every residual and makes a point above the line look like one below it.
  • Judging strength by the sign instead of the size: calling 0.6 stronger than -0.9 because 0.6 is the larger number.
  • Reading the intercept as the slope: saying "the harvest goes up by 40 kg per row" when 40 is the prediction at 0 rows.
  • Swapping the units in a slope: reporting kg per row as rows per kg, which reverses what the rate means.
  • Treating a strong correlation as proof of a cause, when a lurking third variable explains both quantities just as well.
  • Extrapolating far outside the data and trusting the answer, as though the line had been tested at x values nobody ever measured.
  • Reading r = 0 as "no pattern at all", when it only means no straight-line pattern — a clear curve can still give r near 0.
  • Calling a table linear because the first two or three differences happen to look close, without checking the differences right along the row.
  • Calling a table exponential when it is the SECOND differences that are constant: growth that speeds up steadily is quadratic, and only constant ratios make it exponential.

What you should be able to do

  • Use a given regression equation to predict a value and interpret its slope and intercept in context.
  • Compute a residual as actual minus predicted, and say what its sign means.
  • Interpret a correlation coefficient r for strength and direction, and match r to a scatter plot.
  • Explain why a strong correlation does not by itself show that one variable causes the other.

Where this fits in the curriculum

Common Core

  • HSF-LE.A.1.B

    High school — Recognise situations in which one quantity changes at a constant rate per unit interval relative to another.

  • HSS-ID.B.6.A

    High school — Fit a function to the data, and use functions fitted to data to solve problems in the context of the data.

  • HSS-ID.B.6.B

    High school — Informally assess the fit of a function by plotting and analysing residuals.

  • HSS-ID.C.7

    High school — Interpret the slope (rate of change) and the intercept (constant term) of a linear model in the context of the data.

  • HSS-ID.C.8

    High school — Compute and interpret the correlation coefficient of a linear fit.

  • HSS-ID.C.9

    High school — Distinguish between correlation and causation.

Ontario

  • MTH1W.D1

    Grade 9 de-streamed — Describe the collection and use of data, and represent and analyse data involving one and two variables.

    Ontario numbers its Data expectations per grade document and the specific numbering could not be verified line by line here, so this tag names the strand's OVERALL expectation rather than guessing at a specific one.

SAT

Learn these first