🔍Samples & Inference

Nobody can count every frog in a marsh or ask every traveller on a steppe. So you take a sample: a small group you *can* measure, chosen to stand in for the whole population.

The whole method rests on one assumption — that the sample is a fair miniature of the population. When that holds, a share found in the sample can be scaled up:

estimate = (sample share) x (population size)

Sample 40 frogs, find 14 spotted, and the share is 14/40 = 0.35. For a population of 600, estimate 0.35 x 600 = 210 spotted frogs.

Say "about". An estimate carries the sample's uncertainty with it, and reporting it as an exact number claims more than the evidence supports.

A sample is random when every member of the population has the same chance of being in it. That is a property of the *method*, not of the result.

Numbering every household and drawing forty numbers from a sealed jar is random. Taking every tenth name from an alphabetical roll, starting from a random point, is fine too, because alphabetical order has nothing to do with the question.

Two things go wrong, and they are different faults:

• ◆Bias. The method reaches a particular group. Asking only the people queuing for a ferry whether another ferry is needed, or inviting only the unhappy to come and complain, produces a sample that leaned before a single answer was given. A bigger sample does not fix bias; it just makes the wrong answer more precise.
• ◆Too small. The method is fair but three people cannot speak for a steppe.

Diagnose which fault you are looking at before you suggest a remedy.

A sample tells you what *is*. To claim what causes what, the design has to do more. Three designs cover almost every study, and they differ in one thing only: who decided the condition.

• ◆A sample survey draws a sample from a population and asks it something. Nobody is set against anybody, and the result describes the population.
• ◆An observational study sets two groups that were already there beside each other — the growers who *already* water twice a day against the rest. The surveyors only record.
• ◆An experiment hands out the condition itself. The subjects are sorted into groups, one group receives the treatment, another is kept at the usual condition as the control group, and the response is measured at the end.

The word that separates the last two is assignment. In an observational study the subjects chose their own group, and whatever led them to choose it walks into the result with them. In an experiment the surveyors sort them, and if they sort them at random then every other difference — age, habit, weather, keenness — is scattered evenly between the groups. That is what leaves the treatment as the only thing available to explain a difference.

So the two designs are entitled to different sentences:

• ◆Observational study: "these two things go together." Never "this one produced that one."
• ◆Randomised experiment: "this treatment produced this difference."

Random sampling and random assignment are different jobs, and it is worth keeping them apart. Random sampling decides who is in the study at all, and it is what lets the result speak for a wider population. Random assignment decides which group each subject lands in, and it is what lets the result speak of cause. A study can have one, both or neither.

Three things break a design, and a bigger study mends none of them:

• ◆Self-selection. The subjects volunteered, so only the keenest are measured.
• ◆No control group. Everyone was treated, so an ordinary good spell looks exactly like an effect.
• ◆Confounding. Two things were changed at once, so no result can say which of them did the work.

To estimate a population you cannot line up, mark some of it and see how many marks come back.

Catch 60 carp, tag them, release them. Later catch 50 carp and find 6 already tagged. If the tagged fish mixed back in evenly, the share tagged in the second catch should match the share tagged in the whole lake:

6/50 = 60/N

Cross-multiply: 6N = 60 x 50 = 3000, so N = 500 carp.

Sense-check the shape of the answer. The estimate must be larger than the number you tagged, and the smaller the fraction that came back marked, the larger the population must be. If your answer came out below 60, the proportion was set up upside down.

Two fair samples from the same population will not agree exactly, and neither will agree exactly with the truth. The margin of error puts a size on that wobble.

A classroom rule of thumb: for a sample of n people, the margin of error is roughly 1/sqrt(n), expressed as a percent.

• ◆n = 100: margin about 1/10 = 10%
• ◆n = 400: margin about 1/20 = 5%
• ◆n = 2500: margin about 1/50 = 2%

If 52% of a sample of 400 use the north road, the plausible interval is 52% +/- 5%, that is 47% to 57%.

Notice the square root. Four times the sample halves the margin; to halve it again you need four times as many again. That is why serious surveys stop at a few thousand people — the next improvement costs far more than it is worth.

And pooling beats averaging. To combine several samples of different sizes, add the counts and divide once; averaging the separate percentages gives the small samples the same say as the large ones.