Evaluating statistical claims: observational studies and experiments

You get a short paragraph about a study and four choices. Some questions add a table. The last line asks one of these:

• “Which of the following is the largest population to which the results of the study can be generalized?”
• “Based on the design of the study, what is the largest group to which these results can be applied?”
• “Which of the following is the most appropriate conclusion based on the study?”

For the first two, each choice names a group. For the last one, each choice is a full sentence about what the study found.

There's nothing to calculate, so leave Desmos closed. Even when there's a table, you won't need to add anything in it. If the question gives an estimate with a margin of errorHow far an estimate from a sample could be off, like plus or minus 3%.Read the lesson, you want Inference from sample statistics instead.

A bike-share program has $2{,}000$ members. Of these, $900$ have yearly passes, and $360$ of the yearly pass holders live downtown. The program selects $50$ of those $360$ members at random and asks each one how satisfied they are with the bikes.

Which of the following is the largest population to which the results of the study can be generalized?

• A. All $2{,}000$ members
• B. The $900$ members with yearly passes
• C. The $360$ yearly pass holders who live downtown
• D. The $50$ members the program selected

1. Find “at random” and see what it describes. Here it describes how the program picked people for the study. (When it describes who got which treatment, that's the cause question, further down.)
2. Say the whole group they picked from before you look at the choices. The $50$ came from “those $360$ members,” and those are the yearly pass holders who live downtown. The $50$ are the sample, the people the study asked. The $360$ are the population, the group the results can describe.
3. Match a choice to the group you said, word for word.
   • A and B are bigger groups. The other members in them had no chance of being picked, so the sample tells you nothing about them.
   • D is the sample itself. Picking at random lets the results reach everyone in the group the sample came from, and the question asks for the largest group.
   • C keeps both conditions, a yearly pass and living downtown, so C is correct.

“Largest” means the largest group the study lets you describe. The $2{,}000$ is the biggest number on the page, but that doesn't make it the answer.

If the paragraph never says the sample was picked at random, as with volunteers or people who chose to send back a survey, the results describe only the people in the study. Measuring them carefully afterward doesn't change that.

A hiking club has $250$ members. Of these, $90$ have belonged to the club for at least $3$ years, and $90$ led a hike last year. Omar selects $40$ members at random from the $90$ who have belonged for at least $3$ years, then records their weekly miles for the next $12$ months.

What is the largest group to which Omar's results can be applied?

• A. All $250$ members of the club
• B. The $90$ members who led a hike last year
• C. The members who have belonged to the club for at least $4$ years
• D. The $90$ members who had belonged to the club for at least $3$ years on the day the study started

D is correct. A reaches too far, and the other two each change one detail.

By the end of the study, every member Omar picked has belonged for at least $4$ years, so C can look like an update.

Now try one.

+−100%Reset⤢

The table summarizes all paint orders received and completed at a shop during one month. Aziz selected at random $56$ of the $160$ orders received that month that used a resin-to-hardener ratio of $4:1$. For each selected order, Aziz recorded whether the finished paint met the shop's viscosity standard.

Which of the following is the largest population to which the results of the study can be generalized?

1. A

   All $420$ paint orders received that month

2. B

   The $56$ paint orders Aziz selected

3. C

   The $160$ orders received that month that used a resin-to-hardener ratio of $3:1$

4. D

   The $160$ orders received that month that used a resin-to-hardener ratio of $4:1$

Show the answer

• A

  All $420$ paint orders received that month

  This overgeneralizes by dropping the $4:1$ ratioA comparison of two amounts, like 3 to 5.Read the lesson requirement and extending the results to all $420$ orders, including orders with other ratios.

• B

  The $56$ paint orders Aziz selected

  This treats the sample of $56$ selected orders as the population. Because those orders were randomly selected from the $160$ orders with a $4:1$ ratioA comparison of two amounts, like 3 to 5.Read the lesson, the results can be generalized to that larger group.

• C

  The $160$ orders received that month that used a resin-to-hardener ratio of $3:1$

  This keeps the count of $160$ but changes the resin-to-hardener ratioA comparison of two amounts, like 3 to 5.Read the lesson from $4:1$ to $3:1$. Aziz sampled no orders from the $3:1$ group.

• D

  The $160$ orders received that month that used a resin-to-hardener ratio of $4:1$

  Correct answer

  Aziz randomly selected the $56$ orders from the $160$ orders that used a $4:1$ ratioA comparison of two amounts, like 3 to 5.Read the lesson. Random sampling allows the results to be generalized to all $160$ orders in that group.

Solution

Identify the population from which the sample was randomly selected. Aziz chose $56$ orders at random from the $160$ orders that used a resin-to-hardener ratioA comparison of two amounts, like 3 to 5.Read the lesson of $4:1$. Because the sample was drawn randomly from those $160$ orders, the results can be generalized to all $160$ orders received that month that used a $4:1$ ratio. The study cannot be generalized to all $420$ orders because the sample included only orders with that ratio.

Report a problem

A bus company runs $160$ trips a week, and $70$ of them are on its express route. A researcher randomly selects $36$ of those $70$ express trips. The table shows only the $24$ selected trips that carried at least $25$ riders.

The researcher picked from the $70$ express trips, so they're the population. The table can't change that, and it doesn't matter that it shows fewer trips than were picked. The question asks about the results of the study, and the study measured every trip it picked.

+−100%Reset⤢

Nilsen’s fish smokehouse uses 144 drying racks, some removable and some fixed in place. Bianca randomly selected 64 racks from the 96 removable drying racks at Nilsen’s fish smokehouse.

During one morning’s inspection, she recorded each selected rack’s location, the number of trays of fish it held, and whether it had enough unused space for another standard tray. Every selected rack was inspected. The table reports results only for the 42 selected racks holding at least 6 trays of fish. Of the other 22 selected racks, 10 had enough unused space for another standard tray and 12 did not.

Which of the following is the largest population to which the proportion of all 64 selected racks with space for another standard tray can be generalized?

1. A

   The 42 inspected racks holding at least 6 trays of fish

2. B

   The 96 removable drying racks at Nilsen’s fish smokehouse

3. C

   All 144 drying racks used at Nilsen’s fish smokehouse during inspection

4. D

   The 64 drying racks randomly selected for inspection at Nilsen’s smokehouse

Show the answer

• A

  The 42 inspected racks holding at least 6 trays of fish

  The table’s $42$ racks are a subset of the $64$ selected racks; the proportionTwo ratios set equal to each other, which you can solve for a missing number.Read the lesson in the question also includes the results for the other $22$ selected racks.

• B

  The 96 removable drying racks at Nilsen’s fish smokehouse

  Correct answer

  Bianca randomly selected the $64$ racks used to find the proportionTwo ratios set equal to each other, which you can solve for a missing number.Read the lesson from the $96$ removable drying racks at Nilsen’s fish smokehouse. Random sampling supports generalizing the proportion to that population.

• C

  All 144 drying racks used at Nilsen’s fish smokehouse during inspection

  This extends the results to fixed racks, which had no chance of being selected. The sampling pool included only removable racks.

• D

  The 64 drying racks randomly selected for inspection at Nilsen’s smokehouse

  This limits the results to the $64$ racks in the inspected sample. Because those racks were randomly selected, the results can be generalized to the population of removable racks from which they were drawn.

Solution

Bianca randomly selected $64$ racks from the smokehouse’s $96$ removable racks, so those $96$ racks form the population the sample can represent.

The table shows that $21$ of the $42$ selected racks holding at least $6$ trays had space for another tray. The passage reports that $10$ of the other $22$ selected racks did too. Together, these results give a proportionTwo ratios set equal to each other, which you can solve for a missing number.Read the lesson of $31/64$ for all selected racks. Fixed racks were excluded from selection, so the largest population to which this proportion can be generalized is the $96$ removable drying racks at Nilsen’s fish smokehouse.

Report a problem

Some studies just watch what people already do. Others give each person a treatment, which is whatever the researchers decide for that person, such as a desk type or a study method. Only the second kind can show cause, and only if the researchers decided at random who got what.

An office asks for volunteers and gets $40$. A researcher randomly assigns $20$ of them to standing desks and $20$ to seated desks. After $8$ weeks, the standing-desk group reports less back pain.

Because chance decided who got which desk, the two groups should be about the same in every other way, such as age and how much they exercise. That leaves the desk to explain the difference.

Now a study that only watches.

A researcher randomly selects $200$ of the $2{,}000$ workers at one company. Workers who already use standing desks report less back pain than workers who use seated desks.

Here the workers picked their own desks. The ones who chose standing desks may also exercise more, and that could explain the difference. Or it could run the other way, if people with less back pain find standing more comfortable. So the most you can say is that the two things are associated: “At this company, using a standing desk is associated with less back pain.” “Tends to” works too.

Random sampling and random assignment do separate jobs, and a study can have any mix of the two.

A researcher randomly selects $90$ of the $1{,}500$ students at one high school. She records how many hours each one sleeps on school nights and each one's average on first-period quizzes. Students who slept more tended to have higher quiz averages. Nobody was told how much to sleep.

Which of the following is the most appropriate conclusion based on the study?

• A. At this school, sleeping more causes higher quiz averages.
• B. Among all high school students in the state, sleeping more is associated with higher quiz averages.
• C. At this school, sleeping more is associated with higher grades in every class.
• D. At this school, sleeping more is associated with higher first-period quiz averages.

Check the verb first, then the group, then what was measured.

• A says “causes,” but nobody was assigned an amount of sleep.
• B names every high school in the state, and the sample came from one school.
• C covers every class, and the study measured only first-period quizzes.
• D passes all three checks.

B and C both say “associated with” and are still wrong, so the right verb isn't enough on its own. Watch for the opposite mistake too. A choice saying the results apply only to the $90$ students, because nothing was assigned, mixes up the two jobs. Not assigning rules out cause, and the random sample still lets the results reach all $1{,}500$ students.

For each question, say the group the sample was picked from, then check whether anyone was assigned a treatment at random.