Paired multiple-choice questions reveal students' incomplete statistical thinking about variation during data analysis

Biologists consider variability during biological investigations. A robust quantitative understanding of variability is particularly important during data analysis, where statistics are used to quantify variation and draw conclusions about phenomena while accounting for variation. Many students struggle to correctly apply a quantitative understanding of variation to statistically analyze data. We present quantitative and qualitative analyses of introductory biology students' responses on two pairs of multiple-choice questions querying two concepts related to the quantitative analysis of variation. More students correctly identify a mathematical expression of variation than correctly interpret it. Many students correctly interpret a non-significant p-value in the context of a very small sample size, but fewer students do so in the context of a large sample size. These results imply that many students have an incomplete quantitative understanding of variation. These findings suggest that instruction focusing on conceptual understanding, not procedural problem solving, may elevate students' quantitative understanding of variation.