I’ll Have a Dose of Confirmation Bias, Heavy on the Bias

Matthew Ladner:

So how do private school students do in Science compared to public school students. I wasn’t sure, so I went to the NAEP data explorer to find out.

Private school students outscore public school students, but private school students tend to be more affluent than public school students, and there can be differences in special need and language profiles. Fortunately the NAEP data explorer allows you to take such factors into account. To maximize the comparison, we will only look at the NAEP science scores of children eligible for a Free or Reduced priced lunch under federal guidelines, and who have neither a special education nor an English Language Learner designation. This is about as close to apples to apples comparison you can hope for in NAEP data.

So NAEP changed the framework of their Science exam in 2009, making the 2009 and later exams incomparable to those given before 2009. The comparison of general education poor children between public and private schools is sporadically available in both NAEP science frameworks. You can’t compare old NAEP science to new NAEP science, but you can compare public and private school scores within each year. So let’s start with 4th grade:

……

So for those of you scoring at home, in 8 possible comparisons, private school general education poor children outscored six times. It was close (within the margin of sampling error) a few times but every time the result was lopsided it was lopsided in favor of the private school children. Quite frankly science scores should be higher in both public and private schools for low=income kids, but the available evidence does show an overall private school advantage. Unless you happen to be Stephanie Simon working through a sizable case of confirmation bias, in which case this is what you saw: