The insidious thing about tests is that they seem so straightforward. I write a bunch of questions. My students try to answer them. And so I find out who knows more and who knows less.
But if you have even a minimal knowledge of the field of psychometrics, you know that things are not so simple.
And if you lack that minimal knowledge, Howard Wainer would like a word with you.
Wainer is a psychometrician who spent many years at the Educational Testing Service and now works at the National Board of Medical Examiners. He describes himself as the kind of guy who shouts back at the television when he sees something to do with standardized testing that he regards as foolish. These one-way shouting matches occur with some regularity, and Wainer decided to record his thoughts more formally.
One thought on “How to Abuse Standardized Tests”
Comments are closed.
I’m often up for a lay book on psychometric application. There’re several that do a good job of explaining in practical terms the pros and cons and misapplications. This book by Wainer is not one of them, and continues the charade. Typical of such charades, he often tells the truth and lies at the same time. He shows the flaws of his arguments, but like the small print in consumer contracts which nobody reads, he hopes and knows you won’t catch him on it.
I’ve only read the first chapter which discusses SAT tests, ETS’s big money maker. He concludes, of course, that SAT scores are and must be required and determinative for acceptance into college. He proves that, only if you have no knowledge and miss the tricks he is performing.
The topic of the first chapter is the comparison of the SAT scores of kids who submit the scores to Bowdoin College vs the SAT scores of those who don’t. The SAT is optional at Bowdoin. Not surprising, ON AVERAGE, those not submitting score significantly lower. Now to his proof. Those not submitting have a 0.2 lower first year GPA than those submitting, ON AVERAGE.
He concludes that if the goal of college admissions is to choose kids who do better in their college courses, then you must use the SAT scores to make admission decisions.
I’ll bet you missed the truth and the disconnect between it and his conclusion. First, the SAT scores on average differed significantly, but the GPA differences were only 0.2 points. Second, this comparison and prediction was for first year college; if the SAT has any predictive power, one would expect the correlation to be strongest first year. That is the key — after first year, the predictive power of the SAT of college success is insignificant.
A. Ericsson’s and others work on “deliberate practice” shows the insignificance of tests like the SAT in predicting success. In these studies, the correlation between such tests and success is about 0.2 which translates to saying that 4% of the variation is due to whatever is being measured by these tests, 96% of ones success is based on other factors.
Another almost laughable con Wainer uses in his first chapter is almost a direct steal of the cons illustrated in the seminal book “How to Lie with Statistics”. To illustrate the differences between SAT submitters and non-submitters, he plots frequency distributions of the SAT and separately the GPA scores. This is legitimate. But to further his big lie, the vertical axis of his plots use the actual frequencies of submitters and non-submitters, not the percentages, which would have been the honest and accurate way to display the results. Because the vast majority of kids submitted SAT scores, their frequency distribution plots towers over the vastly smaller frequency distribution plot of the non-submitters.
So instead of seeing two equal area distributions side by side with the non-submitter distribution a little off to the left, showing lower SAT scores or lower first year GPA scores, with the distributions overlapping significantly, one sees the non-submitter distribution, a tiny insignificant plot, occupying almost no area, to the lower left, and this overwhelmingly large area plot representing the submitters. A parent vs child image.
My bet is you’ll have the unwashed masses of conservatives drooling over this obvious con as though it were the Second Coming, and liberals, who couldn’t make a substantive argument if their lives depended on it, decrying the results and perhaps making some effort at protest.
But the truth is this book is an obvious con, if the first chapter is any indication.