It’s easy to find policy-theory enthusiasts, such as @slatestarscratchpad, who make a habit of using standardized test scores as a proxy for school quality / teacher quality / education quality generally.

It’s also easy to find people talking about how standardized test scores aren’t the be-all and end-all of educational value, or even about how they’re basically meaningless. 

But you don’t often see people suggesting that, if you’re trying to measure the impact of a school or a teacher on a student’s future prospects, standardized tests might be literally the worst remotely-relevant metric you could choose.

At least in America, almost all the high-stakes tests measure one of two things:

(1) Ability to do simple math problems quickly in a high-speed decontextualized environment

(2) Reading comprehension of simple decontextualized passages that must be absorbed and analyzed quickly

Both of these things seem like they’re pretty much proxies for IQ.  Both of these things seem like they will, in fact, correlate with a student’s future success prospects…in the same way that IQ does.  (Speed and dexterity with abstract-reasoning tasks are widely useful.)  Both of these things seem like they pretty much can’t be taught, or at least like the effects of education will be swamped by the effects of natural talent. 

And there are so many other things where education quality can have a huge impact!  At least in theory!  Students can be instilled with curiosity, and a sense that learning new things is likely to be fun and rewarding.  Students can be taught intellectual courage, so that they don’t cringe away from challenges.  Students can be taught the organizational skills and self-management strategies that allow them to break down, and execute, big long-term complicated projects. 

Hell, students can be taught actual content.  Which is important!  They can be trained to recognize the references and tropes that underlie the elite culture of their societies.  They can be given grounding in the essential facts of history and science.  They can, with coaching, develop skills – writing, drawing, programming, you name it. 

All of these things are things that a sensible person would pay a teacher to teach.  All of these things are things where education quality seems like it should make a huge honking difference.

So why exactly do we evaluate teachers and schools based on student improvement on pseudo-IQ tests, where outside impact will be minimal?