« Dreams of AI Design | Main | Against Modal Logics »

August 27, 2008


Robin, why are you going here from some/many to all? If we want to learn how to overcome bias better, shouldn't we be interested in the minority that behaves more rationally, rather than focus solely on the majority (or even near unanimity) that behaves less rationally?

Isn't there a proven correlation between teacher IQ and educational outcomes?

The finding that there is little to no correlation between educational outcome and teacher qualifications is actually not new (http://www.theaustralian.news.com.au/story/0,20867,21765738-2702,00.html) but I do find it a bit surprising: after all, if teachers' IQ's correlate with outcomes, and higher IQ teachers are better at passing qualification tests (i.e. qualifications correlate with IQ), then qualifications should correlate with outcomes.

Does it mean that teachers' qualifications (master's, etc.) are not correlated with IQ? It's all about sitting in class and not falling asleep?


Studies have consistently shown that graduate coursework (e.g., a Master's degree) does not affect teacher productivity.

Yes, I can easily believe this. Some of the worst university professors I've had were very successful in their chosen field. The ability to teach is quite a skill in its own right.

However, regarding high school teachers, I guess it depends on just how bad some of the teachers in question are. For example, my sister when she was 14 had a mathematics teacher that didn't know how to expand the equation x(x+1). Having a graduate degree (or any degree) might not make somebody a good teacher, but at least of they do have such a degree they should have an elementary understanding of what it is that they have to teach.

Perhaps a simpler solution is the one I proposed at the time: When the students sit the national exams at the end of the school year (which happens in New Zealand), once every five years the teacher has to sit the exam alongside the students. If they don't get a decent passing grade, they shouldn't be allowed to teach the subject at that level. It's not a large cost on the teacher, presumably they don't need to study for the exam if they teach the subject so it's only 3 hours of their time every 5 years. Of course the potential for embarrassment with teachers failing subjects will mean this idea is unlike to ever be implemented...

"Studies have consistently shown that graduate coursework (e.g., a Master's degree) does not affect teacher productivity"

Does this refer to a Master's in Education or a Master's in the field in which the teacher instructs?

Hopefully, you lost me.

Rafal, have a cite for this "proven correlation"?

Considering this article and the one on medicine, what does this mean for science and/or other areas (ex: MBA programs) where top schools supposedly graduate "better" or more qualified individuals?

Most scientific papers are probably wrong
The End of Business Schools? Less Success Than Meets the Eye

I think there is a dichotomy between the reality of individual drive, effort, and talent, and the notion that society can construct institutions that produce the results of those qualities. In my field (software engineering), the top performers by ALL metrics are those who possess a personal, internal passion for the subject material, combined with a brain that is suited to it. I don't detect any correlation at all between those things, and being university-taught or self-taught. That also goes for so-called "academic" knowledge of computer science and information theory, contrary to the common stereotype.

In my opinion, it should be obvious that individuals following an internal drive to consume knowledge are going to outperform those who merely shuffle through a system that is supposed to feed them that knowledge. Such individuals will take advantage of whatever learning resources are available, whether it is a university or the public library, and the quality of teachers won't matter very much because they are learning, as opposed to being taught.

My guess: teachers without full certification are much less likely to get promoted/avoid the sack unless they're actually good at teaching, so that population is more pressured for and selected for competence - enough to cancel out the effect of the extra education and the likely higher IQ that correlates with that.

Also, what Andy said :)

private school students of fully certified 12th grade math and science teachers do not appear to outperform students of private school teachers who are not fully certified.

I'm with Andy.

Since private schools usually have admissions criteria (beyond the obvious financial one), aren't they more likely to have a population of students with higher scores (IQ's?), supposedly more capable of learning, than the unselected student population? Shouldn't the conclusion of this article be more about the students than the teachers? Something like, "students with high potential, selected for equal motivation, have the same outcomes, regardless of their teachers?" In a comment yesterday on the "Top Docs" post, I mentioned that the top-ranking urology residency program in the US in 1978 was a community hospital with one urology resident and no formal teaching at all. "The quality of the residency depends on the quality of the resident" is the cliche in medcine.

Indeed, on this blog, there are brilliantly educated contributors who apparently have no formal education at all. Perhaps this is the way to go, when the student is perceived to be smarter than the teacher.

I permanently lost all respect for the school system in second grade, when my "math teacher" didn't know what a logarithm was. As a result, I... never relied on the school system to teach me anything, and studied on my own... hm, I think I see the problem...

The plural of anecdote is not data, but my experience does suggest that a major factor in how well a student does, is how little they expect they expect the school system to help them. I've had this with tutorees - having to tell them, "No, you can't expect the college to teach you! If they can help, that's great, and unexpected, but if you want to come out of there with knowledge, you've got to take responsibility on your own." So one possible mechanism - though the plural of anecdote is not data - is that any increase in teacher effort is met by a student decrease in effort. I don't really believe this, but it would be worth looking into if someone wanted to find a mechanism.

Where does enjoyment (less pain) factor in?

If the students/patients have no noticibly different results, perhaps there is a difference in getting there. If teachers X and Y can both get you to result A, but teacher X can get you there with considerably less pain (e.g. more interesting lectures, side topics), why not pay more for teacher X? The same story can be told for doctors. Is there any data on student or patient evaluations of pleasure/enjoyment (or, less pain) associated with these ends?

Vote of hands here, does *anyone* expect that teaching certifications correlate non-trivially with teaching quality? I personally wouldn't even expect teaching quality to produce more than 20% of the variance in educational outcomes given large (>10 students? >5students, certainly I don't think 20 vs 40 matters, 20 is already a lecture) class sizes, and 10% wouldn't surprise me. I would be SHOCKED if teaching credentials explained as much as 20% of teaching quality variance, and 10% still sounds too high. Need a large sample size to detect a statistically but not practically significant effect given my priors, and that's before considering the selection out of less capable non-certified teachers. Still, I'm glad to have a study to cite in confirmation of the inefficacy of teaching certifications, or even somewhat less plausible things like homeopathy.

Also, GREAT COMMENTS everyone. Thanks!

Interesting point Eliezer. It seems applicable to me in that I only lost confidence in the system gradually as the result of such things and thus learned much less than I could have. Sincerely doubt the teacher effort > low student effort effect though. Of course, what I really should have been taught (based on my experiences Penn State, Florida State, Drexel, Penn and Columbia) is that you MUST get into a top school with a top program in your undergrad major if you want effort spent on actually learning the material to contribute to your academic performance. At all the programs but Columbia neuroscience/psychology/biology my experience was that the ONLY way to get good grades in most classes was to play the system and any learning you did was strictly recreational and had to be done on your own time.

BTW all, since no-one else has mentioned it, the obvious way to find out if teachers can have an impact is NOT to test if teacher certification correlate with outcomes but to look at the outcomes for a few famous teachers and see if their students SES or some other characteristics are collectively statistically anomalous.


I think teachers can absolutely have a huge impact on learning. Maybe this won't show up in test scores or certifications (I don't know), but some are simply way better teachers than others. I do think you need quality students to make a quality teacher worthwhile, of course.

Where I went to school, there were some CIS and EEL classes that were nearly full-time jobs in and of themselves. The profs forced you to show them that you knew the material inside and out or you didn't pass. However, most students took these same classes with other profs who were much, much easier. Some of the students who take the same course with a harder prof would have learned more anyway, but without being forced to do so, most wouldn't.

I still think the hard classes were a waste of time compared to any sort of inter or apprenticeship, but they beat the hell out of all the other courses I took.

Robin, so whats the solution? How do we go about reducing information asymmetries if we can't rely on certifications at all? Surely certifications, reputations and other competence-related signals provide useful information in other fields? I would find it hard to believe that those things just signal status and nothing else.

Eliezer: My own experience (good schools and lack of motivation) makes that sound very plausible, but then that could just be me trying to shift blame.

Teaching quality does not correlate with certification because in so many cases undergraduate preparation for teaching degree is inadequate. For over twenty years as a superintendent of schools I have been interviewing all finalists for teaching positions. When I interview the teachers I am continually disappointed in the preparation that they have received as part of their undergraduate training. So often, the school districts are responsible for really teaching teachers how to teach. One example is the lack of coursework on integrating technology at the undergraduate level in teacher preparation programs. Couple of

When I took Caroline Hoxby's economics of education class, we learned a lot about high school teacher certification. We learned that it may be that certification actually has a negative effect on education quality. The opportunity cost of spending the extra year(s) gaining this certification (without necessarily gaining any real skill) drives high-aptitude potential teachers into other fields. In addition, the spurious signal value of the extra degree may attract lower aptitude people because they can gain extra salary after attaining the degree.

Having just finished the masters program at GMU, I would like to say that I learned very close to nothing that I will use in actual teaching. The only productive aspect was a student teaching internship, where I co-taught with an experienced teacher. Honestly, there was a lot of Myers-Briggs type indicators and Lev Vygotsky, but nothing of any practical value. I hope none of my professors read this site (some of your classes were at least interesting, and, if you read this blog, you are probably part of that "some".)

If anything, the MA in education is a signal of commitment.

Also, if we are going to rant against teachers/the educational system, you would not believe the amount of self-serving bias among teachers; the bile they spew against man's 23rd greatest achievement, wikipedia, sheesh, not to mention the anti-market bias.

There are so many biases among educators, such as seeing EVERYTHING as a problem of ignorance that can be solved with, wait for it, education (fellow teachers claim that we simply need to teach "ethics", which they, of course, find themselves capable of doing). I once tried to make the point that educators are generally going to be biased toward overestimating the benefits of education, and that, if students don't feel an assignment is worth their time and effort, we should seriously consider shifting our priors in that direction (I didn't say priors). I literally thought I was going to be physically attacked.

Teaching, like medicine or law, is just another guild. Grades, class rank and a school's reputation are just proxies for merit assessment of newly minted guild members. Licensure is just a barrier to entry to protect the guild's monopoly.

If teaching and medicine is anything like law then similarly once you're licensed and start practicing you find that much of what you were tested on is useless or obsolete at best and wrong at worst.

For example, a typical bar exam question will read something like this: "Your client climbed through an open window into his neighbor's house. He pulled the sofa into the center of the the room and set it on fire with a match. Then he left. The sofa fire burned for a while and then went out on its own. The ceiling was singed. Is your client potentially guilty of A) Arson; B) Burglary; C) Breaking and Entering; or D) None of the above?" It's our wtf??? moment when we realize that the whole education and licensing process serves no real purpose other than to artifically depress the supply of lawyers so as to maintain, if not heighten, the demand for those able to survive the hazing.

Once you get into the courtroom neither your pedigree nor your GPA count one whit. I suspect the same may be true of other guilds.

BTW, Ray Stasco, PhD knew the secret behind effecive teaching: first you must get the student's attention.

The paper seems interesting, thanks for bringing it to our attention, but from only the first 3 pages it seems badly paraphrased, focusing not on informing us but highlighting your favorite talking point.

Things I picked up:
* a common rationale is to ensure a minimum level of quality (as often with licensing)
* some people have a beef with TFA
* states may have quirky testing systems
* teachers who reach the profession by alternative means may be systematically different [<-- my first thought]

If there is a blogger (a "Rob" to Tyler's Tyrone maybe?) who could have summarized the last 10 pages and their findings on the above as well as Robin's universal thesis of counterproductive sincerity-signaling, I sincerely would have rather heard from him.

Of course, maybe I could have finished the paper in the time it took to selfishly snark about someone else's goodwill labor.

Robin, I doubt ALL people would still prefer a more credentialed teacher over a less credentialed teacher, or a more credentialed doctor over a less credentialed doctor, if the best empiricism indicated no benefit in doing so. So where's the interest in seeing what is different about these two sets of people? Instead, you write:

"I expect patients are willing to pay more for top med school docs, and parents are willing to pay more for educated and certified teachers. And I expect that this would continue even if patients and parents knew the above results. I suspect most of the demand for teachers, doctors, and many other professionals comes from folks wanting to affiliate with certified-as-impressive people. And merely making patients healthier or making students perform better doesn't count much toward impressiveness, relative to academia-certified impressiveness.

But folks don't like to admit this directly; they'd rather pretend they care more than they do about other outputs. Which is why folks don't want to hear about the above results. The media will oblige them, and so they will continue in their preferred delusions. Bet on it."

That's where you went from what many people might do, to a claim that all people will do this. Or at least a seeming disinterest in what's different about people that don't make doctor or teacher selections based on academia-certified impressiveness if there's no tangible health or education gain.

Are still lost about the criticism in my earlier comment?

Hopefully, I tire of adding disclaimers to every sentence. If I start out saying "most of the demand" I figure I can later just say "folks don't like" without it being assumed I mean absolutely everyone absolutely all of the time.

I think what I'm pointing out is less a place to add disclaimers to avoid gotchas (I find them annoying too) than an area that would seem to be of central concern for someone interested in overcoming bias. If a group of people are successfully overcoming bias in particular situation, I think it's a very interesting question why they're overcoming that bias -what's different about them.

I'm surprised that you don't seem to have that interest here (or in a variety of other similar topics).

Tyler:Tyrone :: Robin:Ramone

Rafal Smigrodzki : if teachers' IQ's correlate with outcomes, and higher IQ teachers are better at passing qualification tests (i.e. qualifications correlate with IQ), then qualifications should correlate with outcomes.

Correlation is not transitive.

Richard Kennaway wrote: "Correlation is not transitive"

Indeed, but I was under the impression that IQ and scholarly qualifications are both measures of a single quality, and the absence of correlation between educational outcomes and teacher qualifications would in this situations imply they are not.


Rafal wrote:
Richard Kennaway wrote: "Correlation is not transitive"

Indeed, but I was under the impression that IQ and scholarly qualifications are both measures of a single quality

What do you mean by both being measures of a single quality? Their correlation surely falls well short of 1 (virtually all correlations reported in social science and psychology do), and therefore either could correlate positively with educational outcomes while the other correlates not at all or negatively.

Vote of hands here, does *anyone* expect that teaching certifications correlate non-trivially with teaching quality?

Not I. It would be great if there were graduate programs that could help a person become a more effective teacher, but the existing programs strike me as mostly worthless. I have no direct knowledge or experience of them, but the people whose judgement I trust almost universally hold them essentially worthless.

Eliezer tutors people? Where do I sign up?

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30