« Logical or Connectionist AI? | Main | Whither OB? »

November 17, 2008

Comments

So why does Google blog search finds zero mentions of this refutation?

Um, because of confirmation bias? I don't think one has to bring in any other theories to explain the zero references.

The 'zero mentions' link includes the following:

__Is That Juice On Your Face?__
**19 Nov 2007** by Russell Ball
You should also check out "Skilled or Unskilled, but Still Unaware of It" which suggests that 'competent' people can be victim to the same effect as the difficulty of the task increases. Left by Rob Crowther on Nov 30, 2007 1:11 PM ...
Caffeinated Coder - http://geekswithblogs.net/coredump/Default.aspx - References


Perhaps google's bot is smart enough to parse OB and realized it should look a bit harder?

Now that I've read the findings, how do I use the information to my advantage? Perhaps find specific quantifiable methods for evaluating task performance and use them to recalibrate metacognitive monitoring.

"My guess: because under this theory you should listen to those you disagree with instead of writing them off as idiots." - yeah - I am sure people would use it this way. But actually it is a double edged sword - it does not give anyone any assurance that they are not on the other side (i.e. that it is them who is unskilled and unaware).

Looks like more support for opening a prediction market on all your projects - because you're going to underestimate how long they'll take, how hard they are, and how good your team is.

How does this result argue for "listening to those you disagree with instead of writing them off as idiots?" It sounds like it argues for doubting your positions, but doesn't tell you how to learn more. And even if your own evaluations are more suspect than you'd like, they're still all you have to go on. Not trusting your own judgment leaves you exposed to the din without any indication where to start.

I think one key thing the original paper mentioned was how people's self-assessed percentile ratings changed when they saw other people's performances: people with good test scores raised their self-assessments to a much more accurate level, while people with poor test scores didn't change their assessments much.

"What happens when you get feedback" was not addressed by the rebuttal paper (which is probably correct about the "noise plus bias" hypothesis), and only marginally addressed by the original paper.

Trying to evaluate your own intellectual performance is a waste of time, because the only way you can do so is by using your intellect. That leads to massive problems with circularity.

As the saying goes, the easiest person to fool is yourself.

The only solution I am aware of is to find an objective measure of correctness, determine how that method rules on your position, and use the feedback to improve your thoughts. Repeat as necessary. (It's always necessary.)

This solution is otherwise known as 'science'.

Cameron, that ref still doesn't appear when I click the link.

Zbigniew, yup.

Doug, yes but that sort of test situation is uncommon.

Chris, a theory where idiots are more common has more support for the hypothesis that the specific person you are talking with is an idiot. And central to the concept of an idiot is that they aren't worth talking to.

Robin: Fascinating. Still appearing as result 6 of 9 for me, now overshadowed by the several new additions that your own post encouraged. Perhaps personalised search history makes google searching even more subjective than it once was!

* And central to the concept of an idiot is that they aren't worth talking to.

Are my human biasses leading me to conclude people are not worth talking to (idiots) when in fact I would benefit from such conversations? This particular bias suggest that there probably are more instances than I believe.

More generally, however, I find I give the label 'idiot' not so much to people who lack skills but to those who reason less abstractly than I and are unwilling or unable to cross that gap when necessary. In those cases attempting to 'reason' leads to conflict independently of any actual difference in stimulus -> response pairings of the respective models. In those situations people can be legitimately 'idiots who aren't worth reasoning with' yet simultaniously worth observing, talking to in a non-reasoning format and learning from.

I'll put it another way. It is easy to use 'excuses' like the earlier findings of "Unskilled and Unaware of It". It is also easy to say "ha! I'm Biassed! I am just finding excuses to call people idiots and not talk to them! Reverse that conclusion!". Yet it is far more difficult to arrive at an unbiassed evaluation of which people are in fact worth talking to, independently of social signalling and ego.

Just as I cannot be sure I am smarter than my ethics, I cannot be certain that I'm smarter than my biasses either. For all I know my biasses encourage me to make excuses not to talk to idiots because I'm just too dumb to comprehend the real reason that they're a waste of my time. Uncovering the bias with the cause for the bias may be counter productive.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31