The media and business worlds love experts with strong opinions and the ability to explain them confidently and authoritatively. We hear them on TV and read them in newspapers and online every day. Yet a 20-year study shows that these are the very people who are least likely to be accurate in predicting what will happen in the future.
In 1984, Philip Tetlock commenced a study to examine the accuracy of expert predictions. He found these experts fell into two main groups that he called foxes, who know many things, and hedgehogs who know one big thing. Foxes draw on many ideas and sources of information and are quite tolerant of uncertainty and ambiguity. Hedgehogs tend to interpret the world using their favourite theory or dogma and are very confident in the ‘rightness’ of their view of the world.[Tschoegl et. al. 2007]
Tetlock, a psychologist, is Professor of Leadership at the Haas School of Business at the University of California, Berkeley. His research found that, in the main, experts were no more accurate in their predictions than ‘a monkey throwing darts’. But he clearly showed that foxes produce much more accurate forecasts than hedgehogs. He also found that when faced with their erroneous forecasts, foxes tended to acknowledge their error and adjust their thinking. Hedgehogs, on the other hand, rationalise their errors away on the basis that they were ‘nearly right’ or ‘unpredictable events interfered in the outcome’ or by pointing to the few occasions when they had got it right.
The implications of this research are many, but one that that reinforces my own experience is to be wary of highly confident people proclaiming a view and running down those with alternative perspectives.
It’s my view that Tetlock’s findings are also relevant in our organisations. We love to treated complex (wicked) problems as if they are technical problems that can be predicted and solved. The experts who stridently proclaim their opinions as being facts are often wrong. The people who stride the corridors and make the most noise are not necessarily the stars. The leaders who are most confident they are excellent at leading people are often the worst leaders.
In media and in organisations, the hedgehogs get the airtime. But the foxes are the ones to listen to.
- Tetlock, P. (2005): Expert Political Judgement: How good is it? How Can We Know?, Princeton University Press.
- Gardner, D. (2011): Future babble: Why Expert Predictions are Next to Worthless, and You can do better, Dutton, London.
- Tschoegl, A.E and Armstrong, S., Review of Philip E. Tetlock: (2007): “Expert political judgment: How good is it? How can we know?” in International Journal of Forecasting, Volume 23, Issue 2, 2007, pages 339-342
Filed in Changing behaviour, Expertise location, Knowledge
I started to read Daniel Coyle’s latest book: The Little Book of Talent: 52 Tips for improving your skills on the commute to work this morning, and it’s fascinating.
I really enjoyed Coyle’s previous book The Talent Code: Greatness Isn’t Born. It’s Grown. Here’s How. The book was all about answering the question; “how do people get great at something?“.
For the book, Coyle looked at the latest research, spent a lot of time talking to the ‘father of expertise’ K. Anders Ericsson, but also visited what he called ‘talent hotbeds’. These are places where great talent has been produced out of proportion to their size and perceived stature; for example, a Russian tennis club, a music school in Dallas, a soccer field in Brazil, and many others.
It is a fascinating read and brings out, yet again, the importance of deliberate practice, a concept you may have seen us mention on many occasions in this blog. From cab drivers in London doing the knowledge, to Benjamin Frankiln improving the way he wrote, to the Jamacian bob sleigh team, immortalised in the film ‘Cool Runnings’. They all used elements of deliberate practice in building skill and improving performance.
A key element of deliberate practice is the presence of some kind of coach, teacher or mentor to help provide guidance and give feedback on performance. We all know the value of a great coach, but what is the right kind of person to help us really achieve something great?
From Coyle’s research he says you should seek out someone who:
- Doesn’t remind you of a courteous waiter - you don’t want someone that smiles a lot and says things like; “Don’t worry, no problem, we will take care of that later“.
- Scares you a little – look for someone who watches you closely and is honest, sometimes unnervingly so
- Gives short, clear directions - most great teachers/coaches/mentors do not give long winded speeches. Instead, they give short, clear directions of what they want.
- Loves teaching fundamentals – they start from a focus on the basics, the foundations, the fundamentals and build from there. And they will always go back to these to ensure they are being done, and done right.
- Is older (all other things being equal) – teaching is like any other talent: it takes time to grow. Great teachers are first and foremost learners, who improve their skills with each passing year
Canberra has an interesting event this month! actKM has invited Patrick Lambe to conduct a workshop on Leveraging and Valuing Expertise. This workshop is part of the open research project “Leveraging and Valuing Expertise” (http://usingexpertise.com). Log in to share your stories!
When: 9:00 am to 3:30 pm on Friday, 13 February
Where: University House Common Room, ANU
Costs: $50, includes morning tea
* Introduction: the nature of expertise and experience
* Grounding: Anecdote circles with participants exchanging their stories of how expertise is leveraged and used (or misused) in their organisations
* Sensemaking: we work with the stories to identify patterns and key issues in the participants’ situations
* Planning: we work with an expertise transfer framework and the Straits Knowledge KM Method Cards to build outline plans for some of the participants’ situations
* Close: closing discussion looking at general patterns and sharing any relevant case examples
Social scientist, Harry Collins, has spent his career hanging out with gravitational wave physicists and learning to talk like one. Harry’s research is on expertise and working out whether someone who can talk like a physicist can be indistinguishable from some who can do physics if you only talk to them.
The key to the whole thing is whether people have had access to the tacit knowledge of an esoteric area—tacit knowledge is know-how that you can’t express in words. The standard example is knowing how to ride a bike. My view as a sociologist is that expertise is located in more or less specialized social groups. If you want to know what counts as secure knowledge in a field like gravitational wave detection, you have to become part of the social group. Being immersed in the discourse of the specialists is the only way to keep up with what is at the cutting edge.
Harry did a simple test to work out whether we can sound just like an expert.
The original version we did was with color-blind people. What we were attempting to demonstrate is something we call the strong interactional hypothesis: If you have deeply immersed yourself in the talk of an esoteric group—but not immersed yourself in any way in the practices of that group—you will be indistinguishable from somebody who has immersed themself [sic] in both the talk and the practice, in a test which just involves talk.
If that’s the case, then you’re going to speak as fluently as someone who has been engaged in the practices. And if you can speak as fluently, then you’re indistinguishable from an expert. It’s what I like to call “walking the talk” [I think he means talk the walk because in my book walking the talk means you can do what you say]. You still can’t do the stuff, but you can make judgments, inferences and so on, which are on a par. We picked color-blind people because they’ve spent their whole lives immersed in a community talking about color. So we thought color-blind people should be indistinguishable from color-perceivers when asked questions by a color-perceiver who knew what was going on. And we demonstrated that that was in fact the case.
I guess this means that the only way to determine someone has real expertise is to see them in action. This simple point is particularly important in light of the problems Australia is facing when some overseas doctors are gaining their Australian credentials and then patients discover their incompetence [the latest example from Melbourne]. But this approach is not going to be easy for every type of job. Think about those jobs that involve the application of subtle judgement where the outcome remains unknown for years (and tracing the outcome to the decision is impossible)—I’m thinking of policy makers, engineers, leaders. In these cases we have to rely on stories of experience told by people we trust.
Interview from Scientific American
via Mind Hacks
Last week I was running an open space to kick off a new community of practice for engineers. While I was wandering around the room I overheard one of the participants make this point about self rating your expertise.
The guy who has done this job for 20 years rates himself as good. But the guy doing it for two years rates themselves as expert. They don’t know what they don’t know.
Just watched Malcolm Gladwell give a talk to the New Yorker Conference—2012: Stories From the Near Future (lots of interesting videos to watch). The topic of Malcolm’s talk is ‘genius’ and he contrasts two extraordinary men: Michael Ventris, who deciphered the ancient Mycenaean script know as Linear B, and Andrew Wiles, the mathematician who developed a proof for Fermat’s Last Theorem (If you are interested to learn about the story of how Wiles accomplished his proof I recommend you read Simon Singh’s Fermat’s Last Theorem).
Gladwell makes two good points in his talk:
- persistence and collaboration might be more important personal traits than lone genius in a complex and changing world; and
- a person needs to invest 10,000 hours of concentrated and reflective practice to achieve mastery—this amounts to about 10 years.
I was also impressed with how Gladwell told his stories from the point of view of the level of detail he provides—i.e., lots. He’s not an emotional storyteller but one who is effective in sparking interest in an intellectual idea.
I’ve been having a blast the last couple of days. I signed up for last.fm after hearing Euan and Johnnie Moore talk about it. Last.fm keeps a track of the music you listen to (Here you can play the music I’ve been listening to, http://www.last.fm/user/Unorder/) and then you can hear a bunch of recommendations streamed directly to you. It’s just like listening to the radio without commercials or radio announcers. There is a heap of other connections you can make, such as finding the people who listen to similar music etc.
Technorati Tags: last.fm
This looks really interesting:
Expertise is about more than evidence. It is also about judgement and wisdom. Our argument is not that we should reject the received wisdom in favour of the wisdom of crowds. But we need to go beyond a simple model of ‘evidence-based policy.’ Drawing on recent case studies and research with ‘lay members’ of expert committees, this pamphlet looks to a new model of expertise which is more diverse, takes better account of uncertainty, is aware of its context and trusts the public.
The pamphlet is 87 pages (down-loadable pdf) in the style, I guess, of the polemics of the 18th and 19th century. But perhaps less controversial. The work is available under a creative commons licence and I will be having a good read.
You can find the background to the pamphlet here, which says “The good folk of Defra have asked Demos and Liverpool University to consider how lay people can play a part in expert scientific advice.”
Stephen Colbert, the comedian who brought us the devastating roast of George W. Bush at the 2006 White House Correspondents Dinner, has outlined how you can be an expert in anything. Good advice for people wishing to fine tune their bullshit detectors. Here are Stephen’s 6 headings. Check out his article in Wired for the detail instructions.
- Pick a field that can’t be verified.
- Choose a subject that’s actually secret.
- Get your own entry in an encyclopedia.
- Use the word zeitgeist as often as possible.
- Be sure to use lots of abbreviations and acronyms.
- Speak from the balls, not from the diaphragm.
[thanks to Les Posen for the link]
I remember a great story told by Margaret Wheatley about how the US Federal Aviation Authority successfully landed all the planes in US airspace on September 11. I was searching around for it today and found it. Here is it:
Later, they realized that the reason they succeeded was the strength of their relationships. They trusted each other as they were communicating across the country. There was a real esprit décor; they were smart. They could make new policies. They could make up rules that worked in the moment. So after Sept. 11, as any good organization would do, the FAA wanted to learn why this had worked so well. But of course, being a federal agency, they wanted to learn what worked so they could put it into a rulebook. After its research, the FAA did something extraordinarily brave. They decided not to write a rulebook about the incident; they understood that what had made it work was people’s intelligence, dedication, and relationships. That’s a lesson we all need to learn right now. The only way through an uncertain time is to have a certainty about your values, your purpose, and a certainty about each other. We call it trust, but it’s even more than that. It’s knowing, as my friend’s daughter who plays rugby says, “When you’re moving a ball down the field, you can’t see the people right behind you, but you may need to pass the ball to them, so they just keep signaling to you and they just keep staying with you, with you, with you.”