Recently, there has been some discussion about The Wisdom of Crowds and Wikipedia. There’s this article which is commented by a blog, which in turn is commented by another blog, and so on. People have interesting opinions on this phenomenon, but instead of jumping into the fire, I thought I’d offer a slightly amusing anecdote.

I’ll take the easy way out and copy a brief description of the WoC idea: “The Wisdom of Crowds is a book written by James Surowiecki about the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group.” Short and to the point. Ironically, the quote is from a Wikipedia article about WoC; feel free to go there to read more about the book.

So what we have is the idea that a group of people, given certain criteria, can give better estimations than any single expert in a field. Some people have found anecdotal evidence to support this…but of course I have to be a cynic and offer counter-evidence.

A few months ago at work we started up version 1.2.6 of our product, and this time we were going to go about it in a strict professional manner. No ad hoc management like with 1.2.5, and proper time/risk/budget estimations were to be made. “Oh well, how hard can it be,” everyone thought. “We have our Way of Working guidelines that we have to follow, so everyone should be happy as long as we do just that.” One of the things specified is how to perform a proper time estimation. It seems that the people in charge of our guidelines have read The Wisdom of Crowds, but apparently not close enough.

Imagine this situation: there’s this room full of people. Everyone from test, development, management, documentation and administration is there. For every module and component of 1.2.6, everyone in the room gives an estimation of how long it would take to implement and test it in best case and worst case, as well as an estimation of the probable time it will take.

I have seldom seen so many clever people waste their time on something as retarded as this.

First of all: no one outside the development team has any idea how long something will take to implement. That’s the first problem. Considering that the developers were probably less than 25% of the assembled people, it’s completely ridiculous to even think about estimating the project like this.

Secondly: I assume that crowd wisdom is what was intended to be the effect of this exercise, but the ones writing the guidelines seem to have forgotten something. There are a few key criteria that separate wise crowds from irrational ones:

• Diversity of opinion
• Independence
• Decentralization
• Aggregation

Sure, I agree that the aggregation point (a mechanism for turning private judgments into a collective one) is present, as well as decentralization (people can use their local knowledge in making the decision). However, what about independence and diversity of opinion? How can anyone honestly believe that everyone will have an opinion about details that don’t directly concern their line of work – and, even more improbably, base that decision on personal opinion instead of just following the others? It’s ludicrous to even think that anything other than herd behaviour can be the result of an estimation like this!

The most interesting aspect of it all must have been that we had the product owner present as well. What’s his main wish? That everything costs as little as possible, of course. So what does he do? He sometimes deliberately mentions low time estimations in order to drag down the average. Sigh.

…What was the result of all this, you ask?

Out of five planned increments, two were wholly completed. Let’s just say that we probably won’t see as many people present at the table for the 1.3 estimation.

This entry was posted on Monday, July 17th, 2006 at 1:36 pm and is filed under Software Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.