After witnessing a string of situations where people proposed what were, to me at least, wildly improbable theories of a problem, I began to question why it was that people kept throwing out these elaborate explanations of seemingly straightforward problems. Upon reflection, I believe it has a lot to do with people's egos, and their desire to be part of something historical. These three factors drive the generation of complex, unlikely theories in favor of simple, probable ones:
- The War Story Factor - You can't constantly retell a story about the time that you had a frustrating off-by-one error, even if the debugging process was arduous and the stakes were high. People only want to hear "fix-it" stories where the solution to the problem either required some seemingly mystical leap of logic, or where the actual underlying cause turned out to be incredibly bizarre and improbable. Thus, in proposing an unlikely hypothesis, we hope that it's true so that we get a new war story to add to our arsenal.
- The Genius Factor - We all want to look smart in front of our peers, and what better way to do so than by proposing a chimerical theory of the problem, only to be proven right in the end! If your theory is wrong, then you shrug your shoulders and move on; it's likely that no one will hold you to account for the hours or days you spent pursuing this theory. In this way, improbable theories have really only an upside for the proposers: you'll look like a genius if you are right, and just another wrong guesser if you're not.
- The Hero Factor - Related to the genius factor is a desire by many software people (and I assume trade and service people of all stripes) to be seen as the savior or hero. You want people to say, "We were really stuck. Thank goodness that you came up with that crazy theory that we hadn't considered. We would still be struggling with this problem!" You get to swoop in and save the day with your fantastic theory.
So we want our problems to be solved, and solved quickly, but we secretly hope they have complex, mind-bending solutions so that we can boost our egos when we solve them. We are rewarded for doing the opposite of what is generally effective.
How can we change this behavior from a team standpoint? For a start, keep track of who proposes which theories, and who tends to be right more often than not. Offer kudos to people who solve problems quickly and efficiently rather than those who solve "hard" or "unusual" problems. We tend to judge the "hardness" retroactively based on the perceived unlikeliness or difficulty of the solution, even though most debugging problems look hard when you don't know the solution, and this penalizes people who try simple solutions first and plow through lots of fixes in a shorter period of time.
The second thing to do is to make people feel bad about improbable theories. This can be as simple as encouraging mild ridicule of team members who consistently make bizarre leaps of faith in their hypotheses. At the very least, force people to make their reasoning explicit. Is their hypothesis really the one that is least inconsistent with the evidence, or is there a much more parsimonious explanation? With a carrot for rapid debugging and a stick for improbable theorizers it should be possible to eliminate the ego factor and improve your team's hypothesis-making.
