A few weeks ago, I described the probability-testability matrix, and emphasized that you should consider both how likely a theory is, and how difficult it is to test that theory, when choosing how to proceed in a debugging effort. I have always been surprised that people seem to have trouble with this, and choose very low probability things to try first regardless of testability. I haven't had the resources to actually carry out a study on this, but I have a strong feeling that a cognitive bias is to blame, although I don't know if it already has a name in the literature. I've dubbed it the "Holding out Hope" bias. The idea is that people are biased against taking actions that leave them with worse options, even if selecting the better options means that the likelihood of success is higher early on.

To be more concrete, let's say that I have 4 theories for a problem, that I've guessed are 70, 50, 5, and 1 percent likely, respectively (note that it does not sum to 100% since I am simply giving a probability for each to be the outcome, and I might feel strongly about more than 1). If I start with the 70% theory and I'm wrong, then I've got something that's less likely to try. If I test the 50% and I'm still wrong, then I'm looking at a couple of low-probability theories and I'm starting to feel very uneasy. If I instead start with 1% and it's wrong, I can say "well I didn't think that was it, and I've got some much better options". I can continue to "Hold out Hope" that my higher likelihood theories are correct. It results in failure feeling like I'm improving my chances instead of reducing them.

Of course, by trying the things that we think are less likely first, we are greatly increasingly the number of theories that have to be tested, and that makes the whole effort take longer, even if it helps with morale. Remember that it's the fixing part that feels good, so avoid the Hold out Hope bias and go for the high probability stuff first.