Competing Hypothesis Analysis

Summary

What Most of Us Do

Most of us choose the first solution to an intelligence problem that seems satisfactory instead of looking at all possible solutions to pick the best one.

When we try to arrive at a judgment, we usually pick our favorite hypothesis. Often we do this on the basis of our gut feeling.
Then we look at our information to see whether or not it supports our judgment.
If the information seems to support our judgment, we say "great!" and don't look further.
If our information doesn't, we reject the information as bad, or we pick mother judgment and go through the same process until we find a judgment and evidence we think match.
What we're doing is slopping with the first judgment that seems right.

What's Wrong?

Focusing on one hypothesis at a time has several weaknesses:

We tend to see what we're looking for and not explore information relating to other judgments.
We overlook the fact that most evidence is consistent with several different hypotheses.

Most of us can't think about several hypotheses at the same time and how each piece of information fits into each judgment. Looking at a single, most likely hypothesis is much easier.

A Better Way

Analysis of competing hypotheses gives you a technique for systemat ically examining a range of possible judgments and choosing the one you believe is correct.
This technique will help you analyze difficult intelligence issues that require you to carefully weigh alternative conclusions and to show how you arrived at your judgment.

The Method

Identify all possible hypotheses to consider. Have a brainstorming session with a group of analysts who have different perspectives.

Raise every possibility no matter how remote. Don't evaluate any hypothesis-that comes later.

Make a list of the significant evidence. Include as evidence the absence of things you would expect to see if a hypothesis were true. The absence of evidence is as important as evidence you can see.

Interpret "evidence" broadly to include everything that affects your judgments: logical deductions, assumptions, indicators, specific developments.
Make a general list of evidence that applies to the situation as a whole.
Then consider each hypothesis individually. Note factors that either support or contradict each one. Also note the absence of evidence.

Consider how each piece of evidence relates to each hypothesis. To help you do this, create a matrix with hypotheses across the top and evidence down the side.

This step differs from our natural, intuitive approach to analysis. However, the finished matrix will help you understand which evidence carries the greatest weight in deciding which hypotheses are most or least likely.
In this step you work across the matrix. In step 5 you work down the matrix, looking at how one hypothesis relates to all the evidence.
After you set up your matrix, take one piece of evidence and ask yourself if it's consistent with, inconsistent with, or irrelevant to each hypothesis. Make a notation accordingly under each hypothesis.
After doing this for the first item of evidence, go to the second, third, and so forth.
Use whatever kind of notation you like, such as pluses and minuses; C, I, and N/A for consistent, inconsistent, or not applicable; or brief words.
Evidence has no diagnostic value is it's equally consistent with all hypotheses. That is, it doesn't help you determine that any one hypothesis is more or less likely than another. Deleting this kind of evidence helps keep your matrix manageable.
If a piece of evidence is significantly more or less likely for one hypothesis than another, note it under that hypothesis. For example, use double pluses or minuses.

Refine your matrix. Reconsider the hypotheses and delete evidence that has no diagnostic value.

Now that you see how the evidence breaks out under the hypotheses, reconsider your hypotheses. Do you need to add new ones? Reword others? Combine a couple?
Reconsider the evidence too. Are there factors not on the list that are influencing your thinking about which hypotheses are most or least likely? If so, add them. Delete items that now seem unimportant or have no diagnostic value - save them on a separate list so you have a record of what information you considered.

Draw tentative conclusions about the relative likelihood of each hypothesis. Do this by trying to refute hypotheses rather than confirm them.

Steps 2 and 3-listing and comparing the evidence as it relates to all the hypotheses-and step 5 force you to spend more analytic time than you would have on what you thought were less likely hypotheses. Thus, you give all the alternatives a fairer shake.
Begin by looking for evidence that lets you reject a hypothesis. You can't prove a hypothesis is true when evidence for it is consistent with other hypotheses. However, a single piece of inconsistent evidence may be enough to reject a hypothesis.
The hypothesis with the most minuses - or whatever notation you usedprobably is the least likely hypothesis. (The one with the most pluses is not necessarily the most likely.)
Ordering the hypotheses by the number of their minuses gives you a rough ranking of their probability. Of course, some evidence is more important than other evidence, and a plus or minus can't capture some degrees of inconsistency.
The matrix shouldn't dictate the conclusion to you. It should accurately reflect your judgment of what the important factors are and how they relate to the probability of each hypothesis. It also gives you an audit trail of how you arrived at your conclusion.
You may disagree with what hypothesis the matrix shows as probable or unlikely. If so, you've left out factors that have an important influence on your thinking. Go back and put them in so the analysis reflects your best judgment.

Analyze how dependent your conclusion is on a few critical pieces of evidence. What would be the consequences for your analysis if that evidence were wrong or misleading or could be interpreted differently?

This step helps you identify critical assumptions that you haven't recognized and that, if wrong, would invalidate your conclusions.
Go back to step 3, where you decided which evidence was most diagnostic, and to step 5, where you weighed the relative likelihood of the hypotheses.
Single out the evidence that was most influential in causing you to reject or downplay the probability of the alternative hypotheses.
Scrutinize this evidence.
What assumptions underlie your understanding and interpretation of this evidence?
Do possible alternative explanations or interpretations exist?
Does the source have any motive for deceiving you?
Do you need to go back to original source materials to check the accuracy of translation, transcription, or someone else's interpretation?
When you write your paper, identify assumptions you used to interpret the evidence and note that your conclusion depends on the validity of these assumptions.

Report your conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.

Consumers have to make decisions using a full set of alternative hypotheses, not a single one that you tell them is most likely. Decisionmakers may need contingency plans in case one of the less likely alternatives toms out to be true.
To make your argument for a judgment complete, also discuss the hypotheses you rejected and why you rejected them.

Identify milestones that might indicate events are taking a different route from what you expected.

The situation you're following may change, or you may get new information that alters your appraisal.
You and the reader benefit when you lay out things to look for in the future or things unknown to you that could change your hypothesis.

Index

What Is Spero News