Competing hypotheses

What it is

Analysis of competing hypotheses is a way of choosing between explanations. You write down every explanation that could account for what you see, list your evidence and assumptions once, then check each item against every explanation to find which ones it contradicts. The explanation you keep is the one left standing after the contradictions are counted, not the one with the longest list of supporting facts. Use it when the decision turns on why something is happening.

Where it comes from

Richards J. Heuer, Jr. built the method inside the CIA's Directorate of Intelligence in the late 1970s, where he headed the methodology unit of the political analysis office until his retirement in 1979. It first appeared in public as an article titled "Analysis of Competing Hypotheses" in the Agency journal Studies in Intelligence in summer 1981. Heuer gave it its full form as Chapter 8 of Psychology of Intelligence Analysis, published in 1999 by the CIA's Center for the Study of Intelligence as a collection of the papers he wrote for the Directorate between 1978 and 1986. He calls it there "an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method." Heuer and Randolph Pherson later catalogued it among fifty techniques in Structured Analytic Techniques for Intelligence Analysis (2010).

What it corrects

The failure has a name in Heuer's book: satisficing, borrowed from decision analysis. You form an early sense of the likely answer, then read the file for whether it supports that answer. It usually does, and confidence climbs with the size of the file. The defect is not laziness. It is that the same reports also fit the explanations you never wrote down, so their agreement with your favourite says almost nothing about whether your favourite is true. Ordinary care makes this worse: a careful person gathers more evidence, most of it undiagnostic, and confidence rises without anything being tested.

How it works

1. List the explanations, including ones you think unlikely, and let people who see the situation differently add to the list.
2. List the evidence and arguments once, in one place. Include your assumptions about motive and capability, not only the documents.
3. Build a matrix with the explanations across the top and the evidence down the side. Work across each row, marking whether that item is consistent with, inconsistent with, or irrelevant to each explanation.
4. Cross out the evidence consistent with every explanation. It has no diagnostic value, however solid it is. Merge explanations that nothing in the list separates.
5. Now work down the columns and try to disprove. Rank the explanations by weight of inconsistency, not by weight of support.
6. Test the two or three items your ranking rests on: what changes if one is wrong, misleading, or open to a second reading?
7. Report where every explanation stands, and name the observations that would move the ranking later.

Worked example

In 2000 Iraq's Military Industrialization Commission ordered 60,000 aluminium tubes: alloy 7075-T6, 81 mm outer diameter, 900 mm long, 3.3 mm wall. A first batch was intercepted in Jordan in 2001. The CIA's weapons centre read the tubes as rotor material for uranium centrifuges. Analysts at the Energy Department's laboratories and the State Department's intelligence bureau read them as combustion chambers for the Nasser-81 rocket, an Iraqi copy of the Italian Medusa, reverse engineered from 1986 and declared to inspectors in 1996. Both dissents were printed in the October 2002 National Intelligence Estimate.

Most of the evidence sat in both columns. The alloy discriminated nothing: the Medusa and the American Hydra-70 rocket are both built from 7075-T6. The strongest point for the nuclear reading was tolerance, which Colin Powell put to the Security Council on 7 March 2003 as specifications "more exact, by a factor of 50 percent or more, than those usually specified for rocket motor casings." IAEA inspectors, back in Iraq from December 2002, looked for a fact that would separate the two readings. They found it by interviewing the Iraqi engineer who had set the tolerances: he tightened them because a test rocket stuck in its launch tube and exploded. On 27 January 2003 Mohamed ElBaradei told the Council that the specifications appeared "consistent with reverse engineering of rockets" and that the tubes were "not directly suitable" for centrifuges. The Iraq Survey Group agreed after the invasion. The nuclear case rested almost entirely on evidence that fitted both explanations, and the fact that told them apart was available to anyone who asked what the tolerances were for.

In a Business Case Weekly case

In the Nokia case you sit on the group executive board in 2006. Volumes are climbing, logistics costs are the envy of the industry, share is above a third of the world, and a small internal team says those numbers measure the phone business you built rather than the one software is about to create. Competing hypotheses does one job at that fork: it shows that the whole scorecard is consistent with both readings, so none of it decides between them. That pushes you to ask which single observation would look different under each reading, and whether the company already collects it. The method sorts the evidence; it does not tell you which reading is right.

In your answer

• "Two explanations fit these facts: … and … ."
• "This number is consistent with both, so I am not counting it as support."
• "The evidence that would count against my own reading is …"
• "The next thing I would look for is …, because it falls one way if the first explanation holds and the other way if the second does."

Common misuse

The usual failure is the matrix as arithmetic: score every cell, add the columns, hand in the lowest total. Scoring buries the two judgments the method exists to expose: whether a source is reliable, and whether an item really discriminates. A second failure is the decorative alternative, a rival listed so weakly that it was never in contention. One test catches both. Point at a single cell and change its mark. If that forces you to think again about the ranking, the matrix is working. If it only changes a total, you built a spreadsheet.

References

• Richards J. Heuer, Jr., Psychology of Intelligence Analysis, Center for the Study of Intelligence, CIA, 1999. Chapter 8 is the method itself, about twenty pages, an evening's read.
• David Albright, Iraq's Aluminum Tubes: Separating Fact from Fiction, Institute for Science and International Security, 5 December 2003. The tube-by-tube record of both hypotheses.
• Randolph H. Pherson and Richards J. Heuer, Jr., Structured Analytic Techniques for Intelligence Analysis, CQ Press, 2010. No stable public copy found.