Competing hypotheses

What it is

Growth has slowed, and the right move depends on why. Maybe demand is weak. Maybe new customers find the product hard to start using. Maybe the company cannot keep up with demand. Each explanation fits the same facts.

Analysis of competing hypotheses is a way to choose between possible explanations like these. You write down every one that could fit, and list your evidence and assumptions together. Then you check each item against every explanation for contradictions. You keep the one the evidence contradicts least, even if another has more facts in its favour.

Where it comes from

In the late 1970s Richards J. Heuer, Jr. built the method in the CIA's Directorate of Intelligence. There he ran the methods team of the political analysis office until he retired in 1979. The method first appeared in public in summer 1981, as an article of the same name in the CIA journal Studies in Intelligence.

Heuer set out the full method in Chapter 8 of his 1999 book Psychology of Intelligence Analysis. It collects papers he wrote for the Directorate from 1978 to 1986, some on contract after he retired. In it, he calls the method "an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method." Below, his last two steps are merged into one: report the conclusions and name what to watch. In 2010 Heuer and Randolph Pherson listed the method among fifty techniques in Structured Analytic Techniques for Intelligence Analysis.

What it corrects

Heuer calls the mistake satisficing, a decision-analysis term for stopping at the first answer that seems good enough. Early on, you sense the likely answer, then go through everything you have, looking for support. You usually find it, and the more you have, the surer you feel.

Laziness is not the problem. The same facts also fit the explanations you never wrote down. So when they agree with your favourite, you learn almost nothing about whether it is true. Working harder makes this worse. A careful person gathers more evidence, but most of it fits every explanation. You feel surer, yet you have tested nothing.

How it works

1. List the explanations, even unlikely ones, and ask people who see things differently to add more.
2. List the evidence and arguments in one place, with your assumptions about what people want and can do.
3. Build a matrix: a table with explanations across the top and evidence down the side. In each cell, mark whether the item fits that explanation, contradicts it or is irrelevant.
4. Cross out evidence that fits every explanation: however solid, it cannot help you choose. If nothing on the list separates two explanations, merge them.
5. Go down each column and try to disprove that explanation. Rank the explanations by the evidence against them.
6. Test the two or three items your ranking rests on. What if one is wrong, misleading or open to another reading?
7. Report where each explanation stands and what could change the ranking later.

Worked example

In 2000 Iraq's Military Industrialization Commission ordered 60,000 aluminium tubes of alloy 7075-T6. Each was 81 mm across the outside and 900 mm long, with 3.3 mm walls. A first batch was stopped in Jordan in 2001.

The CIA's weapons centre believed the tubes were rotors for the centrifuges that enrich uranium. Analysts at the Energy Department's laboratories and the State Department's intelligence bureau disagreed. They believed the tubes were combustion chambers for the Nasser-81 rocket. Iraq had begun copying it from the Italian Medusa rocket in 1986 and declared it to inspectors in 1996. Their objections were printed in the October 2002 National Intelligence Estimate.

Most of the evidence fitted both explanations. The alloy was no help: the Medusa and the American Hydra-70 rocket are both made of 7075-T6. The nuclear explanation's strongest point was the tolerances, how exact each tube had to be: more exact than a rocket body seemed to need.

Inspectors from the IAEA, the UN's nuclear agency, started inspections in Iraq again on 27 November 2002. They looked for a fact that would separate the two explanations. On 27 January 2003 Mohamed ElBaradei, the agency's director general, reported to the UN Security Council. He said the tubes "would be consistent with the purpose stated by Iraq" and, "unless modified, would not be suitable for manufacturing centrifuges."

The deciding fact came when inspectors found and questioned the Iraqi who had set the tolerances. He had tightened them so the rocket would not stick in its launch tube. In a test, one had stuck and exploded. On 7 March 2003 ElBaradei told the Council that inspectors had learned from documents and interviews why the tolerances changed. That same day Colin Powell, the US Secretary of State, still cited them to the Council. He called them "more exact, by a factor of 50 percent or more, than those usually specified for rocket motor casings."

After the 2003 invasion, the Iraq Survey Group searched Iraq for banned weapons and sided with the inspectors. The nuclear case rested almost entirely on evidence that fitted both explanations.

In a Business Case Weekly case

In the Nokia case it is 2006 and you sit on Nokia's executive board. Nokia sells more phones each year. Rivals envy its low logistics costs. It holds over a third of the world market. The obvious reading: the company is strong. But a small team inside Nokia disagrees. The team says these numbers measure the phone business you built, not the one software is about to create.

Here, competing hypotheses means checking these numbers against both readings. Every number fits both, so none helps you choose. Instead, ask which single observation would look different under each reading. Does the company already track it? Which reading is right is still your call.

In your answer

• "Two explanations fit these facts: … and … ."
• "This number fits both, so I am not counting it as support."
• "Evidence against my own explanation would be …"
• "Next I would look for …: it points one way under the first explanation and the other way under the second."

Common misuse

The usual mistake is to turn the matrix into arithmetic: score every cell, add the columns, hand in the lowest total. When you add up scores, you skip the two judgements the method is for. Is this source reliable? Does this item really tell the explanations apart? The second mistake is a rival added for show, so weak that it never had a chance. You can catch both with one check. Change the mark in a single cell. If that makes you rethink the ranking, the matrix is working. If it only changes a total, you have built a spreadsheet. If no mark in a rival's column could ever lift it in the ranking, the rival is just for show.

References

• Richards J. Heuer, Jr., Psychology of Intelligence Analysis, Center for the Study of Intelligence, CIA, 1999. Chapter 8 is the method: about twenty pages, an evening's read.
• David Albright, Iraq's Aluminum Tubes: Separating Fact from Fiction, Institute for Science and International Security, 5 December 2003. Both readings of the tubes, point by point.
• Randolph H. Pherson and Richards J. Heuer, Jr., Structured Analytic Techniques for Intelligence Analysis, CQ Press, 2010. No stable public copy found.