Judgment Playbook

Kind vs wicked learning environments

Distinguish feedback that teaches from feedback that rewards luck, delay, or hindsight.

5 minute readtaught in course chapter 9
EvidenceUncertainty

What it is

Kind versus wicked learning environments is a test you run on feedback before you trust it. A learning environment is kind when the same decision recurs and its result arrives quickly, clearly, and attached to the choice that caused it. It is wicked when the result is delayed, rare, filtered by your own earlier choices, or produced mostly by luck. The method sorts an outcome by the environment that produced it, so you know how much that outcome is able to teach.

Where it comes from

Robin Hogarth drew the distinction in Educating Intuition (University of Chicago Press, 2001). Hogarth, a psychologist who directed the University of Chicago's Center for Decision Research from 1983 to 1993, asked when experience produces trustworthy intuition. His answer was that the environment decides. Repeated judgments with immediate, accurate feedback train real skill. Missing, slow or distorted feedback trains confidence instead. He sharpened the framework with Tomás Lejarraga and Emre Soyer in 2015, defining a kind environment as one where the information available while you learn matches the information that matters when you apply the judgment. Soyer and Hogarth later wrote The Myth of Experience (PublicAffairs, 2020) for general readers. David Epstein's Range (2019) carried the two words into business writing, where most people now meet them, but the framework is Hogarth's.

What it corrects

A competent team reviews results, updates its beliefs, and repeats what worked. That procedure is correct in a kind environment and misleading in a wicked one. When feedback is slow and scarce, the sample that reaches the team is small, late, and shaped by their own past decisions. A lender never learns how the rejected applicants would have repaid. A firm never learns what the deal it walked away from would have returned. Ordinary care cannot fix this, because care gets applied to the data that arrived, and the gap is in the data that never did. More diligence on a biased sample produces a more confident wrong belief. The lesson then hardens into a rule nobody in the room can argue with, because everyone in the room watched it work.

How it works

  1. State the claim the outcome is being used to prove.
  2. Name the feedback that claim needs, and the date it arrives.
  3. Ask whether this decision repeats under comparable conditions, or is a single draw.
  4. List what the result cannot separate: luck, timing, selection, and your own influence on it.
  5. Build a kinder loop. Use a smaller repeatable version, an earlier measurable step, or a forecast written down before the result, and fix the date you will read it.

Worked example

Microsoft's search engine, Bing, changed its environment on purpose. Ronny Kohavi, who ran its experimentation platform, wrote in August 2013 that "on a typical day, there are over 250 experiments running on Bing", and that "when ideas are evaluated objectively in a controlled experiment, less than a third move the metrics they were designed to improve". Before that platform, product judgment lived in a wicked environment: a shipped feature's effect on revenue arrived mixed with season, competitors, and every other change made that week, so nobody's ranking of ideas was ever scored. One episode shows the cost. An employee proposed lengthening ad headlines by merging the title with the first line of text below it. Program managers treated it as low value and it sat unbuilt for more than six months, until an engineer ran it as a controlled experiment. Revenue rose 12%, which Kohavi and Stefan Thomke report as more than $100 million a year in the United States alone. What changed was the environment. Each idea now earned an attributable result within days, and a ranking that had felt reliable turned out to be close to noise.

In a Business Case Weekly case

In the Y Combinator case, the accelerator funded around 400 companies in the Winter 2022 batch, then cut the next batch by 40%, to 250. Limited partners were asking whether size had diluted quality. This method does not answer that question; it asks what evidence could ever answer it. A batch's real outcomes take most of a decade to appear. The case reports that four companies hold more than 84% of all public market value the firm has created, so the record rests on a few draws from 2009. The accelerator's brand also makes its picks easier to fund, so the feedback partly manufactures itself. Read that way, the 4.5% unicorn rate describes a portfolio built over twenty years rather than a test of this year's selection. The work at the fork is to name the earlier, repeated measurements that would show batch quality moving before 2030.

In your answer

  • "This result supports …, and it cannot separate that from timing or selection, because …"
  • "The first feedback that would test the claim is …, and it arrives by …"
  • "This decision happens once, so rather than wait for the outcome I would repeat …"
  • "I am recording the forecast now — … — so the review on … can be read against it."

Common misuse

The imitation is a label. Someone writes "this is a wicked environment", concludes that nothing here can be known, and leaves the plan exactly as it was. That version costs nothing and changes nothing. The opposite error is calling an environment kind because a number moves fast, when the fast number measures something other than the decision: clicks standing in for retention, signed pipeline standing in for revenue. One test covers both cases. Name a measurement, a date, and a result that would change what you do. A diagnosis that produces no such line has done none of the method's work.

References

Decision Quality: six links

Next method

Treat decision quality as a chain whose weakest link limits the whole recommendation.

Open next tool