When an AI answer feels wrong before you know why

The fire that would not behave like a kitchen fire

Gary Klein recounts a young fire lieutenant arriving at a one-storey house where flames were coming from the rear. The lieutenant assumed the fire was in the kitchen and sent a hose crew inside. They applied water, stopped to see the effect, and watched the fire return at the same intensity. They tried again. The fire still did not diminish.

The lieutenant brought the crew back to the living room to decide what to do next. He became uneasy and ordered everyone out. As they were leaving, the living-room floor collapsed. The fire had been in a basement they did not know was there.¹

Klein later interviewed the lieutenant and asked why he had evacuated. The only explanation the lieutenant could offer at first was extrasensory perception. With further questioning, he recalled that the room had been much hotter than he expected for a fire in the next room. It had also been quieter. Klein’s reconstruction was that the missing noise, together with the unusual heat, violated what the lieutenant expected and contributed to his alarm.

The account makes an attractive case for expert intuition, but it supports a narrower conclusion. The lieutenant noticed a consequential mismatch before he could explain it, and his first explanation was not the one Klein eventually reconstructed. Klein’s published account does not include the original incident record or interview transcript, so it cannot prove the exact psychological cause of the evacuation. It does show why the unexplained reaction was worth investigating.

When the first reaction could not settle the move

In the second game of the 2016 match between AlphaGo and Lee Sedol, AlphaGo placed a stone that became known as Move 37. Fan Hui’s participant commentary says many players initially thought the stone had been placed in the wrong position. His own response began with surprise and incomprehension. He then examined how the move worked with AlphaGo’s other stones and came to value its potential across the centre of the board.²

Lee had left the room before Move 37 appeared. Fan reports that he returned, saw the board, and then considered his response. The commentary also identifies AlphaGo’s later Move 43 as an overplay. Neither a startled expert nor a strong machine receives automatic authority. Fan’s appraisal moved from incomprehension toward an account of the move’s central potential; the same commentary calls the machine’s later move an error. AlphaGo’s eventual win does not prove the value of every move.

The firefighter and Go accounts create opposite pressures. In one, an unexplained alarm was later connected to specific cues. In the other, an initially implausible AI move survived expert scrutiny. The next step is to make each explanation predict something different.

Preserve the disagreement before explaining it

I’ve had this happen in my own writing. ChatGPT changed control to influence in a draft I wrote about here. Maybe control went further than I could justify. But maybe the edit also took something out: my concern that a system’s repeated choices could shape what I get to consider and choose.

If I ask the same model why the edit bothers me, it can give me a convincing explanation for either possibility. By then, I’m reacting to its explanation as well as the original edit.

Moshe Glickman and Tali Sharot studied 1,401 participants across perceptual, emotional and social-judgment experiments. In one perceptual task, people gave an independent response before seeing an algorithm’s answer. Repeated interaction with a biased algorithm made later independent judgments more biased. Interaction with an accurate algorithm made them more accurate.³

These were bounded tasks with measurable answers, not long-form writing or expert interviews. They do not show whether sustained AI use improves or corrupts a person’s conceptual standards. They do show why the person’s judgment before further model exposure is useful evidence. The conversation can affect the later judge.

Put the explanations to a test

Consider a constructed founder example. No person, company, figure or decision below is real, and none of the rules is financial advice.

An AI interviews a founder about when customers may receive 60-day payment terms. It proposes a simple rule: approve customers that buy at least $250,000 a year and have paid on time for two years.

Two past decisions contradict it. Both customers met those conditions, had the same margin and faced the same delivery schedule. The founder refused one request and approved the other.

The orders differed in two ways. The refused order required an $80,000 cash outlay before payment, and its custom goods could not be resold within 30 days. The approved order required $30,000 and used standard goods that could be resold within 30 days without a discount.

At least two possible rules fit those decisions:

• Cash rule: for these otherwise comparable orders, approve 60-day terms if the pre-payment cash outlay is $50,000 or less; otherwise refuse.
• Resale rule: for these otherwise comparable orders, approve 60-day terms if the goods can be resold within 30 days without a discount; otherwise refuse.

Both rules predict the two past decisions. We still can’t tell which one better describes how she decides.

Now hold the customer history, annual spending, margin and delivery schedule constant. Construct a third order that requires an $80,000 cash outlay but uses standard goods that can be resold within 30 days without a discount. The cash rule says refuse. The resale rule says approve.

I haven’t supplied her answer. Ask her to decide before the AI explains the case. An approval would fit the resale rule; a refusal would fit the cash rule. Either answer would go against one prediction, leaving the other rule to test again. She might instead ask for missing information or reject the comparison. We’d then need to revisit the example. One matching answer wouldn’t show that we can reliably predict her decisions, that those decisions help the business, or why she objected in the first place.

The sequence is small enough to use in writing, learning or an expert interview:

1. Save the exact answer and the person’s first objection worth investigating before asking for an explanation.
2. Find two relevant decisions or examples, including one the current explanation struggles with.
3. Turn the plausible explanations into clear decision rules, so each says what the person would do next.
4. Find or construct a case where the rules point to different decisions. Record the person’s answer before asking the model to explain it.

The firefighter interview changed an explanation after the event. Fan Hui’s analysis changed his appraisal of a move. Neither, by itself, tells us whether the standard they now use is better. With the founder, we can at least write down two predictions before she makes the next decision.

A yes-or-no answer on this third case would go against one of the rules. Another case could expose a weakness in the rule that still fits, and the founder could reject the assumptions behind the comparison itself. We have something to test. Whether the rule helps her make better business decisions—or only changes how she explains them—is still an open question.

Sources