Why AI Risk Estimates Disagree So Much

Separate the test from the claim

AI-safety headlines often blur together very different kinds of evidence. A controlled evaluation, a real-world incident, a forecast and a hypothetical scenario can all be important, but they do not prove the same thing.

Long-horizon forecasts compound uncertainty. Keep that condition attached to the result. Removing the setup can make a careful research finding sound much broader than the evidence supports.

What matters in practice

Long-horizon forecasts compound uncertainty.

Evaluation results are most useful when the setup is visible: what the model knew, which tools it had, what incentives were created and what counted as success. Without that context, a dramatic behaviour can be easy to overgeneralise.

Different definitions of AGI, catastrophe and extinction produce different questions.

Real-world incidents deserve a separate category because deployment adds people, permissions, infrastructure and operational constraints that laboratory tests may deliberately simplify or remove.

Expert elicitation measures beliefs; it does not directly measure future events.

Uncertainty is part of the evidence, not a reason to abandon the subject. When experts disagree, the assumptions behind the disagreement are often more informative than a single headline number.

What the evidence does — and does not — show

Safety research needs careful boundaries. Results can be important without being universal, and a behaviour observed in a deliberately constructed evaluation should not be treated as if it had already occurred in every real deployment.

• There is no neutral formula that resolves all assumptions.
• Confidence can be poorly calibrated on unprecedented events.

What could change the answer?

New replications, deployment data and incident reports can strengthen or weaken the original interpretation. The right response to fast-moving evidence is to update the conclusion, not defend an older headline after the facts have moved on.

What you can do now

Keep the source and the claim side by side. Read what the researchers or incident report actually showed, note the conditions, and resist upgrading a limited result into a general statement it was never designed to prove.

1. Compare assumptions before percentages.
2. Separate near-term harms from extinction scenarios.
3. Update views when concrete capability or governance evidence changes.

A four-question reality check

When a claim sounds dramatic, work through these four questions before deciding how much weight to give it.

Bottom line

Treat evidence according to what it can actually support. Strong concern can be justified without turning a controlled test into a claim about what every deployed AI system is doing.

Two informed people can produce very different AI-risk estimates because they assume different capability growth, deployment speed, safeguards, geopolitical responses and definitions of catastrophe. The disagreement is information, not merely noise.