Is Prediction the Same as Understanding?

Before the library committee meets again, a researcher shows the chair a model trained on previous meetings. The screen displays five predicted votes in favour of closing the Saturday reading room.

That evening, five people vote yes. Every prediction is correct. A week later, three members admit that they had privately wanted the room to remain open. The chair asks the researcher a new question: what would the vote have been if I had spoken last?

The model has no reliable answer. It has never observed this committee with a different speaking order.

This is a fictional continuation of the committee in the previous investigation, not a report of an actual experiment. Its puzzle is real enough: when a system gets the result exactly right, what has it demonstrated? And what are we adding when we say it understands why?

A correct forecast establishes that something was predictable. It does not, on its own, establish what made it happen.

The Hidden Assumption

We often treat the ability to anticipate an event as proof of insight into it. The friend who knows you will refuse dessert seems to know you. The analyst who predicts an election seems to understand the electorate. A machine that forecasts a committee's decision can appear to know the minds around the table.

Sometimes that impression is deserved. A prediction may come from a carefully tested account of how a process works. But the same outcome can be anticipated through a shortcut, a habit, a coincidental clue, or a dependable feature of the environment that will later disappear.

Part of the confusion lies in the word understanding. It can mean being able to use a system, identifying its causes, explaining a person's reasons, or constructing a theory that survives attempts to falsify it. Those achievements overlap, but they are not interchangeable.

Before awarding the model a deeper ability, we should ask what question it was actually tested on.

Statistics and scientific method

Two Different Assignments

Suppose two researchers examine years of library decisions. One wants the most accurate forecast of the next vote. The other wants to know whether the chair's speaking first suppresses dissent.

The first researcher can benefit from any dependable signal available before the vote: the agenda, attendance, history, or the chair's known position. The second must distinguish influence from association. Perhaps the chair speaks first only on decisions already settled privately. Perhaps difficult proposals are assigned a different procedure.

Statistician Galit Shmueli drew a consequential distinction in her 2010 paper 'To Explain or to Predict?': explanatory and predictive modelling answer different questions and can require different choices at every stage of analysis. A model designed for one purpose is not automatically strong at the other.

This is why a forecast's test score cannot settle a causal debate. The predictor may be well validated for its stated task and still lack the information needed to answer what would happen under a changed procedure.

Shmueli (2010), To Explain or to Predict? · Statistical Science

Causal inference

The Difference Between Watching and Changing

There are at least three questions we might ask about the committee. Given the chair's usual behaviour, how likely is a yes vote? If the chair is required to speak last, how would the distribution of votes change? For this particular meeting, would a member have dissented if the chair had stayed silent?

The first asks about association. The second asks about intervention. The third asks a counterfactual about an event that has already occurred. Judea Pearl's causal framework separates these kinds of questions because evidence sufficient for one need not identify the others.

That hierarchy is a framework for causal queries, not a simple scale for ranking intelligence or consciousness. And it does not imply that observational data are useless for causal inference: they can be informative when joined to defensible assumptions, research design, or other evidence. The missing step is justification for moving from what was observed to what a deliberate change would cause.

Our model knows that a chair's early endorsement accompanies agreement. It might have learned a causal influence, or merely a cue that the decision was settled before the meeting. Those explanations make similar forecasts while recommending very different reforms.

Pearl (2019), The Seven Tools of Causal Inference, with Reflections on Machine Learning · Communications of the ACM

A Test That Could Tell the Stories Apart

The committee could record members' private provisional views before the meeting, vary the speaking order where appropriate, and document which new information reaches the table. Over several comparable decisions, researchers could examine whether a chair's early endorsement changes expressed support.

That would be a more discriminating test than asking the same prediction system for another forecast in its familiar setting. It would also be difficult to conduct well. Different agenda items have different stakes; committees are small; changing a process can change who attends and how seriously people prepare.

The point is not that a randomized trial must precede every claim of understanding. Rather, competing explanations should imply different observations somewhere. If no conceivable change would distinguish them, their apparent certainty deserves caution.

At the same time, even a measured effect of speaking order would leave questions open about motive. Did members defer to expertise, avoid embarrassment, or interpret silence as agreement? Explaining a social action may require evidence about reasons as well as voting patterns.

Machine learning

When a Shortcut Looks Like Knowledge

A model can perform well by exploiting a signal that works in the training environment but does not travel well beyond it. Researchers call this shortcut learning. A system may recognize the surrounding context more reliably than the underlying object or mechanism its users believe it has learned.

Robert Geirhos and colleagues argued in a 2020 perspective that many failures of deep learning can be understood through shortcuts: rules that score well on standard benchmarks yet struggle under more demanding conditions. Their argument concerns a recurring failure mode, not a declaration that every successful model is superficial.

For the committee, imagine one predictor that follows the chair's early statement and another that models whether disputed information has circulated. Both could score perfectly on a small set of meetings where the chair always spoke first. After a procedural change, their forecasts might diverge.

A held-out test drawn from the same old routine would not necessarily reveal the difference. That is a limitation of the evidence we chose, not evidence that prediction itself is worthless.

Geirhos et al. (2020), Shortcut Learning in Deep Neural Networks · Nature Machine Intelligence

Several Correct Models, Several Different Stories

Even without an obvious shortcut, a prediction task may leave crucial features of a model unsettled. Two systems can receive similarly strong scores while relying on different relationships, particularly when the test data resemble the training data.

In a 2022 study in the Journal of Machine Learning Research, Alexander D'Amour and a large team described this as underspecification. Across applications including medical imaging, clinical risk prediction, language processing, and computer vision, they documented how models that appeared comparable in their original testing domain could behave differently elsewhere.

This finding gives our committee puzzle a sharper edge. Predictive success can be compatible with several incompatible accounts of what matters. The model that is accurate today may fail tomorrow because the rule it relied on was never uniquely established by the evidence.

That does not mean every prediction conceals a mistaken theory. It means that where several explanations fit, additional tests should examine the differences that matter before an institution relies on the model.

D'Amour et al. (2022), Underspecification Presents Challenges for Credibility in Modern Machine Learning · JMLR

An observed forecasting failure

The Flu Forecast That Followed the Searches

The stakes can be larger than a reading room. Google Flu Trends attempted to estimate influenza-like illness from patterns in search behaviour. In February 2013, its estimate of the proportion of US doctor visits associated with influenza-like illness was reported as more than twice the estimate from the Centers for Disease Control and Prevention.

David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani examined that failure in a 2014 Science article. Their criticism concerned the hazards of treating a large volume of platform-generated data as a stable measure of disease, as well as problems in how a forecasting system was assessed and updated.

Search activity can be informative about illness, but it also reflects media attention, platform design, and how people respond to a perceived threat. A relationship that forecasts well for a period may shift while the biological process of infection remains much the same.

The lesson is not that online signals must never be used for public health. It is that a predictive proxy should not quietly become the phenomenon it was designed to approximate.

Lazer et al. (2014), The Parable of Google Flu: Traps in Big Data Analysis · Science / Harvard University

Can a Good Explanation Be a Poor Forecast?

Now reverse the question. Someone may understand important features of a process yet lack the measurements needed to predict an individual result. A doctor can understand why a treatment carries a risk without knowing which patient will experience it. A researcher can identify a cause of group conformity without forecasting every vote.

Uncertainty, incomplete observations, competing causes, and genuinely stochastic elements can limit predictions. Explanatory insight does not promise omniscience. If we demanded flawless forecasts as the admission price for understanding, we would reject much of useful science.

But there is a danger in excusing every failed forecast by claiming deeper understanding. A proposed mechanism earns credibility through evidence, including predictions or interventions on which it could have been wrong. An explanation insulated from all possible tests is too comfortable.

The productive question is therefore what the theory should predict under specified conditions, and whether it continues to survive those tests.

The Strongest Case for Prediction

A defender of predictive systems has a serious argument: a theory that repeatedly anticipates unfamiliar outcomes, especially after deliberate changes to the system, has demonstrated more than luck or superficial correlation.

Reliable forecasting is indispensable in many domains. A causal model that cannot anticipate consequences under the conditions for which it was built deserves scrutiny. And useful prediction can reveal patterns that prompt explanations no one had previously considered.

In a 2021 Nature perspective, Jake Hofman and colleagues argued for integrating explanation and prediction in computational social science rather than treating them as rival ambitions. They distinguished the goals while describing how disciplined out-of-sample testing and causal research can inform each other.

So the answer cannot be that prediction and understanding inhabit separate worlds. Prediction can be evidence for understanding, especially when rival accounts imply different future outcomes. What it cannot do, by itself, is prove that one particular story about those outcomes is the right one.

Hofman et al. (2021), Integrating Explanation and Prediction in Computational Social Science · Nature

An Explanation of a Model Is Not Yet an Explanation of the World

Suppose the researcher can display the model's internal feature weights and show that the chair's speaking order contributed heavily to its predicted vote. That is useful. It tells us more about how the system reached its output.

Yet it does not settle whether the chair's speaking order actually caused the committee to agree. A feature can be influential inside a model because it is a reliable sign of something else. Interpreting a model's computation and explaining the world it describes are different tasks.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin introduced LIME in 2016 as a method for approximating the reasoning of a classifier near an individual prediction. Work of this kind can help users inspect and challenge systems, while the approximation itself must be evaluated for faithfulness and scope.

For the committee, a credible explanation should connect model behaviour with independent evidence about the meeting. The software's account of its own calculation is one witness, not the entire case.

Ribeiro, Singh & Guestrin (2016), Why Should I Trust You? Explaining the Predictions of Any Classifier · ACM SIGKDD

When the Forecast Becomes Part of the Event

There is a final complication in predicting people. A forecast may reach the people whose behaviour it anticipates.

If the chair sees five expected yes votes before discussion, she might call for a quick vote. A doubtful member might hesitate because the result seems settled. Another might deliberately oppose the forecast. Each response changes the setting the original model described.

This is a logical possibility in our invented committee, not evidence that any particular prediction system creates conformity. But it shows why a forecast used to make a decision is different from one kept out of view.

The model might predict the meeting accurately when hidden and fail when displayed. That is information worth knowing before building a policy around the prediction. It also raises a moral question: even if a model accurately predicts an outcome, does that grant anyone the right to bring it about?

Where the Fields Collide

Statistics asks whether the predictions hold up on data the model has not seen. Causal inference asks which interventions would change the outcome. Machine-learning research tests whether the learned rule survives a change of environment. Philosophy of science asks what kind of explanation deserves belief.

Social psychology adds something a voting spreadsheet cannot supply on its own: people have beliefs about other people's beliefs, reputations to protect, and reasons they may not express in a meeting. Institutional design determines which of those reasons can safely enter the record.

These fields are strongest together. A reliable predictor can identify a pattern worth investigating. A causal account can explain which changes matter. A careful institution can then decide whether acting on that knowledge is justified, considering the people affected and the uncertainties that remain.

Accuracy, explanation, and permission are three different achievements. Confusing them is dangerous precisely because a system may be excellent at one of them.

What Would Count as a Better Test?

For the library model, we would want to know how often its forecasts work on genuinely new meetings, whether it reports uncertainty accurately, and whether its performance deteriorates when chairs, agendas, or procedures change.

We would also compare rival hypotheses. Does early endorsement cause agreement, or signal a prior private consensus? What does the model say when members disclose their initial views anonymously? Which evidence from participants would support or undermine the inferred mechanism?

No single test answers everything. People can misreport motives, experiments can disturb the practice being studied, and a small committee offers few opportunities for comparison. Those limitations argue for carefully bounded claims, not for surrendering the effort to understand.

When a system's purpose is merely to forecast next week's attendance, verified predictive performance may be sufficient for that narrow task. When it is used to silence objections, redesign deliberation, or judge people's intentions, the explanatory burden grows.

What We Know — and What We Don't

Research distinguishes predictive modelling from causal explanation. Work on shortcuts and underspecification demonstrates that strong performance in one testing environment can coexist with sharply different behaviour elsewhere. The Google Flu Trends episode documents a real case in which a much-discussed forecasting approach badly missed a public-health measure.

These findings do not establish that every accurate model lacks understanding, or that a causal theory must be readable in ordinary language to be scientifically useful. Nor do they prove that we possess one agreed test of understanding across humans, scientific theories, and artificial systems.

The opening committee is invented. We have not observed its members' private preferences or experimentally changed its meeting rules. Our rival explanations illustrate what would need testing; they are not findings about actual library boards.

The warranted conclusion is narrower and more demanding: to call a prediction evidence of understanding, we should say what kind of understanding we mean and identify the tests that distinguish it from an effective shortcut.

Back to the Reading Room

The researcher returns to the library with a modest proposal. Before the next vote, everyone will write down a preliminary view. The chair will speak last. The minutes will record which facts were known before discussion and which surfaced during it.

Perhaps the vote will still be unanimous. Perhaps two people will object. Either outcome could be reasonable. What matters to the investigation is that the changed procedure gives the rival explanations a chance to disagree.

If the model adjusts correctly, that would strengthen confidence in its account, though it would not settle every question about the members' reasons. If it fails, the failure could be a valuable discovery about the boundary of its knowledge.

The original forecast was right. Understanding begins when being right is no longer the only question we ask.

The Next Question

Now imagine a recording of that meeting circulates online. In it, the chair appears to confess that the vote was arranged in advance. The voice is familiar. The room looks real. The words fit what people already suspect.

The clip could be genuine, misleadingly edited, or fabricated. A convincing appearance offers a prediction about what an authentic recording should look and sound like. It does not establish that the recorded event occurred.

If producing believable evidence becomes easier than checking its origin, how should people decide what to trust? What happens when reality becomes harder to verify than to fake?