What If Explainable AI Is Explaining the Wrong Thing?
Explainable AI promises to make black-box models easier to understand. But Gustau Camps-Valls argues that, in science, an explanation of a model can easily be mistaken for an explanation of the world itself. His alternative starts with a deceptively simple principle: think first, act second, model last.
Gustau Camps-Valls
Professor in Electrical Engineering, Universitat de València
Hi! PARIS International Visiting Professor 2026
Hi! PARIS Summer School 2026 · 5–6 minute read
The essential points
Explainable AI can reveal what a machine-learning model has learned, but that is not the same as identifying the physical or causal mechanisms behind a phenomenon.
Post-hoc explanations can reproduce model biases, overlook unobserved variables, and create confidence in conclusions that may be scientifically fragile.
Researchers are moving toward approaches that combine explainability with causality, physics, mechanistic interpretation, and structural causal models.
Camps-Valls proposes a different scientific workflow: Think, Act, Model, beginning with causal questions before building the model, rather than explaining it afterward.
An AI model predicts an extreme weather event. A second tool produces a colorful map showing which variables mattered most. Temperature appears here. Soil moisture there. Atmospheric pressure somewhere else. The result looks like an explanation. But what, exactly, has been explained?
This is the question Gustau Camps-Valls brought to the Hi! PARIS Summer School. A professor at the Universitat de València working at the intersection of artificial intelligence, causal inference and Earth sciences, Camps-Valls has spent years using machine learning to study complex systems.
His concern is not that explainable AI, or XAI, is useless. Quite the opposite. It can be an important diagnostic tool. The problem begins when we ask it to tell us more than it actually knows.
“XAI can explain what the models have learned,” he said. But researchers can too easily take the next step and assume that an explanation of the model is also an explanation of nature. That distinction is small in wording and enormous in consequence.
The Model Is Not the World
Explainable AI emerged partly in response to a basic problem with increasingly complex machine-learning systems: a model may make accurate predictions without making it obvious how it reached them. Methods such as feature-importance scores, SHAP values or counterfactual explanations try to open that black box. They can indicate which inputs played an important role in a prediction or how changing one variable might change an output.
For Camps Valls, these tools can be useful. But their object of explanation matters.
They explain the behavior of a model. They do not automatically explain the process that generated the real-world phenomenon being modeled.
That gap becomes particularly important in fields such as climate science, neuroscience or the social sciences, where researchers are rarely satisfied with knowing that two variables move together. They want to understand why.
Earth systems make that problem especially visible. The atmosphere, oceans, land and cryosphere interact across space and time. Researchers may want to know what drives a heat wave, why drought leads to displacement, or what would happen if one component of a system changed.
These are not simply predictive questions. They are causal ones. And an algorithm that identifies a highly predictive variable has not necessarily identified a cause.
When a Convincing Explanation Is Still Wrong
The danger is misplaced confidence. A machine-learning model can contain biases. Its training data can be incomplete. Important variables may be impossible to observe or measure. The model may rely on shortcuts that happen to work well in the available data.
An explainability method applied afterward can faithfully reproduce those weaknesses. In other words, the explanation can be accurate about the model and still misleading about reality.
Camps-Valls points in particular to unobserved confounders: variables that influence a process but are absent from the dataset. If they are missing from the model, an explainability tool cannot suddenly recover them simply by examining the model’s predictions.
This creates one of the problems he repeatedly returns to: researchers can be right for the wrong reasons. A model may predict well. Its explanation may appear plausible. Yet the underlying mechanism inferred from that explanation may still be incorrect.
There is another trap: confirmation bias. If an explanation produces a pattern that matches what researchers already expect, it is tempting to take that agreement as validation. If the result looks unfamiliar, they may change the model, adjust the method or search for another explanation.
The scientific question quietly shifts from “Is this true?” to “Can I make this result fit what I expected to see?”
For Camps Valls, this is where convenience becomes dangerous. Explainability tools are increasingly easy to apply. A researcher can train a model, run a package and generate polished visualizations with relatively little friction. But seamlessness has a cost if it removes the moments when scientists are forced to question their assumptions.
More Than Feature Importance
So what should replace the standard approach? Not one universal technique.
Camps-Valls instead points to several directions that move explainability closer to scientific interpretation.
One is hybrid modeling, where machine learning is combined with physical or parametric models. Rather than asking a neural network to learn everything from data, researchers can constrain part of the system with knowledge they already have about how the world works.
Another direction is mechanistic interpretability. Instead of looking only at which input variables influence an output, researchers examine what a model has learned internally. In weather and climate applications, this can mean investigating whether the internal representations of a model correspond to recognizable physical structures or phenomena.
A third path is equation discovery: using data to identify interpretable mathematical relationships rather than relying entirely on opaque predictive systems.
These approaches involve trade-offs. A fully interpretable model may not always deliver the same predictive performance as a large black-box system. A hybrid model may explain only part of the process.
But the objective changes and the question is: What kind of model gives us knowledge we can scientifically defend?
Bringing Causality Into Explainable AI
This is where Camps-Valls sees one of the most important frontiers: the intersection between explainability and causality. Standard machine learning is extremely good at detecting statistical relationships. Causal inference asks a different set of questions.
What happens if we intervene? What would have happened if something had been different? Which variables actually drive the system, rather than merely predict it?
Camps-Valls described research moving in both directions: using explainability to help formulate causal hypotheses, and using causal knowledge to make explanations more meaningful.
Consider counterfactual explanations. An AI system might suggest that a prediction would change if a person’s age were lower. Mathematically, that change may be possible inside the model. In the real world, however, time only moves in one direction.
A meaningful counterfactual therefore needs constraints grounded in reality. Structural causal models and causal graphs can provide some of those constraints. Instead of allowing every variable to interact freely, researchers explicitly represent assumptions about which variables influence which others.
That structure does not remove uncertainty, bumakes the assumptions visible. And for Camps-Valls, making seams visible may be preferable to presenting a perfectly smooth explanation that hides where knowledge ends and inference begins.
Think, Act, Model
Behind Camps-Valls’s critique of XAI is a broader critique of how data-driven science can be practiced. The familiar workflow often begins with modeling. Researchers collect data, train a machine-learning system, obtain a prediction and then ask an explainability tool to interpret what the model has learned.
Camps-Valls wants to reverse that order. He calls his framework TAM: Think, Act, Model.
First, think.
Before choosing an algorithm, ask the causal question. Why might this phenomenon occur? What mechanisms could connect the variables? Which factors might be missing? What would a plausible causal graph look like?
Then, act.
Ask what an intervention would mean. What response should follow from changing one part of the system? Which assumptions can be challenged or tested?
Only then, model.
At that stage, researchers can decide whether the problem calls for a machine-learning model, a hybrid approach, a causal model, an interpretable equation, or several models that can be compared against one another. The order matters because it changes what explanation is for.
Instead of using XAI after the fact to justify a model, explanation becomes part of the scientific reasoning that precedes and constrains the model.
It also requires collaboration. Camps-Valls argues that this type of work cannot belong only to computer scientists. Domain experts are needed to identify plausible mechanisms, challenge causal assumptions and recognize when a statistically convincing explanation makes little sense in the system being studied.
Explaining Less, Understanding More
Camps-Valls deliberately frames his argument provocatively. He is not calling for researchers to abandon explainable AI. He is asking them to be clearer about its limits.
Used carefully, XAI can help diagnose models, compare their behavior, identify unexpected patterns and generate new hypotheses. It can be part of a rigorous scientific workflow. What it cannot do is turn correlation into causation simply by producing an explanation. That distinction will become more important as machine-learning systems enter fields where prediction is only the beginning of the question.
In science, knowing that a model works matters.
Knowing why the world works that way is something else entirely.