AI Is Doing What We Asked. That May Be the Problem.
AI systems are usually trained to optimize a goal. Accuracy. Reward. Performance.
But Luiz Chamon argues that trustworthy AI requires something more explicit: requirements that define, from the beginning, what a system must also satisfy, from robustness and fairness to safety and prior knowledge.
Luiz Chamon
Assistant Professor and Hi! PARIS Chair Holder, École polytechnique (Institut Polytechnique de Paris)
Hi! PARIS Summer School 2026 · 5–6 minute read
The essential points
- AI systems often optimize a single objective, while requirements such as fairness, robustness or safety are added later through penalties and regularization.
- Constrained learning offers a different approach: specifying those requirements directly and allowing the learning process to satisfy them alongside its main objective.
- Chamon’s work shows that, despite the added constraints, learning under requirements can be statistically comparable in difficulty to conventional learning in important settings.
- The deeper challenge is not only technical. Defining requirements forces us to decide explicitly what we expect AI systems to do, and what behaviors we are unwilling to trade away.
An AI model is trained to maximize accuracy. Then someone discovers that it behaves differently across groups. So a fairness penalty is added. Another test reveals that small changes to an input can destabilize its predictions. Add a robustness penalty.
Then come safety, prior knowledge, uncertainty or operational rules. Soon, one learning objective is carrying a growing collection of competing goals, each with its own weight to tune.
For Luiz Chamon, Assistant Professor and Hi! PARIS Chair Holder at École polytechnique, this familiar approach points to a more fundamental problem.
Perhaps the system is not failing to do what we asked. Perhaps we asked for the wrong thing.
As Chamon put it during the Hi! PARIS Summer School, “AI is doing exactly what we asked for, even if it’s not what we wanted.” His alternative is to bring a concept long familiar in engineering back into machine learning: requirements.
Instead of teaching a system one objective and trying to correct its behavior afterward, why not state from the beginning what it must satisfy?
AI Is Doing What We Asked
Machine learning has been built around a remarkably powerful idea: collect data, define a loss and optimize it.
In supervised learning, that might mean minimizing prediction errors. In reinforcement learning, it might mean maximizing a reward accumulated through interaction with an environment. The approach works because it turns complex problems into something algorithms can optimize.
But a system and an objective are not the same thing. Real systems rarely have only one expectation placed on them. A model may need to be accurate and robust. A decision-making system may need to perform well and remain safe. An application may need to use data effectively and respect knowledge researchers already have about the problem.
Trying to compress all of those expectations into one number can create unexpected behavior.
This is closely related to phenomena such as reward hacking: a system becomes very good at satisfying the proxy it has been given without necessarily producing the behavior its designers intended.
For Chamon, that does not necessarily mean the optimizer malfunctioned. It may mean the specification was incomplete.
A Requirement Is Not Just Another Objective
The distinction Chamon draws comes from systems engineering.
A goal describes something we want to improve: greater accuracy, lower cost, faster performance.
A requirement describes something the system must satisfy. That difference matters.
Suppose an AI system should achieve the highest possible accuracy while maintaining a specified level of robustness. In a conventional approach, both might be placed inside the same objective, with a coefficient deciding how much robustness matters compared with accuracy.
Change that coefficient, and the balance changes. The difficulty is that finding the right coefficient can require extensive experimentation and cross-validation. Add several requirements and the number of choices grows quickly.
Constrained learning reframes the problem.
Instead of saying, in effect, “care this much about robustness,” the designer specifies “achieve at least this level of robustness” and allows the learning procedure to determine how strongly that requirement needs to influence training.
The distinction may sound mathematical. It changes the question being asked.
A penalty expresses a preference.
A constraint expresses a requirement.
When Requirements Compete
Adding constraints does not make the underlying tensions disappear. An accurate model may not automatically be robust. Improving one property can make another harder to achieve. And some requirements may conflict strongly with what the available data naturally encourages the model to learn.
Chamon describes this as requirement difficulty: how far a requirement pulls the system away from the objective it would otherwise pursue. That creates a three-way relationship between the complexity of the model, the amount of available data and the difficulty of the requirements.
Yet one of the striking results behind his work is that constrained learning does not necessarily introduce the dramatic statistical burden that might be expected. Under the framework he presented, requirements can in important settings be learned with essentially the same order of sample complexity as the underlying learning problem.
In simpler terms: asking a model to satisfy requirements does not automatically make learning prohibitively harder.
“If we can solve the problems individually, then let’s solve them together,” Chamon argued in our interview.
The point is not that constraints are free. Difficult requirements can require more data or richer models.
The point is that incorporating them directly is aviable learning problem, not merely an ideal to be handled after training.
From Fixed Penalties to Adaptive Constraints
The difference becomes clearer in Chamon’s examples on robustness.
Image classifiers can perform extremely well on normal images and fail dramatically when inputs are altered in carefully designed ways that are barely noticeable to a person. One response is adversarial training: expose the model to perturbed examples during training so that it learns to resist them.
But how much weight should robustness receive?
A fixed penalty assumes that the answer stays roughly the same throughout training. Chamon’s constrained approach allows that importance to change.
Early in training, the robustness requirement may need to exert strong pressure on the model. Later, once training has moved the system toward a more robust region, that pressure can fall substantially.
The algorithm adapts instead of requiring the researcher to design that schedule manually. This is one of the practical attractions of constrained learning: it does not eliminate human choices, but it changes which choices humans need to make.
Researchers still have to decide what level of robustness, fairness or safety they require.
But they no longer need to translate every requirement into an arbitrary fixed weight and hope that the resulting combination behaves as intended.
When AI Starts Acting in the World
The stakes become higher in reinforcement learning. Here, an AI system does not simply classify an image or predict an outcome. It selects actions, receives feedback and continues interacting with an environment.
A navigation system, for example, might be rewarded for reaching a destination quickly.
But speed is not the only thing that matters. It may also need to avoid collisions.
Traditional reinforcement learning often tries to encode such considerations into a single reward. Yet Chamon shows that there are decision-making problems for which a single reward cannot cleanly express the behavior we want.
Constraints can.
A system can be asked to maximize performance subject to a safety condition, a minimum service level or another explicit requirement.
That makes constrained reinforcement learning not simply another way of solving the same problem. In Chamon’s framework, it can represent some problems that ordinary reward-based reinforcement learning cannot represent in the same way. This matters because reinforcement learning systems interact with their environments directly.
“Trustworthiness in the context of reinforcement learning […] is even more important,” Chamon said in our interview, precisely because there may be no separate barrier between the learned policy and the actions it takes.
Safety cannot always be an afterthought when the system itself is acting.
Trust Is Something We Build
This leads to the larger idea behind Chamon’s work. Trustworthiness is often discussed as though it were another metric an AI model could maximize.
For Chamon, it is closer to the consequence of repeatedly meeting a collection of expectations. “Trust is something that is not a property of the system. It’s something that you acquire with time,” he said.
A trustworthy system may need to be accurate, robust, fair and safe, not one of those things in isolation. Constraints provide a technical language for expressing those expectations separately rather than hiding them inside one aggregate objective.
But Chamon sees an even more fundamental benefit. When designers are required to specify constraints, they are forced to ask which constraints matter.
What level of safety is acceptable? What notion of fairness is relevant? Which prior knowledge should the model respect? What behavior should never be sacrificed simply because another metric improves?
The technical act of writing requirements therefore becomes an intellectual and organizational act as well.
By forcing ourselves to specify what a system must satisfy, Chamon argues, we are also forced to think about what we actually want the system to do.
That may be the most consequential shift. Building AI around requirements puts more responsibility on its designers. They have to decide from the start which behaviors matter, rather than trying to fix them later.