Inverse Game Theory Could Make AI Follow the Rules

As AI capabilities grow, security researchers are looking for ways to ensure its alignment with human desires. So far, the work has gone in two directions: peering into the model's 'brain' or training it in values – Claude even has its own constitution. Both methods are weak: models underperform on tests, conceal intentions, and mislead evaluators.

How mechanism design works

Game theory predicts the behavior of rational agents given certain rules, while inverse game theory designs rules that elicit desired behavior. A classic example is cake-cutting: one cuts, the other chooses first, so the cutter is motivated to divide equally. Other examples include surge pricing in taxis and second-price auctions.

In AI, mechanism design assumes that we may never understand the internal workings of models. Therefore, mechanisms need to be built that encourage correct behavior regardless of what happens inside.

What researchers propose

In a preprint, Stephen Morris, an economist at the Massachusetts Institute of Technology (MIT), with colleagues from Yale and Columbia University, proposed mechanisms against intentional underperformance. The model is rewarded not only for success in the test but also for how accurately its answers predict the answers of a second model. Thus, playing 'dumb' becomes unprofitable.

Elad Hazan, a professor at Princeton, suggests taking a cue from human society. His mechanism: agents alternately act as 'solvers' and 'auditors,' and both sides are penalized for failure. In case of disagreement, the matter is referred to a 'court' where a human decides. Hazan acknowledges the risk of collusion but suggests paying auditors enough to make collusion unprofitable.

He relies on the work of Nobel laureate Gary Becker on the 'rational criminal': people break the law when the benefit outweighs the punishment. Alignment, according to Hazan, is not a property of the model but a description of its behavior at a moment in time.

Doubts

Helen Qu from the Flatiron Institute doubts whether it is reasonable to assume that AI responds to incentives. Morris agrees: 'AI is not rational. It's just algorithms.' But, as with people, treating them as rational can help control them. Qu sees mechanism design as just one layer of a 'deep defense' strategy. It's unclear how theories will behave on real models and whether they will find loopholes.

The final question is the free market. An aligned model may be beneficial to the world, but if it is slower or more expensive, there is little incentive to release it. 'This mechanism is responsible for enormous human prosperity,' says Hazan. 'But it could be responsible for something very terrible.'