New Research Finds Competition Can Make the Race for Artificial General Intelligence Less Safe

As artificial intelligence systems become increasingly powerful, policymakers around the world are grappling with a difficult question: How can society encourage innovation (and the now plausible race to truly intelligent AI) while managing potentially catastrophic risks?

A new National Bureau of Economic Research working paper by Ethan Bueno de Mesquita, dean and Sydney Stein Professor at Harris; Associate Professor Wioletta Dziuda, who also serves as Deputy Dean for Faculty and Research; and their co-author Mattias Polborn of Vanderbilt University suggests that the answer may depend as much on economics as on technology.

In "The AGI Race and Existential Risk," the researchers develop a model of competition among firms racing to develop artificial general intelligence (AGI), systems that could perform a wide range of cognitive tasks at or beyond human levels. Their analysis focuses on a central tradeoff facing the industry and policymakers: resources devoted to moving faster are resources that cannot be devoted to making advanced AI systems safer.

"We started by thinking about market structure," said Bueno de Mesquita. "Should there be one firm? Should there be many firms? Eventually, the paper became more broadly about understanding the incentives driving safety and speed, and what policy levers governments might use to shape those incentives."

The model begins with two assumptions. First, firms believe there is enormous value in being first to achieve AGI. Second, while AGI could generate tremendous benefits, many researchers and industry leaders also acknowledge the possibility (however uncertain) of catastrophic, existential outcomes if advanced systems are developed or deployed unsafely—a possibility referred to, quite directly, as “doom”.

One of the paper's central findings is that a high degree of competition can make the race to AGI riskier. As more firms enter the market, each firm devotes a larger share of its resources toward speed and a smaller share toward safety. The result is faster development and a higher probability of harmful outcomes.

"The more firms you have, the riskier the race becomes as the firms try to outpace one another," Dziuda said. “Our model challenges the assumption that more competition necessarily produces better outcomes. While competition often benefits consumers and spurs innovation, we show that in a race where being first carries enormous rewards, competition can also create incentives to cut corners on safety.”

In effect, the paper describes a classic collective-action problem. Individual firms may prefer a slower and safer race, but competitive pressure makes it difficult for any one company to slow down on its own.

Researchers show that firms can sometimes benefit from credible commitments to slower, safer development. If one company can convincingly commit to investing more in safety and moving more cautiously, competitors should respond by slowing down as well. The result is a safer race overall—and one that can leave all participants better off.

In other words: being the firm that brings about doom is bad for business, but the negative consequences affect everyone, whether your firm brought about doom or not.

Perhaps more surprisingly, the model suggests there may be circumstances in which firms continue racing even when the expected value of achieving AGI has become negative.

"Why would they still race?" Dziuda asked. "Because other firms are racing. If a catastrophic outcome occurs, whether they're in the market or not, they're affected anyway. So, they may as well participate and hope to be the one that wins."

That dynamic helps explain a conundrum that has emerged in recent years: some leaders of AI companies have simultaneously warned, very publicly, about the risks of advanced AI (think, for instance, of HAL 9000 in 2001: A Space Odyssey, or other potentially ruinous outcomes) while also pushing aggressively to develop it. The model helps explain why some AI leaders have simultaneously called for stronger regulation while investing billions of dollars in development. systems.

“These findings offer a new way to think about public calls for AI regulation,” Bueno de Mesquita added. “Calls for industry-wide rules need not be interpreted solely as acts of public-minded restraint. In some circumstances, firms may support regulation because common constraints reduce pressure to sacrifice safety in order to keep pace with rivals.”

The paper also examines several policy interventions that have become central to debates about AI governance, including restrictions on computing resources, industry consolidation, public investment, and government participation in AI development.

The authors find that the effects of these policies are often complex and inter-connected.  For example, many proposals focus on limiting access to computing power or other key inputs. But the researcher’s model suggests that resource restrictions are not always helpful. In some market situations, especially one where there are fewer players, providing firms with additional resources may actually improve safety outcomes by allowing developers to devote more resources to safety alongside capability development.

"We discovered that the answer isn't always to tax or restrict resources, it can be a combination of policies that seek to limit the number of competitors and then offer support so that they have room to pursue both safety and new capability,” Dziuda explained.

The working paper also finds merit in publicly supported AI development. A government-sponsored AI project designed to prioritize safety rather than speed could improve overall outcomes by providing a safer pathway to innovation while encouraging private competitors to behave more cautiously. Switzerland is now advancing a similar model.