A new study published in the Proceedings of the National Academy of Sciences argues that weak artificial intelligence safety regulations may produce outcomes more dangerous than no regulation at all. Drawing on game theory and theoretical economics, researchers from Cornell University and Carnegie Mellon University created a model to determine the most effective regulatory approach. Their conclusion: regulations must be strict and apply to every entity in the AI supply chain—from model developers like OpenAI, Google, and Anthropic, to downstream companies that integrate AI into real-world applications such as medical diagnostics or customer service chatbots.
If regulators focus solely on downstream firms—the companies that apply AI in specific contexts—the general-purpose AI developers tend to cut corners on safety measures, including third-party audits. They assume that the downstream companies will pick up the slack, creating a "free-riding" dynamic. Benjamin Laufer, the study's principal author, explained: "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." This offloading can reduce overall safety, as general-purpose developers have less incentive to ensure their models are robust before deployment.
The study arrives amid intense debate in the United States over how to govern artificial intelligence. Two primary camps have emerged. One faction, often described as anti-regulation technologists, advocates for minimal federal guardrails. This group, aligned with the Trump administration's approach, insists that heavy regulation would stifle innovation and hinder the U.S. in the global AI race against China. They view stricter safety rules as unnecessary hindrances that could slow progress. The opposing camp pushes for comprehensive federal regulation, arguing that the AI industry, motivated by profit, systematically underestimates or downplays the risks of uncontrolled development. These risks, they warn, range from AI hallucinations that cause real-world harm, to broader societal impacts like job displacement, data center pollution, and threats to democratic processes.
The authors of the study suggest that safety and revenue need not be mutually exclusive. Under the right regulatory framework, both can be achieved simultaneously. Their model shows that when regulators require both general-purpose AI producers and downstream specialists to invest sufficiently in safety—meeting meaningful standards—the overall utility for all players improves. Utility is defined as revenue share minus investment cost. The situation mirrors the classic prisoner's dilemma from game theory. In this dilemma, two rational actors can choose to cooperate or betray. If both cooperate, they achieve the best possible outcome. But without knowledge of the other's choice, each is tempted to betray to secure their own benefit, resulting in a worse outcome for everyone. Strict regulation that applies across the supply chain forces cooperation, eliminating the option to free-ride and ensuring that both parties invest in safety, leading to the most favorable results.
The historical context of technology regulation provides valuable parallels. For example, pharmaceutical regulations require rigorous testing of both raw ingredients and final products, ensuring safety at every stage. Similarly, aviation safety mandates compliance from parts manufacturers to airlines. In AI, a fragmented approach—regulating only the final application but not the underlying model—leaves critical gaps. General-purpose models developed by companies like OpenAI, Anthropic, and Google are increasingly used as foundational tools by hundreds of specialized firms. If the model itself is not fundamentally safe, any downstream application built upon it inherits those flaws. The study highlights that even if downstream companies conduct additional safety checks, they may not have the resources or access to fully audit the model's training data, inherent biases, or potential emergent behaviors.
Critics of the study might argue that targeting upstream developers is logistically difficult because these models are often proprietary and their inner workings opaque. However, the researchers counter that third-party audits, transparency requirements, and shared liability can be effective tools. They point to emerging frameworks like the European Union's AI Act, which imposes different levels of obligations based on the risk classification of AI systems. The study suggests that similar measures, if strictly enforced across the entire supply chain, could prevent the free-riding problem. Without such comprehensive regulation, the theoretical model predicts a race to the bottom, where safety is compromised for speed and cost savings.
The broader implications of this research extend to the ongoing policy discussions in Washington. The Trump administration has favored a light-touch approach, issuing executive orders that encourage voluntary safety commitments rather than binding regulations. Industry leaders have largely supported this stance, arguing that it allows for rapid innovation. However, the study's authors caution that voluntary commitments are insufficient because they do not create enforceable obligations. In the prisoner's dilemma analogy, voluntary pledges are akin to promises without mutual assurance—each player still has an incentive to defect. Only binding, verifiable regulations can guarantee that both parties invest adequately in safety.
Furthermore, the study examines the concept of "regulatory capture," where the regulated industry influences the regulatory process to its advantage. The pro-innovation camp often accuses advocates of strict regulation of attempting to stifle competition, but the researchers note that weak regulation can also serve the interests of dominant firms. Large AI developers may prefer minimal oversight because it allows them to maintain market dominance without incurring the costs of safety compliance, while smaller companies might struggle to keep up with any regulatory burden. The optimal approach, according to the model, is one that levels the playing field by imposing uniform safety expectations on all players, from the largest labs to the smallest startups.
Real-world incidents have already demonstrated the dangers of inadequate AI regulation. In healthcare, AI diagnostic tools have produced false positives or missed critical conditions because the underlying models were not properly validated for specific patient populations. In finance, biased algorithms have denied loans to minority applicants, amplifying existing inequities. These failures often stem from the disconnect between general-purpose models and their specialized applications. The study argues that addressing these problems requires a holistic regulatory strategy that holds model developers accountable for the fundamental safety of their products, not just the final use cases.
Laufer emphasized that "people think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology. To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity." This perspective challenges policymakers to think beyond siloed approaches and to design regulations that align incentives across all levels of AI development and deployment.
As the debate over AI governance continues, this study provides a crucial theoretical foundation for understanding the unintended consequences of weak regulation. It suggests that half-measures may not only be ineffective but actively harmful. The lesson is clear: effective AI safety requires strict, enforceable, and comprehensive rules that target every actor in the supply chain. Anything less could leave society more vulnerable than if we had no rules at all.
Source: Gizmodo News