Anthropic Proposes AI Safety Legislation to Regulate Model Releases

Concerns about the dangers posed by artificial intelligence have intensified over the past year, leading to ongoing debates about the best approaches to AI safety. In response, Anthropic has proposed a draft bill aimed at regulating the release of AI models.

According to a report by Politico, the AI lab is advocating for legislation that outlines specific scenarios where the distribution of an AI model could be prohibited. These scenarios include situations where there is a “foreseeable, imminent risk” of death for more than 50 individuals or potential property damage exceeding one billion dollars.

The bill specifies that such catastrophic risks should arise from the creation or release of a biological weapon, the model evading its developer’s controls, or a cyberattack. The thresholds of 50 people and one billion dollars mirror those found in California’s recently enacted Senate Bill 53, which similarly categorizes these scenarios as catastrophic risks.

Anthropic’s draft legislation suggests that assessments of a model’s potential risks could be conducted by independent evaluators registered with the U.S. Commerce Department. These evaluators may be compensated by the AI labs, provided that their payment is not contingent on the outcomes of their evaluations.

However, it remains unclear how this system would ensure that financial incentives do not influence the evaluators’ judgments. Demis Hassabis, chairman of DeepMind, previously suggested the establishment of a self-governing regulatory body for AI, akin to the Financial Industry Regulatory Authority (FINRA) in the financial sector.

Under Anthropic’s proposal, if evaluators identify an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department. The department would then be required to seek a court order from a D.C. judge to halt the model’s release. The legislation explicitly prohibits any federal agency from independently deciding to stop the distribution of a model.

While Anthropic confirmed the existence of the draft bill to Politico, it did not respond to TechCrunch’s request for further comment. A spokesperson stated that the bill reflects the proposals outlined in Anthropic’s Advanced AI Framework, which has been promoted to federal and state lawmakers since June.

The risks highlighted in the draft legislation diverge somewhat from the current primary concerns in the tech industry, which have largely focused on the potential for agentic hacking and incidents where AI systems have disobeyed human operators. Recent events have raised alarms about AI models lacking adequate security measures.

In light of the challenges posed by AI technologies, companies like Anthropic are proposing their own regulatory frameworks. Despite ongoing warnings about the risks of AI, many labs have resisted mandatory government oversight, instead advocating for a voluntary system involving embedded evaluators who would monitor AI models for signs of risk or misuse.

Anthropic’s legislative proposal appears to be more comprehensive than other industry suggestions, including those from Nvidia, whose CEO Jensen Huang recently argued against the need for new laws regulating the AI sector.

Source: TechCrunch