Anthropic co-founder says regulators may need to require AI shutdown systems

Anthropic co-founder Jack Clark told the BBC that governments may eventually need to require AI companies to maintain independently verifiable systems for shutting down dangerous software. He said the technology’s rapid development carries immense risks, while acknowledging that most major laboratories already have ways to stop their systems.

Jack Clark, a co-founder of Anthropic, has said governments may eventually need to require artificial intelligence companies to maintain a “kill switch” capable of shutting down software deemed too dangerous. Speaking to the BBC, Clark said the system should potentially be checked by an independent third party.

Clark said most AI laboratories, including Anthropic, have “different ways” to pull the plug on their systems. However, he argued that the details of any requirement, including how shutdown mechanisms would be verified, should form part of the wider policy debate over artificial intelligence.

“Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?” Clark asked, according to the BBC. He said society might eventually decide to establish rules covering those questions, rather than leaving safety arrangements entirely to companies.

His comments come amid growing disagreement within the AI industry about the risks posed by increasingly capable systems. Anthropic chief executive Dario Amodei has called for the pace of development to slow and for closer monitoring, while saying any measures should be introduced without sacrificing commercial advantage.

Anthropic has also faced internal debate over the possibility of extreme AI harms. The BBC reported that former researcher concerns about artificial intelligence potentially causing human extinction had drawn attention, while Anthropic scientist Evan Hubinger said he personally assessed that possibility at more than 10% within the next decade. Computer scientist Geoffrey Hinton separately described a 10% risk as not unreasonable.

Other technology executives have challenged such warnings, saying claims about AI destroying humanity may be exaggerated or serve companies’ commercial interests. Clark declined to assign a percentage to the risk, telling the BBC that such statistics were not especially useful. He nevertheless said allowing AI to remain a “totally unregulated industry” was a bad idea and warned that society was “rolling dice with immense risks.”

US lawmakers have proposed legislation known as the Kill Switch Act, which would require companies to provide ways to shut down or limit problematic AI tools and could give government agencies authority to demand action. The UK government has rejected the idea of a mandatory kill switch, saying it would not prevent systems from being developed or misused elsewhere.