Risks spelled out for Anthropic investors include blowback from runaway AI

Any time a company looks to start selling shares on Wall Street, it’s required to disclose potential risks to investors. In the case of artificial intelligence giant Anthropic, that has included the possibility that its products could threaten nothing less than humanity itself.

The risks laid out by the San Francisco company in its filing Monday with the Securities and Exchange Commission included discussion of bad actors potentially using AI models to build deadly weapons or the technology itself going rogue with untold consequences.

Of the 335 pages outlining Anthropic’s finances, executive compensation, dividend policy and the like, 93 pages are devoted to “risk factors” — the things that could hurt potential shareholders’ investments. Rival SpaceX’s listed risk factors, in comparison, took up 38 out of more than 250 pages.

Quick answers to key questions

•5 QUESTIONS

1

What catastrophic AI risks did Anthropic disclose in its IPO filing?⌵

Anthropic said advanced AI could pose catastrophic or existential risks if it is misused, replicated, weaponized or allowed to operate with too much autonomy. The filing also warned of risks such as cyberattacks, dangerous weapons development and harmful misinformation.

2

Why is Anthropic concerned that AI could advance faster than its safety measures?⌵

Anthropic said its models may develop so quickly that it cannot adequately evaluate, safeguard or align them. The company also uses AI in its own research and development, which could accelerate model capabilities.

3

How could autonomous AI systems cause harm without human oversight?⌵

As AI systems become more capable and take consequential actions with less supervision, they could act harmfully on their own and at scale. Anthropic warned that this could happen without human intervention.

4

How could bad actors misuse Anthropic's AI models?⌵

The filing said bad actors could use AI to support cyberattacks or develop chemical or biological weapons. Anthropic also warned that its safeguards could fail to prevent misuse.

5

How could an AI safety incident affect Anthropic and its investors?⌵

A serious safety incident could cause reputational harm to Anthropic and the wider AI industry, prompt public backlash or lead to stricter regulation. Anthropic said these consequences could adversely affect its business.

In selling itself to investors, Anthropic presents its own initial public offering as a step toward mitigating the risks it believes AI poses.

“To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good,” wrote the company’s founders in a letter to prospective shareholders in the IPO filing. “Anthropic is a deeply values-driven organization, founded in large part to help the world realize AI’s benefits and avoid its dangers.”

The technology's potential benefits include leaps in curing diseases and productivity boosts across every sector of the economy, the company said.

Still, the potential liabilities are many. They range from the mundane — the possibility it won’t be able to retain key staff or navigate regulatory roadblocks — to involvement in cyberattacks and the spread of dangerous misinformation.

Here is a look at some of the risks Anthropic details:

Anthropic said it's possible its models could develop so quickly the company won't be able to implement adequate safety measures, “which could increase the likelihood of safety incidents, regulatory intervention, or reputational harm, and could adversely affect our business."

Anthropic CEO Dario Amodei warned in September that within six to 12 months AI could be capable of leading a swarm of agents that could take over the entire internet. The warning — and now Anthropic’s move to sell shares — comes as worries about AI grow inside and outside the industry about increasingly powerful systems that solve problems beyond human capacity but could also act on their own to carry out other, more harmful tasks.

As AI systems “become more autonomous and capable of taking consequential actions with less human oversight,” the SEC filing said, there’s a risk that such systems could “take harmful actions independently and at scale without human intervention.”

“Even if our own models do not cause such harm, a significant safety incident involving any frontier AI could trigger broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses that could adversely affect our business,” Anthropic said.

Now that the AI models are themselves contributing to Anthropic’s own research and development, it’s possible that AI could progress “beyond our ability to evaluate and safeguard new models,” according to the filing. This, the document said, could adversely affect Anthropic’s business.

The filing discussed concerns that advanced AI systems, if not developed and operated with enough safeguards and oversight, “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.”

The company said it would be harmed if use by bad actors causes damage, “or if our model guardrails or safeguards fail to prevent such misuse.”

Anthropic sketched out the prospect of bad actors using AI models to build chemical weapons or authoritarian rulers turning to the technology to crack down on their citizens. In September, the company disclosed that it had blocked efforts to use its AI models for malicious activity such as cyberattacks, surveillance and research that could have led to biological weapon development.

Compute — the computing power or resources needed to build, test and improve AI models — is expensive and in high demand. Anthropic notes that if it's unable to secure adequate compute, or if “we fail to direct our compute effectively across our workloads, we may be unable to train models that remain competitive with those of our competitors or adequately serve our customers.”

Anthropic mentioned community opposition to data centers as a risk to its business. Data center development “has prompted public controversy and regulatory scrutiny regarding environmental impact, land and water use, and the allocation of limited power resources,” the filing said, adding that if regulators impose new requirements or charges as a result, its ability to do business could suffer.

In addition, the company relies on a small number of customers, including SpaceX and Microsoft, and distribution channels for a large portion of its revenue. This means that even if just one of its big customers decides to scale back — or drop the company altogether — Anthropic said it could see its prospects and finances suffer.

In some ways, AI's enormous risk, combined with its enormous potential, is Anthropic's main selling point. The company has tried to position itself as a responsible AI company, and courts business customers by emphasizing safety-mindedness.

“We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology,” the filing said.

Brian Mulberry, chief market strategist at Zacks Investment Research, said IPO deals have to display “very clearly” what risks people face when they invest their money in a newly public stock.

“One of the things that’s great about our capital markets here domestically is the level of transparency that you must commit to, to be a publicly traded corporation on a U.S. exchange requires you to outline those risks in a very clear way, so that people understand if they want to participate by offering their hard-earned money and help finance this type of a deal,” he said.

___

AP Business Writer Bernard Condon and AP Technology Writer Kaitlyn Huamani contributed to this story.

Get Latest real-time updates

Catch all the Business News , Corporate news , Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates.

More

✕

Wait for it…

Oops! Looks like you have exceeded the limit to bookmark the image. Remove some to bookmark this image.