Dario Amodei, the CEO of Anthropic, has become one of the most prominent voices calling for independent oversight of artificial intelligence development. His proposal centers on giving outside watchdogs ongoing access to AI companies so they can verify safety practices and hold developers accountable.
Who Is Dario Amodei?
Amodei’s path to AI leadership began in the sciences. He studied biophysics and earned a Ph.D. from Princeton in 2011, then became a postdoctoral scholar at the Stanford University School of Medicine. He later moved into the technology industry, working at Baidu, Google Brain, and OpenAI.
At Baidu, Amodei focused on developing speech recognition through machine learning, a form of pattern identification. At Google, he worked on safety while helping develop the company’s neural-net research, computer models that loosely mimic brain function.
In 2016, Amodei joined OpenAI, where he continued to work on safety and eventually became vice president of research during the development of the ChatGPT 2 and ChatGPT 3 models. In those early stages, the GPTs were asked to fill in blanks in sentences like “Today, I went to the ______ and bought some milk and eggs,” and “I knew it was going to rain, but I forgot to take my ______.”
Why Amodei Left OpenAI
Amodei departed OpenAI in 2020, just as the company was discovering it could amplify its models’ power through larger and larger language models. He believed OpenAI was not doing enough to install guardrails on what he saw as a budding reality of the technology he had long theorized about. He also questioned whether he could trust the company to set aside its financial interests.
In an interview with Bloomberg earlier this year, Amodei described the difficulty of continuing to work with a company when trust breaks down.
“When you feel that you can’t trust someone, when you feel that their values are not what they say they are, when you feel that they’re not honest, when you feel that they’re not in it for the reasons that they say, when you see disturbing patterns of behavior, dishonesty, that makes it very hard to continue to work with a company, to continue to trust the company,” he said.
Building Anthropic Around AI Safety
After leaving OpenAI, Amodei helped start Anthropic, a company that has made AI safety a central part of its identity. Anthropic even has a sort of constitution for its flagship AI, a guiding document that lays out boundaries for its research.
“Anthropic wants Claude to be genuinely helpful to the people it works with or on behalf of, as well as to society, while avoiding actions that are unsafe, unethical, or deceptive,” the company wrote.
That directive has led Anthropic to clash with the U.S. Department of Defense over developing tools that could be used to autonomously target humans on the battlefield. The company also refused to do work that, in its estimation, amounted to mass surveillance. Although that tension cost Anthropic a $200 million contract, Amodei believes it is time for the industry to take similar stances, especially as some companies began to report trouble controlling their own agents.
The Case for Embedded Evaluators
Amodei is not calling for the rest of the industry’s leaders to submit specifically to the thinking of METR, an AI safety testing laboratory. But in a recent letter, he used METR as an example of the kind of guidance he believes the industry needs.
He proposed that accountability could come through “embedded evaluators” that would supervise AI development companies.
“Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR) whose role it is to verify adherence to safety practices and commitments, report incidents and help assess the alignment of not just completed AI models but training pipelines and processes,” Amodei wrote.
He argued that regardless of what commitments companies make, the public deserves to know what is going on.
“We are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic,” he wrote.
METR and Its Ties to Effective Altruism
METR describes itself as an AI safety testing laboratory. “METR evaluates frontier AI models to help companies and wider society understand AI capabilities and what risks they pose,” its website reads.
Although its website and public materials do not mention effective altruism, its founders have used it as a framing for their work. Effective altruism is the belief that evidence and careful reasoning can help maximize the good that companies and people can do with their time and resources.
Beth Barnes, METR’s founder and CEO, worked at OpenAI alongside Amodei as the company developed early versions of ChatGPT. At an Effective Altruism Global event, she laid out her vision for maintaining AI safety.
“Our overall plan is — it sort of seems like it would be good if it was someone’s job to look at models and decide if they’re going to kill us, think through the ways that that might happen, anticipate them, figure out what the early warnings would be, that sort of thing,” Barnes said.
Paul Christiano, who led research around OpenAI’s efforts to ensure its models followed acceptable strategies for delivering requested results, later founded the first iteration of METR. He too has framed parts of his approach to AI safety as a form of effective altruism.
“My suspicion is that it is more important for the ‘effective altruism’ movement to have a fundamentally good product and to generally have our act together than for it to grow more rapidly,” Christiano said in a 2014 article.
Barnes and Christiano provide informal links to Anthropic, a company that received funding from effective altruism’s largest supporters when it emerged as a way of thinking among tech moguls. Both also worked on evaluations involving Anthropic models, providing safety checks for Anthropic’s flagship AI, Claude.
Funding Ties to Effective Altruism
Sam Bankman-Fried, the founder of the cryptocurrency exchange FTX, led Anthropic’s 2022 Series B financing round before his company collapsed and he was convicted in a multibillion-dollar fraud case. Before FTX’s downfall, Bankman-Fried was one of the highest-profile proponents of the effective altruism movement, publicly saying it shaped his approach to earning and giving money.
Similarly, Skype co-founder Jaan Tallinn led Anthropic’s 2021 Series A financing round. Tallinn has been one of the effective altruism movement’s most prominent supporters, speaking at Effective Altruism Global conferences, helping found the Centre for the Study of Existential Risk and the Future of Life Institute, and donating more than $1 million to the Machine Intelligence Research Institute, an organization focused on AI safety and alignment research.
A Continued Belief in Safe AI Progress
Amodei continues to believe that AI capabilities can grow safely, but only if the industry sets standards for how it achieves that safety.
“I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed. But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right,” Amodei wrote in his letter.
