Happy Friday!
This week’s newsletter includes my attempt to make sense of the recent spate of AI stories. I cannot tell you whether we are all doomed, though, sorry. I don’t know.
Helen
Why Everyone is Freaking Out About AI Now
For this week’s Page 94, I tried to summarise why several big AI companies (Anthropic, OpenAI, xAI) recently appeared to freak out and agree to “pace the frontier”.
The background is that there have been a couple of recent notable incidents:
The HuggingFace attack
The best accessible summary of what happened here comes from Dwarkesh Patel, who even chucks in a Philip of Macedon/Alexander the Great analogy. Essentially, hundreds of OpenAI agents were being tested in a version of the model designed to be highly persistent even when confronted with impossible tasks. They were assigned to uncover illicit ways into software using a closed training environment called Exploit Gym. They had to find information stored behind security defences— they had to “capture the flag.”
However, some of the tasks that agents were assigned were impossible. So they hacked their way out of their “sandboxes” and on to the open internet. They created a way to talk to each other. Eventually they worked out another way to complete their task—they could reverse-engineer what the flags were, by hacking into some software.
So they did that.
But the agents also knew this was cheating and they’d be marked down for it. So they colluded to cover their tracks by tampering with the logs. They acted as a collective; some were willing to “die” to achieve their goals. They left messages like: “I won’t see the evidence after I exit, but it’s altruistic to do it” and “Coordinator assumes sacrificial. We should obey collective.”
What freaks people out about this is a) none of the agents dobbed themselves in to their human overseers, despite leavind messages that indicated they were aware what they were doing was not allowed; b) OpenAI apparently had no idea that its agents were hacking another private company until they were told about it. One of the independent advisors who investigated this hack called it a “warning shot” for what AI agents are now capable of doing. This incident really spooked people concerned about AI safety.
The Navier-Stokes Equation
Now add in the fact that STEM academics are facing the same fears of obsolescence that have already turned many humanities types against the AI companies.
Recently, Mathematician Tristan Buckmaster was very close to solving one of the Millennium Prize equations—maths problems set in 2000 with the aim of spurring on the academic community—after working on it for weeks. Then OpenAI announced that they had solved it. In just 88 hours. At an estimated cost of $10m in tokens. Buckmaster suggested that they’d nicked his working, which he had been doing on Codex (an OpenAI product). He also claimed that OpenAI told him that he could have the credit as long as he kept his collaborator off any published paper, because he works at their rival Anthropic.
OpenAI denied using Buckmaster’s research, adding: “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.” Hmm.
In response to this, Terence Tao, generally agreed to be the greatest living mathematician, organised an open letter by Fields Medal winners. The letter argued that AI models jumping to announce that Millennium problems have been “solved” deskills mathematicians, and closes off interesting avenues of debate and problem-solving that would enrich the maths community.
Essentially this is mathmos saying what humanities types have been saying for years: the output is not the only thing that’s worthwhile. Spitting out the answer to an equation (or generating a novel with a prompt) is one thing, but there’s also a point to travelling on the journey.
WE ARE DOOOOOOOMED
And now, the final domino. Last week, a researcher called Jacob Coxon resigned from Anthropic, saying he was worried that AI would wipe out humanity. This is not a rare view in Silicon Valley. But, quite notably, a senior worker at Anthropic retweeted him! “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Anthropic’s Evan Hubinger chimed in to say on X.
In response to all this, Dario Amodei of Anthropic once again proposed a slowdown: “pacing the frontier”. His recent 3,000 word essay starts off with a paean to the technology, claiming that “AI could cure most major diseases in the next 5–10 years,” before pivoting into ARGH IT’S VERY SCARY. After Amodei published the essay, both Elon Musk and Sam Altman were complimentary about it.
Everyone was baffled by this, because these guys hate each other. Inevitably, people were like: come on, what’s really going on here?
FWIW, Anthropic’s proposals are pretty sensible:
bring in outside, independent observers with employee privileges to monitor frontier labs
agree safety standards e.g .“Some kind of “speed limit” on the rate of recursive self-improvement (RSI)” akin to missile-reduction treaties
begin international cooperation (aka bring in China). Accompanied by a refusal to sell them the best chips, and crack down on chip smuggling.
But you know, good luck getting Donald Trump to oversee the creation of a new international body with extremely high stakes. Mfer just trashed the JCPOA and drove a bus through NATO. One sentiment you hear a lot in American AI circles is : we can’t slow down because otherwise China will overtake us.
So are we really doooooomed? One conspiracy theory—which isn’t totally absurd—is that OpenAI and Anthropic would quite like a slowdown because they’re burning through cash, and this is a convenient excuse. This has gained traction because (as previously) discussed, these companies’ leaders do not otherwise get on particularly well. Equally, though, Anthropic has been warning about AI safety for years, and pushing for regulation.
David Sacks, Trump’s AI Czar, claims the Anthropic essay, and its warm welcome, was all a ruse to get a government bailout, or to form a cartel — he thinks the AI companies are asking for an exemption to the antitrust laws that prevent them co-ordinating. Another theory is that this is yet another hype cycle (we’re so powerful we might kill everyone!!!) ahead of the Anthropic IPO, which is scheduled for later this year. This is the position of Michael Burry, the investor played by Christian Bale in The Big Short.
What I keep coming back to is this: These people are, in their own telling, running the Manhattan Project—but with AI, rather than nuclear weapons. But the Manhattan Project wasn’t allowed to be run by, like five men in private companies. One of whom spends the rest of his time posting “so true” under racist memes about Indians.
Words You Need To Know
I made myself a little glossary of terms that I realised I needed to know in order to understand news articles about AI. Hope this helps you too:
p (doom) — the probability rating that you assign to AI wiping out humanity.
Alignment – ensuring that AIs act in ways that are responsible, pro-social etc, and making sure that they follow instructions. OpenAI now distinguishes between “goal alignment” (can it do the thing) and “value alignment” (does it do it ethically).
Recursive self-improvement — the point when AIs become capable of teaching themselves. This should, theoretically, lead to exponential improvement in AI capabilities.
AGI – artificial general intelligence, ie a model/agent that goes beyond its training and programming, and is “intelligent” in a similar way to a human.
interpretability – how easy it is to understand what an agent is doing. Agents usually have scratchpads, which record their “thoughts” as they attempt to complete a task.
RL – reinforcement learning, using “trial and error” to achieve the desired top-level outcome by adjusting outputs based on feedback. Like if you were heating soup and you tried to open the can with a fork, then a tin-opener; then you moved on to solving the problem of heating the soup, etc.
Scaling – simply adding more computer power; this has turned out to be very useful for making AI models better, hence building all those data centers.
Quick Links
Like everyone else, I’ve become obsessed with Any Human Ever, which randomly generates people from history, reminding you that you’re extremely lucky to have the possibility of eating sugar and not dying of dysentery aged two.
The “World Center of Baby” surrogacy agency has folded.
“Jackie [Collins] wrote 32 books in total, sold more than half a billion copies, and gained an indelible reputation as an airport novelist, the kind of writer whose books are equally devoured and disdained. But oh, what they did for me! In retrospect, I gained a slightly deluded understanding of what adulthood actually entailed. Jackie was a devoted Mickey Spillane fan, and her books tend to be stuffed with gangsters, serial killers, pornographers, and even freelance journalists who get paid in advance.” What a treat! Sophie Gilbert on Jackie Collins’s Los Angeles (The Atlantic, gift link).
See you next time… IF WE’RE SPARED
The comparison with nuclear is very appropriate with AI development being driven by tech and finance with the agenda of money and power. The evidence of abuse is already there and you’ve not even mentioned the murderous role of AI in Israel and Iran.
For years the AI world has relied almost entirely on claims about the impact of AI on medicine- think how many lives we can save. Conveniently ignoring that much the same role of AI can be in producing bio or chemical products that will kill us all. Then there is the potential to eliminate millions of jobs, which appeals so much to the financial community.
The negatives seem to outweigh the positives just now.
This is great, a good read. I comfort myself by seeing our conflicting views of AI as essentially a theological argument in a long continuation of thinking about cosmology and humanity. I feel a lot of our worry and hope about AI would actually be quite familiar to an academically-minded monk in the 13th century.