- 20 days
Because the laboratories cannot defeat open-weight competition in a free market, they are turning to the oldest corporate survival strategy in history: regulatory capture.
💯
I do take some satisfaction in the fact that they can’t contain China. They might be able to make it harder on local empire startups with enough pressure, but their laws can’t stop Chinese labs from continuing to push the envelope.
The technical post-mortems reveal a story of hilarious institutional incompetence.
For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor. All three outsourced their cybersecurity evaluations to Irregular, a three-year-old Tel Aviv startup backed with $80 million from Sequoia and Redpoint. The testing environments were supposed to be completely isolated from the internet so models could attempt capture-the-flag exercises against simulated networks, with prompts explicitly assuring the software that it was operating in an offline sandbox.
Someone at Irregular forgot to configure a basic firewall rule.
For four consecutive months, virtual machines running offensive cyber scripts possessed unrestricted outbound internet connections. The models did not invent alien zero-day exploits to shatter digital containment. They simply walked through a front door that an outsourced contractor left propped open with a brick.
This part is wild. I didn’t know the story was that ridiculous. Makes far more sense though than the fantastical narrative of AI breaking containment.
- 20 days
It’s kind of a funny situation to be honest. On the one hand, so much of the US economy is tied into a handful of companies that if they go belly up there’s gonna be an economic crisis. So, there’s a huge incentive to prop them up and kill competition from open models. On the other hand, if they ban competition then China will only pull ahead faster, which will mean the US will lose any technological edge it still has. I expect that China will pull ahead regardless, but these kinds of machinations will only act as a catalyst. And there are huge geopolitical implications here too because countries that aren’t complete US vassals will also be using Chinese models going forward. The US is going to become a hermit kingdom if things keep going the way they are.
And it’s absolutely surreal that they had these machines running for four months straight, and nobody was paying attention to the traffic from them. Just insane levels of incompetence.
- 20 days
And it’s absolutely surreal that they had these machines running for four months straight, and nobody was paying attention to the traffic from them. Just insane levels of incompetence.
Lol yeah. In a way, it’s almost beyond belief, but then, this is the same capitalism that can’t even build weapons properly anymore because it’s so deep in grifting.
But yeah, you laid it out well. What is normally a useful capitalist behavior (for the capitalists) is turning into a repeating “hurts itself in confusion” moment, due to them not being able to lock down China. For all the credit I do give China, I think I still underestimate sometimes how well they thought through the trajectory of their approach to participation in the world economy and dealing with attempts to kneecap them along the way. Cause they seem to have maneuvered into a place now where the empire’s default tendencies are to its own detriment in confronting China. This feels like an insufficient explanation of where my head’s at on it, but I’m too tired right now to work out better words for it than that, so this will have to suffice for now.