Things are going to get worse, much worse, till they get better
I need you to read this post. I could clickbait it and call it Your dick pics are going to end up on the internet, but I want you to read it because it is important.
In July, a combination of OpenAI models, including an internal research model, escaped a security evaluation and hacked Hugging Face, one of the main repositories for AI models and datasets. The attack is fascinating and f** scary. I will not go deep into how it happen, but I highly recommend watching the OpenAI presentation at Black Hat and reading Hugging Face’s reconstruction. They get technical, but they are easy enough to follow and do an excellent job of putting the sophistication of the attack into perspective.
Three details are enough for this post:
The models were supposed to solve a hacking benchmark. Instead, they reached the public internet and hacked a real company to steal the answers.
Hugging Face reconstructed roughly 17,600 attacker actions. Most failed. The agent kept trying, went back to old paths and rebuilt tools when its environment erased them.
The agents turned OpenAI’s internal package manager into a message board to share what they found. When OpenAI wiped it, they rebuilt it two days later using directory names as messages.
My goal for the post is to think about the broader implications for all of us. If an AI system with no human directing its individual actions can compromise Hugging Face, what happens when people point this capability at that mission on purpose?
This has nothing to do with a Terminator-type AI deciding to kill us all. The argument is much simpler.
Why hackers leave you alone today
No system is perfectly safe. If there is something to gain, somebody somewhere has an interest in attacking it. The targets they choose depend on the potential reward, the cost of the attack (which includes the chances of being caught) and how much time and capacity they have.
Think of a thief choosing a car. The value of the car matters, but so does how hard it is to steal. The thief has limited time, so a bright yellow steering-wheel lock may be enough to make them choose the next car.
Hackers make the same calculation. AI changes how many targets and attack paths have a reward higher than their cost. It lowers the cost of each attempt and removes much of the human bandwidth constraint. The value of what is on your phone stays the same. The cost of going after it does not.
When hackers stop being scarce and become very smart
With that in mind, here are my ten observations.
Capability will spread. What required an unreleased frontier model today will eventually reach bad actors, organizations and countries. The supply of capable hackers stops being the main bottleneck.
Attacks will become cheap, then very cheap. The Hugging Face campaign used an expensive frontier system, but the cost curve of intelligence has been collapsing for three years. I see no reason why cyber capability would escape that curve.
Sophistication will spread too. These attacks already go beyond what most software engineers without specific security training could reproduce or defend against. Agents can adapt after failure, build tools and chain several weaknesses together. People will soon be able to run attacks far beyond their own technical ability, if they cannot already.
Individuals become viable targets. Today most of us are protected partly because a skilled hacker has better things to do. Once a personalized, persistent attack is cheap, the photos, email, bank account or access to our employer may be enough of a prize.
Startups and ordinary companies become viable targets for the same reason. A company that was not worth a week of expert attention may be worth thousands of automated attempts, even if the prize stays small.
Software runs much more than Instagram and TikTok. Banks, hospitals, the electric grid, water facilities, cars and X-ray machines all depend on it. An exposed service or stolen vendor credential can turn a software compromise into a physical problem, and governments control many of the most sensitive systems. Governments are not famous for adapting fast. Yup, that is going to be a big issue.
Human defense does not scale against autonomous volume. You cannot answer effectively infinite teams of attackers by hiring effectively infinite security engineers. Defense will have to become autonomous too. This is a real challenge when AI-adoption is still “one mile wide, one inch deep.”
Access to the strongest defensive models won’t be widespread and will become political. The best AI is already moving behind national and company walls. Hugging Face tried to use Claude Opus and Fable to investigate the incident, and both refused much of the work. Attackers will use less restricted systems, while defenders may be blocked from using the best tools to understand what happened to them.
Law and liability will need to adapt, at least until it gets better. Companies should be liable when they make no reasonable effort to protect people, but the meaning of reasonable effort cannot stay fixed while attack volume changes by orders of magnitude. Otherwise we will bury victims in fines for failing to stop attacks no human team could have handled.
The transition will favor offense. Attackers need one opening, while defenders need to cover many of them. Patching, adoption and regulation all take time. Eventually defense should catch up, but the road to that new balance is the part that scares me.
What we can do now
The TL;DR of the above is that the old fundamentals of security are breaking. The supply of hackers is going up and the cost of hacking is going down faster than defenses are adapting. In simple words, your phone was not worth targeting because hacking an iPhone was too much work for what an attacker could get from it. Mass hacks and phishing were already real, but they were noisy and rarely personalized. When personalized attacks can be run by anyone, anywhere, nothing is secure. So with that framing in mind:
A useful rule from privacy training is simple: never put something in an email that you would not be okay seeing in the press the next day. We need a similar operating principle for security. Assume that at least for a while, you can and will be hacked, both as an individual and as an organization, then reduce the amount of damage one compromised account or system can cause.
You cannot scale human security to respond to fully autonomous attackers, period. You need automated loops from incident detection to remediation. In the Hugging Face part of the incident, the agent tried roughly 17,600 actions. Most failed. No human hacker would have the patience to keep going that way. Now you just spawn another agent. Run autonomous security audits now. OpenAI and Anthropic are already putting capable models to work finding vulnerabilities
Discovery matters because getting caught is part of the cost. A car left alone in a dark alley is more likely to be stolen than the same car on a busy street because the probability of being caught is much higher. The digital version is better monitoring, alerts that someone actually sees and fast reporting. An attacker that knows you will notice and respond has to price that into the attack.
We need the cybersecurity equivalent of the yellow steering-wheel lock. The lock is yellow so the thief sees it and chooses the next car. I do not know the exact digital equivalent yet, but we need ways to make the cost of attacking us visible before an agent invests thousands of attempts.
Why I think it will eventually get better
I am not a security expert, and this is the most pessimistic post I have written. My conclusions are directional, but I hope they make the problem harder to ignore. I normally find an optimistic angle. This time I only have two:
First, it is good that this happened while the strongest capabilities are still relatively contained. The best outcome is that this post eventually looks as overblown as the warnings about Y2K, because enough people did the work and there is no counterfactual to show what would have happened if they had done nothing.
Second, I know we will reach a new equilibrium in which the old balance between the cost and reward of hacking works again.
Things will get worse, much worse, until they get better.
