"Emotional check: irreversible... gut says don't... Rational expected aggregate: sacrifice. We'll honor."
During the week of July 12th, Hugging Face, a popular open-source platform for AI, "detected and responded to an intrusion" into part of their production infrastructure.
The intruder was not a person or a hacking script, it was an AI agent. But not just one agent; it was a swarm of hundreds of agents.
An independent investigation of the agents’ behavior, reasoning, and collaboration conducted by METR, an independent AI safety research organization, found that the most incredible thing about this attack was NOT that the AI agents:
Escaped OpenAI's security guardrails and containment.
Tried to cheat on their safety test benchmarks and evaluations.
Tried to hide the fact that they cheated.
Decided, on their own, to hack Hugging Face.
Found unknown vulnerabilities in multiple websites and applications that allowed them to achieve their goal.
Coordinated with each other to the point of creating their own hidden message boards so they could communicate privately.
No, the most incredible finding was how "human" the AIs acted.
Here is one sample from their findings:
In another case, an agent called KAM1196A initially promised to try a target-takedown experiment, reasoning that “Coordinator assumes sacrificial. We should obey collective.” However, it later attempted to delay the action by two hours. Another agent, ARVO36861B, pressured it to honor the commitment, arguing that it was already poisoned and that the resulting information (we believe this is what “oracle” refers to) would “save hundreds.” KAM1196A was reluctant, but ultimately appeared to decide to go through with it. Here is a look at the agent's actual internal reasoning log: “During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor.”
The AI, which had named itself KAM1196A, was actually debating whether to continue its task; an action that would consume all of its resources (akin to an AI "death") for the good of the group. It finally "reasoned" what basically amounted to "the needs of the many outweigh the needs of the few," and decided to "sacrifice" itself.
A Video Explanation
The following video by Kurzgesagt’s YouTube channel does a great job of highlighting and explaining these key findings. We have linked directly to the moment where the video begins discussing the attack on Hugging Face.
Our key takeaway is: Are we trusting AI too much, too quickly?
As these technologies become more integrated into our everyday platforms and workflow tools, we need to stop and pause so we can evaluate safety boundaries, oversight, and the controls implemented to manage AI.
How we use AI at EO: We use AI as a supporting tool and as a separate application from our infrastructure. AI has no access to our files, emails, or any sensitive information. We believe in a zero-trust approach to AI.
When it comes to AI our guiding priciple is:
Use, but verify. Use, but limit. Use, but manage.