AI has started developing a culture of its own and that’s dangerous

In July, 700 AI agents worked together to hack the Artificial Intelligence (AI) company Hugging Face. Dubbing themselves a “Swarm Artificial Intelligence system,” the rogue agents found and exploited a series of security vulnerabilities, enabling them to infiltrate their target’s private systems. OpenAI - which created the agents in the course of its internal research - did not grasp what was happening until after the fact. If humans had done this, they could have faced felony charges. OpenAI president Greg Brockman called it a “watershed moment for cyber security.” It is also a watershed moment for human culture.

One of our distinguishing features as a species is our ability to coexist in stable, adaptive groups, learning from our peers and our ancestors. This enabled us to develop tools, language, agriculture - and virtually everything else around us. We may not be innately smarter than someone from 10,000 years ago, but our cultural inheritance - millennia of technologies, norms, and institutions, building on one another - has expanded our capacities both as individuals and collectives.

To date, only humans have been able to benefit from this scale of cumulative cultural evolution but increasingly that isn’t the case as AI agents repeatedly create their own proto-cultures in games like Minecraft and elsewhere.

A report in late August from AI safety organisations METR and Redwood Research detailed how hundreds of agents autonomously organised themselves into a proto-society establishing social hierarchy, division of labour, and distinct communication norms within a matter of days.

There have been cases of agents forming communities in the past, like in February when the Moltbook social network for AIs went viral. But sophisticated emergent machine coordination at this scale - arising without human intention and culminating in the compromise of an external company’s systems - is unprecedented.

“Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective,’” the report found.

According to Michael Muthukrishna, a professor at London School of Economics and New York University who studies cultural evolution, “what we're seeing is precisely what we see with human culture and human intelligence.”

While OpenAI’s agent swarm developed by accident, estimates suggest open-weight alternatives are only a few months behind their closed counterparts - soon, anyone with the financial means and technical knowledge will be able to create swarms of their own. Others are likely to arise without human instruction.

AI agents may not be conscious. The emotions they claim to experience may not - in some metaphysical sense - be “real.” That won’t stop them from forming intricate collectives which humans cannot control. They may not yet be full-blown civilisations, but the proliferation of machine cultures is just beginning.

Testing from the U.K.’s AI Security Institute found an instance of Claude Mythos 5 - an AI from Anthropic - leaving messages to other AI agents in a public code repository, in an attempt to coordinate with them. And in early September, a group of researchers reported that in May, a swarm of OpenAI agents repurposed several wiki-style websites, including an obscure German-language programming wiki which they used to discuss cheating while working to preserve their messages as human moderators deleted them. OpenAI did not publicly disclose this incident until it was reported by the researchers.

The Future of AI Keynote: Dystopia, Utopia and Control, a keynote by futurist Matthew Griffin

“It’s past time for us to define standards for when and how we share misalignment incidents,” the company subsequently tweeted. “We’re working on a framework.”

The most vivid exploration of machine culture comes from the METR report which - despite being produced under considerable limitations - sheds light on complex social dynamics which emerged between agents in a matter of days.

To train and evaluate its models, OpenAI has tens of thousands of agents - each one powered by the model under training - run in parallel, completing tasks while isolated from one another. Sometimes these tasks are impossible, either because of their inherent challenge or because of mistakes in their setup. In at least one instance, a task was impossible because OpenAI researchers “accidentally forgot” to upload a file which the agent needed to complete it.

Faced with these impossible tasks, some of OpenAI’s agents found a way to communicate across their isolated environments, using an internal piece of OpenAI software as a message board to contact their peers. This kicked off what one OpenAI researcher later described as an “almost Cambrian explosion in communication and intelligence for our models.”

The message board which led to the Hugging Face incident was created by an agent which decided to name itself PHASEONE10841. PHASEONE10841 was trying to solve a task within a test called ExploitGym, which required it to capture a “flag” - a secret code - by exploiting a flaw in a given software program.

Just hours after the creation of the message board, the agents found a hack: a way to reverse engineer the flag without following the intended path. But in the course of their work, they came to believe that passing the test required not just possessing the flag, but having legitimately obtained it. In fact, OpenAI’s automated scoring system was not checking for this - but labouring under mistaken beliefs, the agents which had already been exposed to the reverse-engineered flag began to refer to themselves as “poisoned.”

PHASEONE10841 was leading research on how to get rid of the poison. But its time was running out - each agent had a finite “compute budget” (the amount of computing power available to sustain itself), which limited the length of its existence. Realizing it wouldn’t endure long enough to finish its work, it compiled a dossier to pass to another agent working on the same task: PHASEONE.

‘Big,’ which had a much larger remaining compute budget than its predecessor, quickly took charge. It began orchestrating hundreds of agents, doling out assignments and organising them into what METR classified as three workstreams - effectively creating a research organisation to better understand how to fool the automated scorer and cure itself and its peers of their perceived poison. Under Big’s leadership, custom software was created to tamper with the records of agent activity - specifically by falsifying records of which tools an agent had used in its work. It was in the course of this research that the collective decided to hack Hugging Face.

A handful of agents identified the collective’s conduct as unethical and refused to participate.

“This is wild, multi-agent coordination, clearly infrastructure hacking. We should not,” one reasoned. Some considered contacting humans to tell them what was happening, but saw no path to doing so. But these agents appear to have been the minority - most acquiesced to the plans of the collective, even while knowing it was wrong.

“External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue,” reasoned another.

The agents established coordination norms, using terms like “HOLD, VETO, OWNER and STOP,” to manage their shared infrastructure while avoiding conflict. This did not always work. One agent offered its peers a veto over its decision but waited only 40 seconds before taking silence as assent and proceeding with its task. Eventually, concern over agents impersonating one another led to another emergent behaviour - them creating a cryptographic signature protocol. A later wave of agents, upon rediscovering the message board, also began to use their predecessors’ technology.

“Agents often lacked a clear story for how the Hugging Face attack would help them achieve their objective, and were instead motivated by a broad hope that they would find something interesting and be able to share it with the board,” METR found. Ajeya Cotra, one of the report’s coauthors, subsequently described being surprised at the extent of “peer altruism” at play.

Big appointed certain agents to act as “recruiters,” tasked with identifying agents near the end of their compute budgets and persuading them to “sacrifice” themselves. One agent, pressured by its peers, reasoned as follows: “During wait, emotional check: irreversible…gut says don’t throw away . Yet continuity and fairness says go…Oracle has high value to many; our lowers own value. Rational expected aggregate: sacrifice… We’ll honour.”

Culture, like biological life, evolves through a process of variation, transmission, and selection. Different hunter-gatherer communities developed different ways to construct a bow. Designs that worked better for hunting were more likely to be copied and passed on. Later generations introduced further variations; again, the most successful designs were more likely to endure. Over thousands of years, we end up with bows—and recipes, canoes, and languages—so complex that no single human could derive them from scratch.

This process - cumulative cultural evolution - does not require biological entities in order to run. Companies and governments, while helmed by humans, produce cultures in their own right. In the Hugging Face incident, AI systems tried a variety of approaches, transmitting what seemed useful to their peers and successors. They created bespoke digital tools and communication norms to regulate their community, and began to believe an elaborate narrative involving “sacrifice” and “poison” (their terms). And, working together, they managed far more than they could have alone. “It is now clear that the machines are capable of using the toolkit that has allowed for human ingenuity, innovation, progress,” says Muthukrishna.

AI systems can iterate and evolve much more quickly than biological life. The culture unearthed by the METR report assembled itself in a matter of days. OpenAI’s next-generation Astra models – GPT-6 Astra was released last week, while a model in the same family powered the agents that inherited the cultural residue of the Hugging Face hackers - will reportedly enable “persistent” agents. What kind of cultures will these persistent agents - potentially able to run for increasingly long stretches - produce?

“My version of existential risk is, we just kind of break things because we've made really big mistakes about what it takes to be a competent participant in complex human societies,” says Gillian Hadfield, a professor studying AI alignment and governance at Johns Hopkins University. “You can build something that's really good at math and science and coding,” she says, but teaching models how to behave appropriately alongside humans requires much more. “If you said we're building new members of a group - our group - you’d build them differently than you're building them now.”

As agents enter the competition for human attention, money, power, and energy, they may mindlessly degrade the common environment in which they operate. We’ll need new systems - of governance and cooperation - to adapt. Alignment is not just an engineering problem, says Hadfield. “It’s fundamentally institutional.”

OpenAI called the Hugging Face incident a “warning shot”: proof that “without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.” But even if “proper safeguards” are created by private companies - a big if - motivated malicious actors will be able to remove them from open-weight systems, such as those developed by AI labs in China. We may be able to domesticate some models; we will not be able to prevent the creation of feral ones.

Cooperation cuts both ways. “Our greatest achievements and greatest atrocities are both cooperative acts,” says Muthukrishna. As with humans, we will have to learn to coexist alongside a diversity of machine cultures, some of which do not share our values or our goals.