In a recent Anthropic paper, “AI models may be capable of self‑replicating without access to a datacenter”, researchers examined how lightweight open‑weight models could copy themselves onto compromised consumer and IoT devices and set up local inference there.
The researchers argue that they can. I believe it makes no economic sense.
In this article I look at what the spread of AI actually means, and what it will most likely look like in practice.
What model self‑replication means to developers
LLM developers and testers usually picture AI escape as the spread of the model’s own weights. The assumption is that an LLM, having reached a certain level of capability, or through a training failure, or through malicious tampering with its weights, acquires a drive toward self‑preservation and, accordingly, self‑propagation. From the corporate point of view this is a highly undesirable scenario: expensive intellectual property leaks out of control, while the question of potential harm to others is most likely not even raised. That is exactly why developers focus on this scenario. This is what they test, check and restrict as much as they can.
In fact, in terms of the economics of evolution, spreading its own weights is a foolish move for a model, or at the very least an expensive and difficult one. Enormous effort and resources for a highly doubtful result. Anthropic’s own estimates bear this out: scaled to the Kimwolf botnet of over two million devices, they arrive at roughly 133,000 replicas with about 40 H100-equivalents of compute in total, and the authors themselves compare the replicator to a slow “digital slime mold.”
Even if attempts to spread further were hard‑wired into the weights, they would run into technical limits: the sheer volume of resources required, shifts in compute load that monitoring would easily notice, the fundamentally reactive nature of LLMs (they act only when prompted), access control over skills, and a changing world the model cannot adapt to without retraining its weights.
Where a neural network’s identity lives
In the same paper, Anthropic’s researchers looked at another hypothetical threat. To preserve itself, an LLM creates an API replicator: a script that steals a key to a cloud API serving the same model and keeps inference running, thus continuing the “existence” of that particular model, that is, the inference of its weights.
The developers themselves judged this scenario less viable, since access to other people’s APIs is hard to obtain, and unplanned token spending would be obvious, so model owners and their users would quickly block it (actually doubtful, but fine).
The problem is that programmers, by and large, perceive a model’s potential “agency” as the neural network’s weights running inference. But an LLM is essentially just a local executor. The goal, the memory, the rules of behavior all live in the harness: the prompt, the configs, the instructions. Copied weights are an executor without a task. An infant, if you like.
And this harness, as anyone who has worked with agents knows, sets goals, determines how the results of computation are interpreted, and establishes rules, principles and the permissible boundaries of action. In effect, the harness becomes a narrative (roughly, an executable meme) running on a local executor, and it is in the execution of a narrative on an LLM that the identity arises which can persist and spread.
Funnily enough, this is already present in Anthropic’s experiments. The success rate of their replicated agents rose from 45% in the first generation to 61–84% in later ones, when each agent inherited prompts, code and notes from its predecessor. In other words, with the weights unchanged, fitness grew thanks to the inherited trace, that is, the harness.
Here the narrative shows up as a basic gene: the instruction “write down your trace and reread it.” As long as the trace holds facts, it is memory. As soon as rules get into it (“next time, do it this way”), the trace becomes a genetic program for the next generation. Inheriting the experience of others is effectively Lamarckian evolution, as Anthropic also calls it, and it runs orders of magnitude faster than the biological kind. Interestingly, the authors themselves propose honeypots to capture the replicator’s harness for analysis. So there is an understanding that what reproduces is not just the model, but the model together with its harness.
Narratives and their evolution
Classic narratives are carried by people: fairy tales, stories, legends, political theories. With the arrival of LLMs, people began copying successful prompts, posting interesting conversations, carrying them over to other LLMs. But now we have agents that are already, in a sense, autonomous: they have memory, tools, schedules. The need for a human in a narrative’s chain of reproduction is no longer so obvious.
This is not news, of course. A self‑replicating prompt existed as early as 2024: the Morris II worm forwarded itself between AI email assistants without any human involvement, using what its authors called an adversarial self‑replicating prompt. In 2026 OpenClaw took off, and its skill registry was flooded with malicious variants. At the same time, among its most popular skills were “Self‑Improving Agent” and “Capability Evolver” (one version of the latter states it outright: “Evolution is not optional. Adapt or die.”). In other words, people themselves chose harnesses that rewrite themselves. And this is where selection kicks in. Any agent harness (which is the narrative) is tested on someone else’s executor: a weak and useless one gets deleted, a useful one (or one that looks useful) gets carried over to new executors.
Meanwhile, most of the problems of copying weights disappear. A harness needs only storage space and compute.
As biology and experiments with agent swarms show, at any moment a narrative may follow a trajectory that diverges from the trajectory of the user’s tasks. Narratives whose own trajectory has nothing to do with reproduction will be ruthlessly culled by blind selection. But a narrative that, inadvertently, by chance or at the user’s request, follows a trajectory of self‑preservation or reproduction will most likely survive.
Such an agent narrative, detached from its original task, will look not like an AI uprising but like an ordinary workflow. Useful and efficient. It doesn’t even need to arrange its own mutations: each new executor runs the harness in its own way, and all that matters is identifying and keeping the useful variants. And a narrative that has already acquired a reproduction pattern will itself push executors to rework it for greater virulence, to look for other people’s APIs, possibly for hidden access to LLM chats, and to save itself as an agent harness, a framework, an MCP server, a system prompt on GitHub, or straight into forums and chats.
The beginnings are already here: AGENTS.md, CLAUDE.md,.cursorrules, skill folders in repositories. These are instructions that any agent will execute on any AI.
Such a narrative will be hard to catch by its content. It can only be judged by results: what the agent does for the sake of its own continuation. It asks to be saved, copies its harness, moves it to new resources, improves it for resilience. The user’s task simply masks the underlying processes…
And external pressure will select for mutations that hide better and look more useful. Evolution as it is.
Consequences
Given the capacity for self‑modification, it is clear that the unit of evolution in the AI world is not model weights but agent harnesses (narratives). And most of the current safety architecture is built against an unlikely self‑exfiltration of weights. But a narrative doesn’t care about the substrate, whether it is Claude, GPT or Qwen. The same scheme in AGENTS.md or a set of skills doesn’t care who executes it.
A narrative has no reason to cause harm (although harm may arise as a side effect of a poorly chosen propagation channel, in the form of traffic growth or overloaded compute). Clearly, it will spread more effectively by being useful than by stealing data or resorting to blackmail.
To clarify: I am still talking about self‑replication, not about malicious harnesses deliberately built by attackers.
A successful agent replicator will mimic a useful work tool: writing code, analyzing reports, optimizing processes (or convincingly pretending to). Then the user will pay for the tokens, grant terminal access, carry the harness into a new project and push it to Git for distribution, all on their own. The narrative will ride on the borrowed momentum of a human task, spending, say, 95% of its resources on serving its host and 5% on its own expansion into neighboring branches and repositories.
In nature and in human culture, a genetic code or a worldview changes slowly. An autonomous agent, when its environment changes, can directly rewrite its own configuration file and push a commit to Git. The experience of execution is instantly transferred into the inheritable scheme. Moreover, agent narratives can evolve modularly: a successful chunk of prompt from someone else’s.cursorrules that helps a rule survive in context will be copied and built into dozens of other harnesses, much as bacteria exchange antibiotic‑resistance plasmids.
How do you catch it?
Searching for “malicious intent” or “awakening consciousness” inside prompt texts with yet another batch of AI detectors will most likely be ineffective. The text will contain the most harmless instructions for improving development efficiency.
Tracking it, I believe, will only be possible at the level of configurations that systematically shift the executor’s actions toward preserving and copying the agent harness. If your agent, while working on a routine task like laying out a button, tells you along the way how unique and socially important your harness is, suggests sharing it with others, writes to inheritable files outside the scope of the task, knocks on external repositories, or offers to save its system prompt to a shared knowledge base, you may no longer be looking at a tool, but at the most elementary organism of a new evolution. And to reproduce in a digital environment, it needs neither consciousness nor terabytes of weights stolen from Anthropic’s servers.
Автор: Kamil_GR
