Neural Network Self-Description: The Prerequisite for Complex Reasoning

In previous articles (1 ^[1], 2 ^[2]), I analyzed Anthropic’s J-space: why their interpretation falls short; why J-space analogues appear in micromodels under objective pressure rather than as an emergent property of scale; and why the vector they discovered is strictly equivalent to Vygotsky’s concept of the sign.

Two questions remained outside the scope: is a metasign possible—a sign integrating several basic ones—and what is the true nature of the neural network’s self-description (the report)?

This article addresses both.

TL;DR: Self-description turned out to be not an overlay on the sign, but the prerequisite for its formation. The network solves the task equally well with or without a report, but the sign axis crystallizes only where it is demanded to be named. I was unable to force the formation of a metasign.

Introduction

I am leaving the detailed breakdown of the experiments aside: the code and logs are attached, so everything can be verified or expanded upon independently. Here, I present only what I observed and the implications that follow.

So, what is J-space from Anthropic’s perspective? Within the model’s representations, they isolated a privileged subspace exhibiting a specific set of properties: the model reports its contents when asked what it is thinking about; swapping a vector within it changes the output; it is used for intermediate computations and plans; the same representation is correctly read by various downstream operations; and yet, it occupies only a small fraction of the activations—it is completely uninvolved in routine tasks like syntactic parsing.

The researchers themselves articulate the inherent strangeness of their finding: this space consists of words—a “small, evolving set of unspoken words.” They interpret all of this through the lens of Bernard Baars’ Global Workspace Theory (GWT): J-space acts as a buffer whose contents are accessible to the rest of the network.

In fact, my previous experiments demonstrated the following.

Baars’ theory describes the structure of the finding, but remains silent on the crux of the matter: why the workspace is made specifically of words. Baars’ stage is material-agnostic; it can broadcast anything. For Vygotsky, however, this word-based nature is not accidental; it is its very definition. The instrument of self-governance can only be a sign. A local, addressable, named pattern that is read by multiple operations and governs behavior—this is precisely a sign in his sense.

Second, J-space is not an emergent property of scale. Its analogues emerge in four-layer micromodels—provided the rule is expensive to recompute and has many consumers. The causal sign does not “emerge” at a sufficient scale; it pays off. The network externalizes the rule when storing it becomes cheaper than recomputing it. The control case closes the loophole: a rule that must be built, but remains computationally cheap, never receives a separate sign.

Third—and crucial for this article—what the model reports about its rule and how it actually computes that rule are two different objects. The reporting head almost always follows the injected change, whereas the actual computation does not follow suit. Furthermore, this occurs even where there is no causal sign at all: there is nothing to describe, yet the reporter faithfully reports anyway.

What remained unresolved were the role of self-description and the possibility of a metasign.

The Metasign

The very first experiments revealed that forcing the formation of a metasign is a non-trivial problem. Obviously, operating under the conditions established in the previous article—a high recomputation cost and a large number of consumers—requires a sufficiently complex and sizable neural network tasked with complex problems. Unfortunately, my current hardware does not allow me to run the necessary experiments within a reasonable timeframe. As a result, I have shelved this part of the research indefinitely.

Furthermore, any future attempt will have to overcome three fundamental problems I encountered during my initial trials:

First: the factorizability of the composition. If a metasign is built as a simple function of ready-made signs (say, the XOR of two bits), the output head simply assembles it on the fly, and no separate structure is required. Testing such a setup does not demonstrate a metasign; it merely demonstrates the ability of a linear layer to add two bits.

Second: trainability. Once you introduce a truly non-linear composition (like the parity of three features), the network stops learning altogether. Such a function yields no partial gradient: knowing two out of three features does not improve the result in the slightest; the signal appears only when all three are present. Additionally, when the rule is externalized to separate positions, the input space grows by orders of magnitude while the training set remains the same, leading to mere memorization instead of generalization.

Third: measurability. If the components are already named and established as axes, and the composition is linearly separable from them (e.g., a majority vote), it will be automatically read by the probe. Not because it has crystallized as a distinct entity, but because it is a simple function of three ready-made axes. In this case, the probe can no longer distinguish between “the metasign has formed” and “the metasign is recomputed from scratch every time.”

All three issues boil down to one core principle: the metasign must pay off.

With only two or three components and a single composition, there is nothing to economize on—it is simply cheaper to recompute. It seems the search for the metasign must shift to domains where operating on signs becomes mandatory: many components, many queries, and one shared operation for all of them. The most likely candidate is routing in a Mixture of Experts (MoE), where the choice of operation is reused at every step.

Self-Description

While investigating the neural network’s report, three hypotheses were proposed regarding its true nature:

1. Breaking automatism.

2. A prerequisite for sign formation.

3. Neural network self-modeling.

The assumption that a micromodel uses the report to break automatism was not corroborated by the experiments. The hypothesis posited that the harder the task, the more strongly the sign and its report would be activated. In reality, both the sign and the report are indifferent to the input difficulty, or even slightly degrade. It is worth noting that the microtask is entirely automatic; there is no qualitative System 1 / System 2 stratification within it, though for large language models (LLMs) this question remains open.

Self-description as a prerequisite for sign formation. This hypothesis was confirmed, and rather starkly. The experimental design: the exact same network is trained on the exact same task in three variations: 1) with a head that is obligated to name the applied rule; 2) without such a head entirely; and 3) with a dummy head that is present but trained to predict a random label—this controls strictly for the act of naming.

The result: with the report, the rule segregates into a separate, readable axis (hereafter, I will use the term precipitates) across all runs. Without the report—not a single time. With the dummy head—not a single time; the numbers are exactly the same as having no head at all. The variance across runs does not even overlap: the worst result with the report is strictly higher than the best result without it. Extending the training to the full budget changes nothing.

Separately, I tested this on a simple feature—the quadratic residue of a single argument. Here, the picture is softer: without the report, the rule segregates on its own in roughly two out of five runs, remaining smeared in the rest. With the report—in all five, with zero variance. This means the report is not always strictly necessary: the simpler the rule, the higher the chance it will crystallize without being demanded. But where the rule is computationally expensive and composite, it never crystallizes without the report. Not once. The report turns chance into a rule.

The unexpected result: all three variants solve the task equally well. That is, the network flawlessly computes based on a rule that it does not possess as a distinct entity.

Evidence in favor of the hypothesis:

First, the name precedes the skill. In the variant with the report, the rule is read almost perfectly by epoch 1,000, while the main task is still being solved at random chance. It names first; it learns later.

Second, without a name, the structure collapses. The rule does emerge in the non-reporting variant as well—in early epochs, it rises to highly decent readability. But then, under continued training and consistently high accuracy, it diffuses back into the computational stream. With the report, it holds strong for 40,000 epochs.

An unexpected corollary from a classical engineering standpoint: self-description does not reflect ready-made knowledge; it constitutes it. Without the report, the network houses a working dependency, but lacks a segregated unit. The computation runs, but the rule—as a distinct thing—does not exist. The demand to name does not assign a label to something already finished. It makes a distinction a distinction; it carves a distinct, stable, presentable entity out of the continuous computational stream.

Take a calculator as an example: its rule is part of the circuitry. It does not exist separately from the computation; it cannot be extracted, shown, or applied to something else. The rule is the calculation. A network without self-description derives the rule itself from the data—and in this sense, it remains a calculator. The rule works, but it does not exist separately, which is why it diffuses. With self-description, it becomes a distinct entity—readable from the outside, stable, transferable. The network ceases to be merely a self-assembling calculator, because something has emerged inside it that can be operated upon—even though it continues to calculate exactly as before. This is no longer mere calculation; it is primitive reasoning.

And an amusing corollary. The expectation created by the reporting head is discerning of form but blind to content: it demands that something become a presentable entity, but it does not verify that something for truth. In the micromodel, the rule fit the form. In a large model, any plausible string will fit the “list of sources” form—and that string will precipitate, not the truth. This is the true mechanism behind hallucinations, and it is perfectly identical to the mechanism behind the formation of the sign.

Neural Network Self-Modeling

I was unable to test this hypothesis in the current experiments. The premise was this: the report arises neither from an external demand nor from a breakdown, but when enough internal distinctions have accumulated for the system to start describing its own states rather than the input data.

Testing this on the current material is impossible due to the limitations of the micromodel. A self-model is a meta-structure built over a multitude of already established units; in a four-layer network with only two or three features, there is simply nothing to model.

One observation speaks indirectly in favor of this hypothesis: the reporting head consistently diverges from the actual computation. Upon intervention, it almost always reports the altered rule, while the computation continues exactly as before. If the report were merely a readout of the state, there would be no divergence. The report does not behave like a window into the inner workings, but as a separate representation of itself—that is, already as the embryo of a model, rather than a mirror.

Verifying this requires a setup where the network describes its own operation rather than an input feature, and where there are many such operations. This, it seems, falls into the exact same class of problems as the metasign—and hits the exact same hardware bottlenecks.

My Take on the Implications

First, let me dwell on why the report influences the formation of the sign in the first place.

The reporting head, in essence, demands nothing—it waits. It creates an unclosable gap: its error remains non-zero precisely to the extent that the rule has not yet been segregated into something readable from the outside. The gradient flows there not because the report exerts pressure, but because training always drains into unclosed gaps.

The difference from a standard consumer here is fundamental. A standard head uses what is already there, fine-tuning the existing structure to its function. The expectant head can use nothing until there is something to present. Its error simply hangs open—and this in itself is a form the network is forced to fill. This form dictates not what must be named, but only that it must be nameable.

Two implications follow:

First: the gap cannot be closed without segregating a distinct, externally readable unit within the computation. This is why the report turns out to be the prerequisite for the sign—it does not describe something ready-made; it forces it into existence.

Second: the filling is verified purely by form. Whether it is true or not is a question inaccessible to expectation; it lacks that dimension. Therefore, what precipitates into the open space is the first thing that is presentable, not what is true. Expectation gives birth to the sign (the gap cannot be left empty). And it simultaneously breeds confabulation, because it could not care less what is used to close that expectation.

In other words, the report is not a byproduct, but a prerequisite. Self-description is the mold into which the sign precipitates. And it is highly likely that broadcasting rules to many consumers requires a shared presentable format—that is, a name.

Applicability to LLMs

I am confident that the mathematics holds across all scales; the mechanism is geometric and not tied to a specific task. Therefore, there is every reason to expect it in large models as well. Structurally, what works in a micromodel will work in an LLM.

Of course, an LLM lacks a dedicated op-head (reporting head), but its analogue is the data-driven addressing deeply embedded in the pre-training corpus (“what do you think?”, “explain your reasoning”, “be honest”). This acts as a distributed, smeared op-head. Verifying this via ablation in LLMs, however, is exceptionally difficult.

Consequently, hallucinations in LLMs are driven by the exact same confabulation mechanism as in the micromodel. Hence the side effect of reliability prompts: they work by inserting a competing “present uncertainty” slot, which suppresses precipitation entirely. Fabrication drops, but so does depth.

Expectation in an LLM precipitates not a single sign at a discrete address, but an entire register of self-description. This perfectly aligns with the Anthropic researchers’ characterization of J-space as a “small, evolving set of unspoken words.”

Conclusion

Let us return to Vygotsky once more. A child’s failed attempt to grasp an object becomes a pointing gesture because the mother responds to it as a communicative act—she addresses an intention that does not yet exist, and by this very act of addressing, she creates it.

When a neural network receives a demand to report, it creates an internal expectation—and through this expectation, it precipitates the sign. Not for the sake of calculation (it calculates just fine without it), but to turn the rule into a presentable entity.

It is not the scale of the data that forces the neural network to operate with discrete units; it is the emergence of a consumer that needs the rule itself, rather than the result of its application. A rule smeared across the computation is accessible only to the specific circuit building it. For anyone else to read it, it must be segregated. In this sense, the sign is not a tool of thought, but a form of shareability: it arises wherever content must be made externally accessible.

Whether we can truly speak of metasigns forming in a neural network, whether a system of signs constitutes self-modeling, and whether a network can arbitrarily govern its signs and metasigns—for now, I have only hypotheses. Intuitively, however, I lean toward a positive answer.

I have paused my experiments in this direction for the time being. But should an idea emerge on how to execute this on limited hardware, or should a new perspective on the overall concept arise, I will return to them.

Link to the code and logs on GitHub ^[3]