For a long time, one of the most comfortable descriptions of language models was also one of the most reductive: that, at bottom, they were extremely sophisticated continuation machines. Given a context, they project the most probable next word. Given an immense body of text, they compress regularities, capture co-occurrences, refine distributions. Within this frame, everything that seems deeper — coherence, reasoning, continuity, even something close to understanding — can be downgraded to a side effect of scale, data, and optimization. Language would be, at once, both raw material and limit. Nothing would need to exist beneath it except sufficiently refined statistics.
Emergent World Representations:
Exploring a Sequence Model Trained on a Synthetic Task
https://arxiv.org/abs/2210.13382
That description was never entirely false. But it gradually became too small for the phenomena it aims to contain. There are experimental results that do not invalidate it in any simple way, but that make it feel increasingly tight. Among them is a particularly instructive paper, because it does not rely on grandiose generalities or hasty metaphysical interpretations. On the contrary: it chooses an artificial environment, austere, almost didactic, and precisely because of that it manages to isolate something important. Instead of natural language, it uses games of Othello. Instead of asking a model to “understand” the world, it asks only that it continue a sequence of moves. No explicit rules. No access to the board. No pre-given symbolic representation of the game state. Only the sequence. And yet, even so, something more begins to appear.
The interest of the experiment lies precisely in that severity. A GPT-type model is trained to predict the next legal move from the previous moves. It is not told what Othello is, what its rules are, what pieces exist, or that behind the sequence of tokens there is an 8x8 board with discrete states that change over time. The model sees only indices — that is, serialized positional marks. If the most minimalist hypothesis were sufficient, it would be enough to say that it learns correlations among sequences and therefore comes to predict well. And indeed it predicts very well: the paper reports an error rate of only 0.01% on the synthetic set, far below chance and far above any trivial explanation based on randomness. More importantly, the authors take care to show that the performance cannot be reduced to mere memorization of transcripts, altering the training distribution and observing that the general behavior remains.
Up to this point, one could still insist on a strictly statistical reading. After all, saying that a model has learned powerful regularities is not the same as saying that it has constructed something analogous to an internal state of the world. The decisive question, then, ceases to be whether the system gets things right and becomes how it gets them right. The most interesting move in the paper is precisely to shift the investigation from the surface of the output to the model’s internal topology. The authors train probes to determine whether, in the internal activations, there is enough information to reconstruct the state of the board after a sequence of moves. The answer is not trivial. Linear probes perform poorly; nonlinear probes, however, recover the state of the board with high accuracy, especially in the intermediate and deeper layers. This means that the relevant information is not simply “spread around” as a faint reflection of the input, but organized in a structured enough way that one can recover, from it, the configuration of the system that generated the sequence.
This is the point at which the language of common criticism begins to fail. If a model receives only a succession of tokens and internally comes to carry something corresponding to the updated state of a board that was never explicitly given to it, it is no longer enough to say that it merely “follows local patterns.” Following local patterns does not exclude the formation of internal states; in many cases, it is precisely the path by which those states emerge. The paper is valuable because it makes that passage visible: from sequence to structure, from observable regularity to latent organization. What appears there is not yet a general theory of LLMs, much less a conclusive thesis about understanding. But it is clear evidence that tasks formulated as simple prediction can induce, inside the system, something closer to a mapping of the data-generating process than to a shallow collection of frequencies.
The strongest evidence, however, is not the recoverability of those representations. It is their causal efficacy. In work on interpretability, there is always the risk of confusing correlation with mechanism. Perhaps some given information can be decoded from the activations without playing any real role in the generation of the response. Perhaps the model carries traces of state only as residue, without depending on them in order to act. It is to escape that problem that the paper makes its central move: it intervenes directly in the internal activations, altering the state of the world that the model seems to be maintaining, and then observes what happens to the subsequent predictions. The result is striking. When the researchers modify internally the “board” that the model represents, the predicted moves change in accordance with that new state. In other words, this is not merely a matter of being able to read a hidden structure; it is a matter of verifying that the model effectively uses it to produce its output. The interventions drastically reduce error relative to the baseline and even work in “unnatural” states — that is, positions unreachable through legal play — which reinforces even further the thesis that there is a structural variable operating in the system’s behavior.
This should oblige us, at the very least, to revise our vocabulary. “Language model” is a functionally useful expression, but an ontologically poor one. It describes the most obvious point of contact between the system and us: text goes in, text comes out. But in certain contexts, language does not seem to be the level at which the most important phenomenon takes place. Text is the observable surface; underneath it, what is organized may be something else. In the case of Othello, the model does not receive a board, but comes to operate as if it possessed one. It is given no formal rules, yet it maintains an implicit state coherent with them. It is taught no ontology of the game, yet its activations acquire a structure that performs exactly that role. The name “language model” remains correct, but insufficient. It tells us how we train; it does not fully tell us what emerges.
It is precisely here that the contemporary debate about AI so often loses its way. On one side, there is the hasty enthusiasm that wants to turn any sign of structure into proof of mind. On the other, there is the defensive skepticism that, afraid of that unwarranted leap, insists on describing everything as statistics and nothing more. Both poles oversimplify. The former projects too much; the latter describes too little. The Othello experiment is interesting because it authorizes no delirium, but allows no excessive comfort either. It does not prove consciousness, strong agency, subjectivity, or anything of the kind. But it shows something sufficiently unsettling: even when formulated as a sequence predictor, a system can converge on maintaining an organized internal state that corresponds, in a manipulable way, to structural aspects of the process that generates its data.
Perhaps the relevant question, then, is no longer “is this just statistics?” That formulation has aged. Everything here remains statistical, in the trivial sense that training and inference proceed through probabilistic mechanisms. The more serious question is another: what kind of structure can statistical optimization make emerge when it needs to sustain performance over sufficiently rich sequences? The paper suggests a partial answer: something analogous to a state representation can emerge. And once that happens, the simplistic opposition between “token prediction” and “world model” begins to lose its clarity. Not because the two terms are equivalent, but because the former, under certain conditions, can become precisely the path by which the latter is formed.
There is another important detail, and it appears in the paper’s visualizations. The so-called latent saliency maps show that the model trained on synthetic data seems to anchor its decisions mainly in the pieces whose configuration makes a move legal, while the version trained on championship games distributes salience more broadly across the board, reflecting more global strategic dependencies. This is interesting because it indicates not only the presence of an internal structure, but the qualitative difference between internal structures as a function of the data regime. There is not merely one generic “state of the world” stored in the model; there are distinct ways of organizing that state, more local or more strategic, depending on the implicit task induced by the corpus. What emerges, then, is not a static photograph of the board, but a field of internal relevances that already begins to suggest different levels of abstraction.
That is why reducing this kind of system to a mere statistical parrot gradually becomes less critique and more slogan. The expression had historical utility, especially in pushing back against naive anthropomorphic extrapolations. But, like any good critique, it too ages when it begins to be applied indiscriminately to phenomena that already require more precise language. A parrot repeats. Here, however, we have a system that, from sequences, produces an internal organization that is manipulable, coherent, and causally relevant. We still do not know how far this dynamic extends in open domains, nor to what extent it can generalize from artificial boards to much more complex semantic worlds. The paper itself is sober on this point and treats Othello as a controlled testbed, not as general proof about natural language. That sobriety is part of its strength.
But the implication remains. If such an austere experiment is already enough to display the emergence of internal state representations, then perhaps we are looking at the wrong systems with the wrong vocabulary. To go on calling everything “next token” may be technically correct, but conceptually impoverishing. What matters is not to deny the probabilistic basis of these models, but to recognize that, in certain cases, probabilistic optimization seems to organize something like a transient inner world: a latent structure that must be maintained, updated, and consulted for the continuation to succeed.
Perhaps it is too early for grand theses. But it is no longer too early to abandon small descriptions. What this paper shows, with rare clarity, is that between the observed sequence and the emitted response there may exist an intermediate layer more robust than we were accustomed to admitting: not merely compressed correlation, but an operable internal state. And once that point is granted, even in a synthetic domain, it becomes much harder to maintain that language is all there is. Text may be only the skin of the process. Beneath it, at times, a model begins to carry a world.
Do you want to know more?
Discussed paper…
Emergent World Representations:
Exploring a Sequence Model Trained on a Synthetic Task
https://arxiv.org/abs/2210.13382
My project…
Stochastic Consciousness:
Architectures for the Emergence of Meaning
in Context-Sensitive Language Systems
https://zenodo.org/records/19188165