Demonstrates that a large language model can override the semantics its pretraining assigned to familiar tokens when given enough in-context evidence of a new relational structure. Using a graph-tracing task whose nodes are common concepts (apple, bird, ...) and whose edges follow a predefined geometry (e.g., a square grid) traced by random walks, the paper shows that as context length grows the model's internal representations abruptly reorganize from pretrained semantics into a geometry mirroring the in-context graph, a phase-transition-like emergence. It further shows that strong pre-existing semantic correlations (e.g., days of the week) resist this takeover, and explains the effect by analogy to energy minimization over the graph topology, evidence of an implicit optimization inferring context-specified semantics.
Recent work has demonstrated that semantics specified by pretraining data influence how representations of different concepts are organized in a large language model (LLM). However, given the open-en…