Non-Ergodic Dynamics and Constraint-Based Capability Emergence in Large Language Models
A theoretical account of why new capabilities appear abruptly in large language models as they scale: language models are shown to be non-ergodic systems whose path-dependent generation breaks the equivalence of time and ensemble averages, so emergence must be modeled along trajectories rather than by averages. A resource-bounded version of Kauffman's adjacent-possible (TAP) equation integrates architectural, training-data, and contextual constraints multiplicatively, predicting discrete phase transitions in the model's semantic space that experiments on three language models corroborate through attention entropy and effective-dimensionality measurements.
A non-ergodic framework for understanding emergent capabilities in Large Language Models Javier
Large language models have emergent capabilities that come unexpectedly at scale, but we need a theoretical framework to explain why and how they emerge. We prove that language models are actually no…