Simons Foundation
46 minutes
Tokens as Interacting Particles Whose Density Obeys a Continuity Equation
With depth as time, the \( n \) token vectors move by \( dx_i/dt = \sum_j P_{ij}(t) V x_j \); their empirical density obeys a continuity equation whose velocity is an attention-weighted average, and tokens tend to cluster as depth grows.