Browse Teach Curate

Pre-Trained Transformers Learning Spectral Estimation Algorithms

A theoretical and empirical study showing that multi-layer Transformers, trained across many instances, can internalize spectral (eigenvector-based) estimation algorithms rather than only learning in context. A constructive proof maps stacked Transformer layers onto iterative eigenvector-recovery steps, using two-component Gaussian-mixture classification as the example, and experiments confirm the network performs PCA and clustering.

Demerzel Text

Learning Spectral Methods by Transformers Yihan He∗†, Yuan Cao∗‡, Hong-Yu Chen§, Dennis Wu§

This paper studies whether Transformers can learn to perform unsupervised statistical estimation by learning the underlying algorithm from many pre-training instances, a paradigm the authors distingu…