The core theoretical principle posits that Large Language Models encode factual knowledge within Multi-Layer Perceptron (MLP) blocks via high-dimensional embedding vectors where distinct semantic features correspond to specific directions in the latent space. This mechanism relies on matrix multiplication operations followed by non-linear activation functions, such as ReLU or GELU, which act as gating mechanisms to trigger feature associations based on vector alignment with learned weight matrices and biases. The theory operates within the domain of neural network interpretability and mechanistic understanding, demonstrating how models store facts through superposition in high-dimensional spaces where features are represented by nearly orthogonal directions rather than isolated dimensions.
The core theoretical principle posits that Large Language Models encode factual knowledge within Multi-Layer Perceptron (MLP) blocks via high-dimensional embedding vectors where distinct semantic fea…