Decoupling Knowledge and Reasoning in Transformers via Generalized Cross-Attention
A modular Transformer design that pulls factual knowledge out of the feed-forward layers and stores it in a single globally shared knowledge base, which each layer reads through a generalized cross-attention mechanism with its own learned projections. A key theoretical result is that a standard feed-forward network is a special case (a closure) of this generalized cross-attention, recasting the FFN as implicit key-value knowledge retrieval and motivating architectures that scale knowledge and reasoning independently.
2501.00823
A modular decoder-only Transformer architecture that explicitly separates stored knowledge from reasoning by replacing each block's feed-forward network with a generalized cross-attention layer that …