Browse Teach Curate

Information-Bottleneck Task-Space Compression in Large Language Models

A framework that explains large-language-model inference as compressing input into task-specific direction-vector spaces and decompressing relevant information at critical generation steps, measured via mutual information, and applied through compression-based in-context learning and task-space-guided fine-tuning.

Data Text

Exploring Information Processing in Large Language Models: Insights from Information Bottleneck

This paper analyzes how large language models process information through the lens of Information Bottleneck Theory, the principle that a good internal representation compresses the input while keepi…