This introduces a research method for AI interpretability in which a model's internal numerical representations, called activations, are decoded into natural-language descriptions of its own processing. Activations are defined as intermediate numerical states produced when a model transforms input text into output text, analogous to snapshots of the model's internal "thinking" or to neural activity, and the described technique trains one instance of a model to translate another instance's activations into plain language, with translation accuracy verified by re-encoding that language back into activation-like numbers and checking for a match to the original.
This introduces a research method for AI interpretability in which a model's internal numerical representations, called activations, are decoded into natural-language descriptions of its own processi…