LLM-Driven Read/Write Policy for Simultaneous Generation
A training-free method that turns an off-the-shelf decoder-only large language model into its own read/write policy-maker for simultaneous (streaming) generation. It fixes a minimum-latency baseline policy (wait-1) and, at each step, compares the model's next-token distribution under the baseline input versus the current input using KL divergence; a threshold on that divergence decides whether enough source content has arrived to write output or whether to read more input, yielding low-latency simultaneous translation and streaming speech recognition without learning a separate policy network.
Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation Shoutao Guo 1,3
This paper proposes LSG (LLM-driven Simultaneous Generation), a training-free method that lets an off-the-shelf decoder-only large language model act as its own read/write policy-maker for simultaneo…