Integrating Large Language Models with Speech Language Models for Spoken Dialogue Generation
A pipeline for generating spontaneous spoken dialogue that combines the semantic coherence of a large language model with the naturalness of a textless speech language model. The LLM generates the dialogue text, which is converted to phonemes; a two-tower transformer duration predictor times each phoneme; and a speech language model conditioned on the timed phoneme sequence renders the speech, preserving turn-taking and non-verbal vocalizations while keeping the content coherent.
2501.00805
Proposes SLIDE, a system for generating spontaneous spoken dialogue that is both meaningful and natural-sounding. Textless speech language models produce fluid, expressive speech (turn-taking, non-ve…