A speech-discretization codec that recasts codebook construction as an information-theoretic graph problem: speech feature vectors become nodes of a graph, and the codebook is obtained by hierarchically and disentangledly minimizing two-dimensional structural entropy, which sets the codebook size automatically rather than by a hand-picked hyperparameter. A matching structural-entropy-guided quantizer adaptively assigns each incoming speech node to the most suitable cluster token, reducing the audio distortion caused by Euclidean-distance quantization. The resulting tokens feed a Structural-Entropy Speech Language Model (SESLM); SECodec reconstructs speech comparably to EnCodec and SESLM surpasses VALL-E on zero-shot text-to-speech.
With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretiza…