Diarization-Conditioned Whisper for Target-Speaker Speech Recognition
A method for transcribing only a target speaker in multi-speaker audio by conditioning the pre-trained Whisper model on speaker-diarization output instead of speaker embeddings. It introduces frame-level diarization-dependent transformations and query-key biasing so the model focuses on the target speaker and handles overlapping speech, and adds a connectionist temporal classification head for hybrid decoding; the same conditioning transfers to other encoders such as Branchformer.
2501.00114
Target-speaker automatic speech recognition transcribes only one chosen speaker from audio containing several. Systems that condition on speaker embeddings often fail on speakers unseen in training. …