Task-Specific Prompt Ensembling for Zero-Shot Audio Classification
A training-free technique that improves zero-shot audio classification with contrastive Audio-Language Models by replacing generic label templates with task-specific hard prompts that encode relevant sound attributes and sources, then ensembling the text embeddings of many such prompts before cosine-matching them to the audio embedding. Students learn how prompt design and prompt averaging can boost multimodal alignment and out-of-distribution generalization without any additional training.
2501.00398
TSPE (Task-Specific Prompt Ensemble) is a training-free method that raises the zero-shot accuracy of contrastive Audio-Language Models such as CLAP. Instead of a generic template like 'sound of a <la…