Patch-wise Cross-modal Feature Mixup for Zero-Shot Image Captioning
A training method for zero-shot image captioning that removes the need for paired image-caption data by synthesizing an image for each caption with a text-to-image diffusion model. Its patch-wise cross-modal feature mix-up fuses each synthetic image's patch features with the CLIP text features of its detected salient concepts, suppressing the unfaithful regions diffusion models produce, while a CLIP-weighted cross-entropy loss prioritizes high-quality synthetic pairs during training.
2501.00437
Zero-shot image captioning trains a caption generator from text data alone, with no paired training images. This paper (PCM-Net) synthesizes an image for each training caption using a pretrained text…