Browse Teach Curate

Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning

A method for making a frozen pretrained multimodal Transformer robust to missing modalities by retrieving within-modality similar instances, reconstructing the absent modality from retrieved same-modality features via a learnable FFT filter, and generating instance-specific dynamic prompts from retrieved context through cross-attention.

Data Text

Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning Jian Lang1*, Zhangtao

RAGPT (Retrieval-Augmented dynamic Prompt Tuning) is a framework for multimodal classification when one modality (text or image) is missing at training or inference. Instead of filling a missing moda…