Browse Teach Curate

Multi-Encoder Video Representations for Video Language Models

Improving Video Large Language Models by fusing features from several heterogeneous visual and video encoders (e.g. DINOv2, ViViT) rather than a single backbone family. The features are spatio-temporally aligned, projected into a unified structure, and fused with cross-attention, giving the VideoLLM a richer, complementary video representation with minimal extra parameters and parallelizable, faster training.

Demerzel Text

Unifying Specialized Visual Encoders for Video Language Models Jihoon Chung * 1 Tyler Zhu * 1 Max

Video Large Language Models connect a pretrained vision encoder to an LLM, but typically use a single encoder family, limiting the visual information they capture. This paper proposes MERV (Multi-Enc…