Extending Video Large Language Models from holistic video understanding to object-level (referring) understanding: a user designates any object by a pixel-level mask/click at a timestamp, and the model reasons about that object's attributes, actions, and relationships throughout the video. Covers the spatial-temporal object encoder design (per-object regional tokens via mask pooling plus temporal aggregation of an object's tokens across frames), object-level video instruction data, and benchmarks that separately score spatial description and temporal/relational reasoning. Exemplified by the VideoRefer Suite (VideoRefer-700K dataset, VideoRefer model, VideoRefer-Bench).
VideoRefer Suite advances object-level (spatial-temporal) understanding in Video Large Language Models across three parts: a dataset, a model, and a benchmark. (1) VideoRefer-700K is a large-scale, h…