Evaluating whether a model understands an hour-long video is not the same problem as evaluating whether it understands a thirty-second clip, and this concept covers how to build a benchmark that tells the two apart. Learners follow the construction of a benchmark of roughly a thousand videos averaging 55 minutes with about fifteen thousand verified question-answer items, and reason about each design decision. Source selection: why filtering a large video corpus by sub-clip duration and then hand-removing redundant footage matters more than raw volume. Annotation: compressing a hundred-thousand-frame video to one frame per second, extracting intra-coded keyframes plus their predecessors to keep temporal continuity, describing each keyframe with a vision-language model conditioned on detected object positions and the frame timestamp, then collapsing a sliding window of frame descriptions into event descriptions because the whole video will never fit one context window. Item design: generating questions at four deliberately separated scopes - one frame, one event, two adjacent events, and the entire video - so that a score decomposes into which scale of reasoning failed, and attaching start and end timestamps to every item so temporal grounding can be checked rather than assumed. Quality control: the rules a human reviewer applies to model-generated items, namely deleting malformed pairs, correcting answers that contradict the footage, and fixing timestamps that do not align. Finally, evaluation practice: why questions about one video must be batched or trimmed to avoid leaking answers between items, and how to read the resulting scores - specialised open-source video models outscoring larger general commercial models, every model degrading as the required scope widens from single frame to whole video, and a model restricted to twenty input frames collapsing entirely, which isolates frame budget as a first-order variable rather than a hyperparameter.
This ICME 2025 paper introduces HLV-1K, a benchmark for evaluating multimodal large language models on hour-long videos. The authors argue that hour-scale video understanding is under-served: existin…