Long-Term Action Anticipation with Large Language Models as Video Token Predictors
ActionLLM casts long-term action anticipation as autoregressive next-token prediction by treating video frames as successive tokens for a large language model. The architecture is streamlined -- learnable future tokens, an action-tuning module, and the textual decoder collapsed to a single linear layer -- and a Cross-Modality Interaction Block fuses visual features with textual action clues, so the LLM's commonsense sequential reasoning forecasts the sequence of future actions from an observed video.
2501.00795
ActionLLM reframes long-term action anticipation -- predicting the sequence of actions a person will perform over an extended future horizon from an observed video -- as next-token prediction with a …