An LLM-based multimodal agent maintains a persistent, per-object memory of a dynamic 3D scene, where each object entry fuses egocentric video with depth maps and camera poses via 2D-to-3D lifting to record category, state, relations, a 3D bounding box, and appearance features. A vision-language model detects object state changes from perceived actions through visual prompting and updates the corresponding memory entries, so an LLM can query the up-to-date scene memory with tools and drive embodied action primitives.
Embodied VideoAgent is an LLM-based multimodal agent for understanding dynamic 3D scenes from egocentric (first-person) observations. Building on VideoAgent, it constructs a persistent object memory …