3D-Consistent Video Generation via Joint Camera-Pose Estimation
A pre-trained video diffusion generator produces temporally smooth frames that are nonetheless not geometrically consistent across views. Jointly fine-tuning the generator with a 3D point-map and camera-pose estimation head, exploiting the shared transformer structure between diffusion video generators and feed-forward correspondence networks, forces the generated video to be 3D-consistent while yielding competitive camera poses. Students learn how a photometric generation loss combined with a 3D-aware reconstruction loss unifies video synthesis and structure-from-motion in one architecture.
JOG3R: Towards 3D-Consistent Video Generators Chun-Hao Huang1 Niloy Mitra1,2 Hyeonho Jeong1,3* Jae
JOG3R investigates whether video diffusion generators are 3D-aware. Using structure-from-motion as a probe, the authors find that a pre-trained generator (OpenSora) yields visually plausible, tempora…