Reconstructing and animating water scenery from nature would allow for compelling and immersive visual experiences. Previous work has examined this task from the perspective of 2D video textures, with the goal of creating a looping video. In our work, we tackle the problem from a 3D perspective, creating a looping 4D dynamic reconstruction, which can be interactively rendered from novel viewpoints, from a single non-looping 2D source video. We represent motion as a 3D static scene flow field that advects canonical Gaussian splats that are cyclically reborn at fixed time periods, supervised using a classic rendering loss. To model non-periodic and potentially stochastic dynamics present in real-world water scenes, we add a non-periodic, time-varying residual term to capture deviations from the static scene flow field. We show quantitatively and qualitatively that our framework enables photorealistic animation of water scenes better than prior art.
Method Overview. (Stage 1) Given a monocular input video, we first create a static reconstruction in the form of canonical Gaussian Splats. (Stage 2) We use depth computed from the static reconstruction and camera poses to lift predicted 2D optical flow to 3D, and use (3D position, 3D scene flow) pairs at every time step in the video to pre-train our static Eulerian motion field, which aggregates motion observed in the scene over all times. (Stage 3) To synthesize input images at each time step, we advect canonical Gaussian Splats using our motion field through forward Euler integration. They are rasterized, and the synthesized images are compared to the ground truth via a rendering loss that is back-propagated to optimize the canonical GS and the motion field. This synthesizes images for a single loop. We additionally learn a non-looping offset to account for the non-periodic nature of water.
For each scene (top to bottom), we show our method alongside every baseline side-by-side. Switch tabs to view different comparison settings.
Animation rendered from a fixed held-out view and held-out time steps.
Animation rendered from the input trajectory at a fixed held-out time step.
Animation rendered from a smooth unseen camera trajectory.
Reconstruction of held-out 30-frame segments from the input video used for calculating evaluation metrics in the paper.
Effect of the looping cycle length L on the reconstructed water motion. Increasing L lets the model represent a richer range of dynamic states of the water, so at L=25 and L=35 the reconstruction exhibits more varied motion. However, longer cycles require more Euler-integration steps and are harder to optimize; as a consequence L=35 is overall blurrier. We use L=15 as it best balances motion richness against reconstruction sharpness.
Removing one component at a time, all rendered from the same viewpoint. w/o residual: without the residual deformation field for position, opacity and color, water appears less stochastic and blurrier due to underfitting to the input video. w/o Eulerian field: a per-frame position, opacity and appearance offset alone cannot represent natural water motion without the Eulerian motion field. w/o scene-flow initialization: the Eulerian motion field can still be optimized from rendering losses alone, but the motion looks unnatural. w/o random start times (w/o Alg. 1 in the main paper): all Gaussians are reborn at the same instant, causing them to pulse together at the end of each cycle rather than smoothly transition between cycles. Our full model combines all components for the most natural, temporally consistent water motion.