DiffuView: Multi-View Diffusion Pretraining
for 3D-Aware Robotic Manipulation

Robotic manipulation from visual observations remains challenging due to the lack of 3D-consistent representations that generalize across diverse viewpoints and sensor configurations. Existing methods, primarily based on masked autoencoders or neural scene representations, struggle to capture robust view correspondences due to a lack of global 3D consistency or the requirement for scene-specific optimization. While multi-view diffusion models have recently shown tremendous success in 3D-aware generative synthesis, their powerful representations offer a promising direction for viewpoint-robust visuomotor control.

We introduce DiffuView, a novel framework that learns unified 3D-aware representations through multi-view diffusion pretraining and deploys them for imitation learning. DiffuView models the conditional generation of target views given source observations within a diffusion framework, enabling the network to implicitly recover scene geometry and enforce view consistency. The pretrained diffusion network is then used as a powerful visual backbone for an action policy, allowing robust control under varying viewpoints and visual conditions. On the MetaWorld and LIBERO benchmarks — in both simulation and the real world — DiffuView achieves superior generalization, improving success rates under viewpoint shifts by nearly 20% over existing methods.

Stage 1 · Multi-View Diffusion Pretraining

Given a source observation captured at a known pose and intrinsics, we compute 3D points and reproject them into the target camera to obtain warped RGB and depth maps. For RGB-D cameras the measured depth is used directly; for RGB-only setups a metric depth estimator supplies the geometric conditioning. We further inject per-pixel camera geometry via dense Plücker ray embeddings. These geometric cues are fused into the VAE latents through a lightweight convolutional encoder, after which the diffusion model applies 2D spatial attention within each view and 3D full attention across views to reason about cross-view geometry.

Stage 2 · Policy Learning

After pretraining, the diffusion UNet is repurposed as a 3D-aware vision encoder — serving both as a feature extractor and as a viewpoint-adaptive encoder. Following VPP, we perform only a single diffusion forward pass and extract multi-scale features from the UNet upsampling layers. A FiLM-conditioned Q-Former, modulated by the CLIP end-of-text token, produces compact task-relevant visual embeddings that condition a diffusion policy. The policy integrates observation and language embeddings via cross-attention, replaces LayerNorm with RMSNorm, adds action causal self-attention for temporal consistency, and uses a noise-timestep conditioned Mixture-of-Experts (Top-2 of 4 experts) to accelerate denoising.

The policy takes only a single view as input during both training and inference. We train on one fixed view (LIBERO agent view) and test across unseen viewpoints generated by rotating the camera around the robot base along the z-axis. DiffuView maintains stable performance under large viewpoint shifts, whereas the baseline collapses once the camera deviates from the training angle.

We proposed DiffuView, a two-stage framework that learns 3D-consistent visuomotor representations through multi-view diffusion pretraining and multi-task diffusion-based policy learning. DiffuView achieves state-of-the-art performance across multiple simulated manipulation benchmarks, and on the proposed Mv-Bench it robustly handles large viewpoint shifts — establishing a new paradigm for representation learning in robotic manipulation.

Limitation & future work. The current pretraining lacks explicit understanding of dynamic temporal information, limiting reasoning about motion continuity. We plan to extend DiffuView toward a joint flexible-view and time pretraining framework for unified spatial-temporal representation learning.