Existing state-of-the-art (SOTA) methods for generalizable 3D scene reconstruction, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), require accurate camera poses as input. These poses are typically obtained through Structure-from-Motion (SfM) methods like COLMAP. However, this requirement is impractical in many real-world scenarios, especially when dealing with sparse views (e.g., just a couple of images) or in textureless areas where pose estimation is unreliable.
Current methods that attempt to jointly estimate poses and reconstruct scenes suffer from a compounding effect: errors in initial pose estimates degrade reconstruction quality, which in turn further degrades pose accuracy. This creates a feedback loop that limits the effectiveness of these approaches.
NoPoSplat introduces a feed-forward network that directly predicts 3D Gaussian parameters within a canonical space from sparse, unposed multi-view images. This eliminates the need for explicit pose estimation during reconstruction. By anchoring the first input view's local camera coordinates as the canonical space, NoPoSplat avoids the need to transform Gaussians from local to global coordinates, thus bypassing the errors associated with pose estimation.
NeRF (Neural Radiance Fields) and 3DGS (3D Gaussian Splatting) have significantly advanced 3D reconstruction and novel view synthesis, but these methods typically require dense posed images as input and per-scene optimization limits their practical application
Recent approaches focus on generalizable 3D reconstruction from sparse inputs**.** These methods typically use task-specific backbones that leverage geometric information to enhance scene reconstruction. Examples include:
However, these geometric operations often require camera pose input and sufficient camera pose overlap among input views. In contrast, NoPoSplat uses a Vision Transformer (ViT) without any geometric priors, making it pose-free and more effective in handling scenes with large camera baselines.
Figure 2: Comparison with pose-required sparse-view 3D Gaussian splatting pipeline. Previous methods first generate Gaussians in each local camera coordinate system and then transform them into a world coordinate system using camera poses. In contrast, NoPoSplat directly outputs 3D Gaussians of all views in a canonical space, facilitating a more coherent fusion of multi-view geometric content (see Sec. 3.3 for details). Furthermore, our backbone does not incorporate geometry priors that rely on substantial image overlap (such as epipolar geometry in pixelSplat (Charatan et al., 2024) or cost volume in MVSplat (Chen et al., 2024)). Consequently, NoPoSplat demonstrates better geometric detail preservation when the overlap between input views is limited.
Classical methods rely on accurate camera poses of input images, typically obtained through Structure from Motion (SfM) methods like COLMAP, which complicates the overall process.