NexusSplats logo NexusSplats: Efficient 3D Gaussian Splatting in the Wild

Yuzhou Tang
Dejun Xu
Yongjie Hou
Zhenzhong Wang
Min Jiang*
Xiamen University
Paper Code

NexusSplats enables efficient and fine-grained 3D scene reconstruction under complex lighting and occlusion conditions.

Abstract

Photorealistic 3D reconstruction of unstructured real-world scenes remains challenging due to complex illumination variations and transient occlusions. Existing methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle with inefficient light decoupling and structure-agnostic occlusion handling. To address these limitations, we propose NexusSplats, an approach tailored for efficient and high-fidelity 3D scene reconstruction under complex lighting and occlusion conditions. In particular, NexusSplats leverages a hierarchical light decoupling strategy that performs centralized appearance learning, efficiently and effectively decoupling varying lighting conditions. Furthermore, a structure-aware occlusion handling mechanism is developed, establishing a nexus between 3D and 2D structures for fine-grained occlusion handling. Experimental results demonstrate that NexusSplats achieves state-of-the-art rendering quality and reduces the number of total parameters by 65.4%, leading to 2.7× faster reconstruction.

NexusSplats overview
Overview of NexusSplats. Our framework operates in three stages: First, the Hierarchical Gaussian Management organizes 3D Gaussians into dynamic nexus kernels, which generate Gaussian attributes and perform Centralized Appearance Learning and Uncertainty Propagation. Second, a raw image, a mapped image, and an uncertainty mask are rendered though tile rasterization. Third, the Boundary-Aware Refinement corrects misclassified scene boundaries. The system optimizes via a combination of color loss and uncertainty loss.

Hierarchical Light Decoupling

We propose Hierarchical Light Decoupling that leverages nexus kernels to hierarchically manage 3D Gaussians and perform centralized appearance learning for efficient and effective light condition decoupling. This design effectively reduces parameter redundancy while achieving superior texture fidelity under varying illumination.

WildGauss
GPU hours: 26.48
FPS: 7.5
 NexusSplats
GPU hours: 7.83
FPS: 11.5
WildGauss
GPU hours: 14.20
FPS: 7.5
 NexusSplats
GPU hours: 6.43
FPS: 11.5
WildGauss
GPU hours: 14.93
FPS: 7.5
 NexusSplats
GPU hours: 6.17
FPS: 11.5

Concurrent works

There are several concurrent works that also aim to extend 3DGS to handle in-the-wild data:

Acknowledgements

We sincerely appreciate the authors of 3DGS and NerfBaselines for their great work and released code. Please follow their licenses when using our code.

Citation

Please use the following citation: 
@article{tang2024nexussplats,
    title={NexusSplats: Efficient 3D Gaussian Splatting in the Wild},
    author={Tang, Yuzhou and Xu, Dejun and Hou, Yongjie and Wang, Zhenzhong and Jiang, Min},
    journal={arXiv preprint arXiv:2411.14514},
    year={2024}
}