SpecGaussian with Latent Features: A High-quality Modeling of the View-dependent Appearance for 3D Gaussian Splatting
Recently, the 3D Gaussian Splatting (3D-GS) method has achieved great success in novel view synthesis, providing real-time rendering while ensuring high-quality rendering results. However, this method faces challenges in modeling specular reflections and handling anisotropic appearance components, especially in dealing with view-dependent color under complex lighting conditions. Additionally, 3D-GS uses spherical harmonic to learn the color representation, which has limited ability to represent complex scenes. To overcome these challenges, we introduce Lantent-SpecGS, an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. This enables a more effective representation of 3D feature fields, including appearance and geometry. Moreover, two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image. Experimental results demonstrate that our method obtains competitive performance in novel view synthesis and extends the ability of 3D-GS to handle intricate scenarios with specular reflections.
最近，3D Gaussian Splatting (3D-GS) 方法在新视图合成领域取得了巨大的成功，能够在保证高质量渲染结果的同时实现实时渲染。然而，该方法在建模镜面反射和处理各向异性外观组件时面临挑战，尤其是在复杂光照条件下处理与视角相关的颜色问题。此外，3D-GS 使用球谐函数来学习颜色表示，但在表示复杂场景时能力有限。为了解决这些问题，我们提出了 Latent-SpecGS 方法，该方法在每个 3D 高斯中引入了一个通用的潜在神经描述符，使其能够更有效地表示 3D 特征场，包括外观和几何信息。此外，我们设计了两个并行的卷积神经网络（CNN）分别解码散点特征图，输出漫反射颜色和镜面反射颜色。一个依赖于视角的遮罩被学习用于合并这两种颜色，从而生成最终渲染图像。实验结果表明，我们的方法在新视图合成方面取得了具有竞争力的性能，并扩展了 3D-GS 在处理具有镜面反射的复杂场景中的能力。