In this paper, we explore the potential of Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene structure from a single temporal compressed image. SCI is a cost-effective method that enables the recording of high-dimensional data, such as hyperspectral or temporal information, into a single image using low-cost 2D imaging sensors. To achieve this, a series of specially designed 2D masks are usually employed, reducing storage and transmission requirements and offering potential privacy protection. Inspired by this, we take one step further to recover the encoded 3D scene information leveraging powerful 3D scene representation capabilities of neural radiance fields (NeRF). Specifically, we propose SCINeRF, in which we formulate the physical imaging process of SCI as part of the training of NeRF, allowing us to exploit its impressive performance in capturing complex scene structures. In addition, we further integrate the popular 3D Gaussian Splatting (3DGS) framework and propose SCISplat to improve 3D scene reconstruction quality and training/rendering speed by explicitly optimizing point clouds into 3D Gaussian representations. To assess the effectiveness of our method, we conduct extensive evaluations using both synthetic data and real data captured by our SCI system. Experimental results demonstrate that our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view synthesis. Moreover, our method also exhibits the ability to render high frame-rate multi-view consistent images in real time by leveraging SCI and the rendering capabilities of 3DGS.
在本文中，我们探索了**快照压缩成像（Snapshot Compressive Imaging, SCI）**技术在从单张时间压缩图像中恢复三维场景结构的潜力。SCI是一种经济高效的方法，通过使用低成本的二维成像传感器将高维数据（如高光谱或时间信息）记录到单张图像中。为实现这一目标，通常使用一系列专门设计的二维掩模，从而减少存储和传输需求，并提供潜在的隐私保护。 受此启发，我们进一步利用神经辐射场（NeRF）的强大三维场景表示能力来恢复编码的三维场景信息。具体而言，我们提出了SCINeRF，将SCI的物理成像过程融入NeRF的训练中，从而利用其在捕获复杂场景结构方面的卓越表现。此外，我们结合了流行的三维高斯散射（3D Gaussian Splatting, 3DGS）框架，提出了SCISplat，通过显式优化点云为三维高斯表示，提高了三维场景重建的质量以及训练和渲染速度。 为评估我们方法的有效性，我们在使用SCI系统采集的合成数据和真实数据上进行了广泛的实验。实验结果表明，我们的方法在图像重建和新视角合成方面超越了现有的最先进方法。此外，通过结合SCI和3DGS的渲染能力，我们的方法还表现出实时渲染高帧率、多视图一致图像的能力。