In simplest terms, a neural radiance field (NeRF) represents a 3D scene as a neural network which can then be queried to synthesize novel views of the scene. In other words, given a set of images of a scene captured from different angles, a NeRF learns to represent the scene and can produce images of the scene from any angle. This is how we generate the videos of the scenes above from a limited set of images!
More specifically, NeRF learns to predict the color and volume density corresponding to any point and space and a viewing direction. By then accumulating color and volume density for samples along rays in space, we can volume render novel views of the scene.
NeRF was first introduced by Mildenhall et. al. in the seminal paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. The original code for the paper can be found here.
EZNeRF is an implementation of NeRF largely following the original paper designed with the goal of having organized, documented, and easy-to-understand code.
Markdown explanations corresponding to each code file are provided, and they explain both higher-level intuition as well as per-line intention for each part of the implementation.
NeRF is not only promising in the fields of computer vision and computer graphics with important applications such as in AR/VR, but it is also a fantastic opportunity to learn about and integrate many concepts in computer graphics, computer vision, deep learning, and statistics.
The goal of EZNeRF is to thoroughly communicate all of these ideas so that a reader (who may be a student, researcher, engineer, or any curious individual) may understand how all of these concepts tie together and produce the results you see above.
EZNeRF does not attempt to be the most efficient or performant implementation of NeRF, nor does it attempt to be the most complete. In fact, EZNeRF currently only supports synthetic scenes with provided camera poses. If you are looking for implementations that are suited for real applications, there are many other implementations and variants of NeRF that produce better results much faster. Check out Nerfstudio and InstantNGP for some great examples.
EZNeRF is implemented in Python with PyTorch and a couple of other libraries. This project uses uv, a fast Python package and project manager. It has a similar interface to pip, but it also helps create virtual environments (like conda) and has reproducible dependency management with a lockfile.
If you would prefer not to install uv and use a different solution, the required dependencies are:
pytorchtorchvisiontqdmwandbtyper
After cloning the repository, you can install the required dependencies by navigating to the root directory of the repository and running:
uv sync --only-managedThis will have uv create a virtual environment at .venv and install all of the dependencies specified in pyproject.toml.
Note
You might be used to activating virtual environments (such as with conda activate or source .venv/bin/activate). You won't need to activate the virtual environment with uv, as we will use uv run to run our scripts.
We use the same data as the original implementation. From the root directory of the repository, run:
./util/download_example_data.shThere should now be a data directory at the root of the repository.
In most cases, all you will probably need to run is the combined eznerf.py script. You can see the full list of arguments, their descriptions, and default values by running:
uv run eznerf.py --helpImportant
You might want to resize the images at runtime to reduce memory usage or speed up training. You can do this by passing the --h and --w arguments to the script.
To train a model on the example data with 400x400 images, run:
uv run eznerf.py --h 400 --w 400EZNeRF uses WandB to log training metrics and visualizations. When you begin a training run, you should see a link to the run's dashboard printed to the terminal. Clicking on the link will allow you to view your training metrics and validation outputs.
The code allows for anonymous logging by default, but you can specify your own WandB API key if you'd like to store your results on your account or keep them private:
uv run wandb loginTo be made available soon.