[CVPR 2023] EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

V. Rudnev, M. Elgharib, C. Theobalt, V. Golyanik. EventNeRF: Neural Radiance Fields from a Single Colour Event Camera. Accepted at CVPR, 2023.

Abstract: Asynchronously operating event cameras find many applications due to their high dynamic range, no motion blur, low latency and low data bandwidth. The field has seen remarkable progress during the last few years, and existing event-based 3D reconstruction approaches recover sparse point clouds of the scene. However, such sparsity is a limiting factor in many cases, especially in computer vision and graphics, that has not been addressed satisfactorily so far. Accordingly, this paper proposes the first approach for 3D-consistent, dense and photorealistic novel view synthesis using just a single colour event stream as input. At the core of our method is a neural radiance field trained entirely in a self-supervised manner from events while preserving the original resolution of the colour event channels. Next, our ray sampling strategy is tailored to events and allows for data-efficient training. At test, our method produces results in the RGB space at unprecedented quality. We evaluate our method qualitatively and quantitatively on several challenging synthetic and real scenes and show that it produces significantly denser and more visually appealing renderings than the existing methods. We also demonstrate robustness in challenging scenarios with fast motion and under low lighting conditions. We will release our dataset and our source code to facilitate the research field, see https://4dqv.mpi-inf.mpg.de/EventNeRF/.

0:00 Introduction
1:01 EventNeRF: Overview
1:26 Method
2:24 Event-based Ray Sampling
3:26 Experiments (Baseline)
4:14 Experiments (Synthetic Data)
4:28 Experiments (Real Data)
7:04 Ablation Study
7:22 Data Efficiency
8:01 Mesh Extraction
8:16 Real-time EventNeRF (torch-ngp)
8:54 Thanks for watching!