ReLight My NeRF

A Dataset for Novel View Synthesis and Relighting of Real World Objects

highlight @ CVPR 2023

Marco Toschi1*, Riccardo De Matteo3*, Riccardo Spezialetti1*, Daniele De Gregorio1, Luigi Di Stefano2, Samuele Salti2
*Joint first authorship, 1eyecan.ai, 2University of Bologna, 3Work done while at eyecan.ai
Paper arXiv Video Data Leaderboard

Left: Ground-truth images from test-set. Right: The NeRF of the object given camera and light poses.

Abstract

In this paper, we focus on the problem of rendering novel views from a Neural Radiance Field (NeRF) under unobserved light conditions.

To this end, we introduce a novel dataset, dubbed ReNé (Relighting NeRF), framing real world objects under one-light-at-time (OLAT) conditions, annotated with accurate ground-truth camera and light poses.

Our acquisition pipeline leverages two robotic arms holding, respectively, a camera and an omni-directional point-wise light source. We release a total of 20 scenes depicting a variety of objects with complex geometry and challenging materials. Each scene includes 2000 images, acquired from 50 different points of views under 40 different OLAT conditions. By leveraging the dataset, we perform an ablation study on the relighting capability of variants of the vanilla NeRF architecture and identify a lightweight architecture that can render novel views of an object under novel light conditions, which we use to establish a non-trivial baseline for the dataset.

Video

Leaderboard

Apple Cheetah Cube Dinosaurs FlipFlop Fruits Garden Helicopters Kittens Lego Lunch Plant Reflective Robotoy Savannah Shark Stegosaurus Tapes Trucks Wooden Average
Yours Easy PSNR
SSIM
Hard PSNR
SSIM
Ours Easy PSNR 26.44 25.66 24.90 25.75 25.85 25.93 25.74 25.12 25.90 26.07 25.84 26.55 25.79 26.24 25.15 25.59 25.87 25.84 25.80 25.69 25.79
SSIM 0.62 0.61 0.54 0.65 0.61 0.62 0.66 0.61 0.64 0.61 0.60 0.67 0.61 0.65 0.62 0.57 0.63 0.58 0.67 0.61 0.62
Hard PSNR 26.25 24.64 23.98 24.98 25.42 25.72 25.08 24.73 24.96 25.77 24.71 25.93 25.20 25.55 24.31 25.32 25.65 25.41 25.16 25.24 25.20
SSIM 0.62 0.60 0.53 0.64 0.61 0.62 0.66 0.61 0.63 0.61 0.59 0.67 0.61 0.65 0.61 0.56 0.63 0.57 0.66 0.60 0.61

Citation

If you find this work useful or reference it in your research, kindly consider citing: 


        @InProceedings{Toschi_2023_CVPR,
          author    = {Toschi, Marco and De Matteo, Riccardo and Spezialetti, Riccardo and De Gregorio, Daniele and Di Stefano, Luigi and Salti, Samuele},
          title     = {ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects},
          booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
          month     = {June},
          year      = {2023},
          pages     = {20762-20772}
        }