PAPER

ScanNeRF: a Scalable Benchmark for Neural Radiance Fields

Luca De Luigi*, Damiano Bolognini*, Federico Domeniconi*, Daniele De Gregorio, Matteo Poggi, Luigi Di Stefano
Eyecan.ai * Equal contribution

In this paper, we propose the first-ever real benchmark thought for evaluating Neural Radiance Fields (NeRFs) and, in general, Neural Rendering (NR) frameworks. We design and implement an effective pipeline for scanning real objects in quantity and effortlessly. Our scan station is built with less than 500$ hardware budget and can collect roughly 4000 images of a scanned object in just 5 minutes. Such a platform is used to build ScanNeRF, a dataset characterized by several train/val/test splits aimed at benchmarking the performance of modern NeRF methods under different conditions. Accordingly, we evaluate three cutting-edge NeRF variants on it to highlight their strengths and weaknesses. The dataset is available on our project page, together with an online benchmark to foster the development of better and better NeRFs.

Cite Us

If you use this dataset in your research, please cite the following paper:

@inproceedings{deluigi2023scannerf,
  title={ScanNeRF: a Scalable Benchmark for Neural Radiance Fields},
  author={De Luigi, Luca and Bolognini, Damiano and Domeniconi, Federico and De Gregorio, Daniele and Poggi, Matteo and Di Stefano, Luigi},
  booktitle={Winter Conference on Applications of Computer Vision},
  note={WACV},
  year={2023}
}

THE DATASET

OBJECTS


ariplane 1	ariplane 2	brontosaurus	bulldozer 1	bulldozer 2

cheetah	dump truck 1	dump truck 2	elephant	excavator

forklift	giraffe	helicopter 1	helicopter 2	lego

lion	plant 2	plant 2	plant3	plant 4

plant 5	plant 6	plant 7	plant 8	plant 9

roadroller	shark	spinosaurus	stegosaurus	tiger

tractor	trex	triceratops	truck	zebra

TRAINING SPLITS


Train 1000	Train 500	Train 250	Train 100

Train sub-split 0	Train sub-split 1	Train sub-split 2	Train sub-split 3

Train sub-split 4	Train sub-split 5	Train sub-split 6	Train sub-split 7

LEADERBOARDS

EVENLY DISTRIBUTED SPLITS

		average	airplane1	airplane2	brontosaurus	bulldozer1	bulldozer2	cheetah	dumptruck1	dumptruck2	elephant	excavator	forklift	giraffe	helicopter1	helicopter2	lego	lion	plant1	plant2	plant3	plant4	plant5	plant6	plant7	plant8	plant9	roadroller	shark	spinosaurus	stegosaurus	tiger	tractor	trex	triceratops	truck	zebra
DVGO*	Train 1000	38.52	38.90	39.82	41.56	35.84	39.16	37.86	37.93	41.34	38.62	40.87	37.95	36.67	39.77	38.05	34.52	39.16	40.31	42.19	33.63	38.08	39.10	36.76	37.15	39.04	40.05	39.96	39.95	40.86	39.07	37.67	34.02	37.97	41.56	37.70	35.06
	Train 500	38.48	38.97	39.85	41.46	35.95	38.96	37.87	37.93	41.01	38.65	40.65	37.71	36.72	39.73	38.11	34.58	39.16	40.34	42.18	33.58	37.97	39.06	36.84	37.16	39.04	40.07	39.62	39.88	40.88	39.25	37.26	34.10	38.11	41.52	37.68	35.10
	Train 250	37.98	38.41	39.46	40.76	35.70	37.96	37.64	37.44	40.00	38.25	39.82	36.63	36.45	39.29	37.66	34.33	38.73	39.72	41.42	33.11	37.71	38.48	36.46	36.64	38.46	39.36	38.84	39.25	40.44	38.82	37.36	33.87	37.74	40.97	37.30	34.84
	Train 100	36.11	36.69	37.60	38.62	34.05	36.12	36.09	35.63	38.01	36.42	37.83	34.59	34.78	37.56	35.97	32.78	36.89	37.44	39.35	30.47	35.86	36.28	34.51	34.85	36.36	37.42	36.61	37.00	38.71	37.37	35.46	32.42	35.70	39.19	35.67	33.63
Plenoxels	Train 1000	33.42	34.59	35.21	34.74	32.05	34.21	33.35	33.90	35.45	32.11	35.23	32.99	32.38	35.52	33.68	30.42	33.50	34.41	36.61	29.33	32.94	34.30	30.87	31.87	33.47	33.79	34.66	32.88	34.96	33.89	32.87	30.55	32.99	35.89	33.67	30.32
	Train 500	30.30	33.49	33.69	30.18	29.78	34.33	32.47	32.41	34.16	25.10	35.33	33.09	31.25	33.35	32.30	26.32	26.41	28.29	34.07	24.17	29.15	28.02	25.30	26.55	28.13	27.44	34.59	25.31	32.73	29.32	30.20	28.67	29.12	32.50	32.80	30.32
	Train 250	25.68	27.44	27.21	24.67	23.68	32.45	29.54	27.14	30.20	21.04	33.74	32.13	26.61	27.55	26.96	22.15	22.20	22.72	27.38	20.49	25.51	24.01	21.12	20.62	22.06	22.03	33.46	19.98	25.81	25.22	24.65	23.34	22.46	25.91	27.53	29.71
	Train 100	21.74	22.81	23.36	20.43	19.34	26.40	23.49	22.01	25.57	18.06	26.90	25.87	21.97	22.81	21.67	19.44	19.33	19.99	23.01	18.46	22.15	20.79	19.13	18.98	19.93	19.57	27.28	17.78	21.74	22.47	20.44	19.32	18.88	22.69	22.44	26.39
Instant-NGP	Train 1000	36.99	37.14	37.86	39.95	34.99	38.12	35.68	36.61	39.96	36.49	38.65	37.82	34.42	37.71	36.46	33.92	38.21	37.21	38.86	33.81	36.43	38.11	34.25	35.57	36.68	37.52	39.18	38.33	39.31	38.60	36.41	33.51	37.82	39.31	36.36	33.49
	Train 500	36.99	36.40	38.38	39.99	34.72	37.65	35.24	36.78	39.44	36.21	39.59	38.22	34.54	36.84	36.93	33.79	38.24	37.23	38.98	34.08	36.55	36.64	35.19	35.43	36.74	37.39	39.66	38.44	39.09	37.96	36.38	33.88	37.91	40.04	36.64	33.32
	Train 250	36.48	37.57	37.61	39.93	34.90	38.30	21.82	36.60	38.82	34.65	38.48	37.68	34.65	37.57	36.69	33.88	37.47	37.42	38.38	34.21	36.97	37.18	35.15	35.50	36.61	37.44	38.94	38.15	39.32	38.36	36.39	33.31	37.49	39.74	36.66	33.12
	Train 100	36.54	37.30	37.44	39.96	34.72	38.09	35.59	36.65	39.93	36.01	39.77	37.80	34.26	36.98	36.43	33.79	34.91	37.03	27.53	33.18	36.79	37.99	35.05	35.36	36.34	37.51	39.37	38.28	39.21	38.52	35.95	33.73	38.03	39.80	36.50	33.12

DENSELY LOCALIZED SUB-SPLITS

DVGO*

Test Split
Train split	0	1	2	3	4	5	6	7
0	39.07	36.54	36.45	35.81	36.51	35.76	36.67	35.97
1	37.14	38.36	36.03	35.49	36.04	35.57	36.28	35.57
2	36.74	36.01	38.91	36.37	36.22	35.64	36.86	36.00
3	36.33	35.75	36.91	38.26	35.87	35.31	36.41	35.74
4	36.77	35.95	36.15	35.65	38.78	36.34	36.83	36.07
5	36.26	35.68	35.72	35.23	36.98	38.09	36.46	35.83
6	36.58	35.96	36.42	35.72	36.57	35.85	39.20	36.58
7	36.22	35.61	36.04	35.56	36.15	35.56	37.26	38.43

Plenoxels

Test Split
Train split	0	1	2	3	4	5	6	7
0	31.05	24.74	24.68	22.37	24.91	22.55	24.46	22.27
1	27.97	30.10	24.62	23.15	24.85	23.60	24.33	22.45
2	25.10	22.62	31.37	25.01	24.02	21.86	25.47	22.67
3	24.81	23.32	28.09	30.17	23.50	21.72	24.85	23.30
4	25.16	22.56	24.09	22.00	31.17	25.22	25.47	22.84
5	24.83	23.17	23.66	22.06	28.18	30.30	25.16	23.87
6	24.20	22.15	24.96	22.64	24.79	22.35	31.36	25.06
7	23.90	22.31	24.73	23.45	24.72	23.21	28.02	30.10

Instant-NGP

Test Split
Train split	0	1	2	3	4	5	6	7
0	36.98	36.10	36.43	36.04	36.34	35.75	36.31	35.92
1	36.23	36.99	36.19	36.24	36.14	35.93	36.07	35.95
2	36.54	36.31	37.40	36.64	36.54	36.17	36.61	36.29
3	36.17	36.18	36.53	37.26	36.19	35.94	36.23	36.21
4	36.39	36.00	36.48	36.12	37.12	36.10	36.46	36.09
5	36.07	36.14	36.20	36.16	36.45	36.94	36.21	36.21
6	36.43	36.25	36.63	36.42	36.58	36.15	37.28	36.48
7	36.17	36.11	36.39	36.36	36.31	36.15	36.50	37.20

* DVGO has been trained and tested with half resolution images due to memory constraints.

Submit Your Results

Send us your results on the private test set so we can add your method to our leaderboard!

To do so, send an e-mail to info@eyecan.ai with subject “Scannerf results submission” and the following info:

The name/s of your method/s.
A description of your method/s.
A download link with your results for every proposed method, with rendered images for every test set sample. We will compute metrics on those images.

Feel free to ask us any questions!