We propose a method for reconstructing a continuous light field of a target scene from a single observed image. Our method takes the best of two worlds: joint aperture-exposure coding for compressive light-field acquisition, and a neural radiance field (NeRF) for view synthesis. Joint aperture-exposure coding implemented in a camera enables effective embedding of 3-D scene information into an observed image, but in previous works, it was used only for reconstructing discretized light-field views. NeRF-based neural rendering enables high quality view synthesis of a 3-D scene from continuous viewpoints, but when only a single image is given as the input, it struggles to achieve satisfactory quality. Our method integrates these two techniques into an efficient and end-to-end trainable pipeline. Trained on a wide variety of scenes, our method can reconstruct continuous light fields accurately and efficiently without any test time optimization. To our knowledge, this is the first work to bridge two worlds: camera design for efficiently acquiring 3-D information and neural rendering.
Yuya Ishikawa (Graduate Student: -2024.3)
Keita Takahashi (Associate Professor)
Chihiro Tsutake (Assistant Professor)
Toshiaki Fujii (Professor)
Yuya Ishikawa, Keita Takahashi, Chihiro Tsutake, Toshiaki Fujii: "Reconstructing Continuous Light Field from Single Coded Image", IEEE Access, DOI: 10.1109/ACCESS.2023.3314340, Sep. 2023. [ IEEE Xplore (open access) ]
Our software (using Python + PyTorch) for the above paper is available. Please find the "ReadME.txt" file for the terms of use and usage. [Get our software ]
We propose a method for compressively acquiring a dynamic light field (a 5-D volume) through a single-shot coded image (a 2-D measurement). We designed an imaging model that synchronously applies aperture coding and pixel-wise exposure coding within a single exposure time. This coding scheme enables us to effectively embed the original information into a single observed image. The observed image is then fed to a convolutional neural network (CNN) for light-field reconstruction, which is jointly trained with the camera-side coding patterns. We also developed a hardware prototype to capture a real 3-D scene moving over time. We succeeded in acquiring a dynamic light field with 5x5 viewpoints over 4 temporal sub-frames (100 views in total) from a single observed image. Repeating capture and reconstruction processes over time, we can acquire a dynamic light field at 4x the frame rate of the camera. To our knowledge, our method is the first to achieve a finer temporal resolution than the camera itself in compressive light-field acquisition.
A follow-up work of CVPR 2022, which is presented at 3DSA 2022. In this work, we extend the reconstruction method so as to accept not a single but a few temporally-successive coded-images as input. We demonstrate that this extension leads to quality improvement of the reconstructed light fields without increasing the network/computational complexities.
Ryoya Mizuno, Keita Takahashi, Michitaka Yoshida, Chihiro Tsutake, Toshiaki Fujii, Hajime Nagahara: "Acquiring a Dynamic Light Field through a Single-Shot Coded Image", IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2022, (2022.6) [Go to CVF webpage] [arXiv preprint]
Ryoya Mizuno, Keita Takahashi, Michitaka Yoshida, Chihiro Tsutake, Toshiaki Fujii, Hajime Nagahara: "Reconstructing Dynamic Light Field from Successive Coded Images", The 13th International Conference on 3D Systems and Applications (3DSA 2022), National Taiwan University/Online (2022.11).
Ryoya Mizuno, Keita Takahashi, Michitaka Yoshida, Chihiro Tsutake, Toshiaki Fujii, Hajime Nagahara: "Compressive Acquisition of Light Field Video Using Aperture-Exposure-Coded Camera", ITE Transactions on Media Technology and Applications (MTA), Vol. 12, No. 1, pp. 22--35,(2024.1.1) [Go to publisher's site]
Ryoya Mizuno (Graduate Student: -2023.3)
Keita Takahashi (Associate Professor)
Michitaka Yoshida (Graduate Student, Osaka University: -2023.3)
Chihiro Tsutake (Assistant Professor)
Toshiaki Fujii (Professor)
Hajime Nagahara (Professor, Osaka University)
Our software (using Python + PyTorch) for our CVPR2022 paper is available. Please find the "readme.txt" file for the terms of use and usage. [Get our software ]
[ New! 2023/5/1] An extended software is released. This version includes both the test/train codes for our method presented at 3DSA2022. Please find the "readme.txt" file for the terms of use and usage. [Get our software ]