Camera Array Project

Go back to Top Page

Camera Array with Active Illuminations (With Adaptive Cost-Volume Filtering)

Our target is development of a view synthesis system that includes the entire process from capturing of multi-view videos to synthesize virtual view in real-time. Depth estimation of the target scene is indispensable for view synthesis from multiview videos. In this paper, we improved the depth estimation method we had developed in a previous work, where an active illumination technique was combined with an efficient layer based algorithm. More specifically, we proposed an adaptive space-time filtering for the cost volumes constructed for depth estimation. The adaptive space-time filtering adapts its shape for each pixel automatically according for depth estimation, resulting in higher quality of the depth estimation especially in dynamic scenes. Our method was tested on a system consisting of 16 video cameras and a Digital Light Processing (DLP) projector to show its effectiveness. We achieved higher quality depth estimation, resulting in higher quality virtual view synthesis with a nearly real-time frame rate.

Project Members

Toshiaki FUJII (Professor)

Keita TAKAHASHI (Associate Professor)

Sho Nishimura (former graduate student: --2015.3)

Shusuke YAMADA (former graduate student: --2016.3)

Related publications

Camera Array with Active Illuminations (Former version)

Given video inputs from multi-view cameras, our goal is to synthesize high quality free-viewpoint images in real-time. To this end, fast and accurate depth estimation from multi-view images is indispensable. Triangulation among the cameras often produces erroneous depth information, especially for textureless regions, resulting in low-quality synthesized views. To improve the quality of depth estimation while keeping fast computational speed, we combined active illuminations using a DLP projector with passive triangulation using multi-view images. The projector casts spatially incoherent patterns to the scene and makes uniform regions texture-rich so that triangulations for those regions can be greatly improved. Moreover, making the illuminations time-varying, we can further stabilize depth estimation using spatiotemporal matching, and remove the artificial patterns from the synthesized virtual views by averaging successive time frames. Our system consists of 16 video cameras synchronized with the DLP projector, and can synthesize high quality free-viewpoint image in about 10 fps.

Project Members

Toshiaki FUJII (Professor)

Keita TAKAHASHI (Associate Professor)

Tatsuro MORI (former graduate student: --2014.3)

Related publications

Pseudo Stabilization Using a Multi-View Camera

This technology is based on free-viewpoint image synthesis from multi-view images. We performed image synthesis in real-time using input from a camera array which had 5 x 5 viewpoints. In the video, the left pane shows synthesized images from the virtual viewpoint, and the right pane shows two of the input images. The first demonstration simply shows free-viewpoint image synthesis, where the camera array was fixed, and the virtual viewpoint was moving around. In the second demonstration, the camera array was moved by hand, but the virtual viewpoint was controlled to compensate the motion of the camera array, so that the video from the virtual viewpoint was virtually stabilized. This motion compensation is enabled by frame-to-frame tracking of natural feature points and self-localization (estimation of self-position and orientation) of the camera array.

Related publications