Super-Resolution Free-Viewpoint Image Synthesis

Our system was demonstrated at SSII2013 . Super-resolved free-viewpoint image synthesis from multi-view video cameras is performed in real-time.

Watch the demo video on Youtube.

Project Members

Takeshi NAEMURA (Professor, The University of Tokyo)

Keita TAKAHASHI (Associate Professor, Nagoya University)

Koichi HAMADA (Ph.D. student, The University of Tokyo)

Ryo NAKASHIMA (Ph.D. student, The University of Tokyo)

This research is supported by the Strategic Information and Communication R&D Promotion Programs (SCOPE) of the Ministry of Internal Affairs and Communications, Japan

Selected Publications

Super-Resolved Free-Viewpoint Image Synthesis Based on View Dependent Depth Estimation

Keita Takahashi and Takeshi Naemura

IPSJ Transactions on Computer Vision and Applications (TCVA), Vol. 4, pp. 134--148 (2012.10)

Abstract: We present a method for synthesizing high-quality free-viewpoint images from a set of multi-view images. First, an accurate depth map is estimated from a given target viewpoint using modified semi-global stereo matching. Then, a high-resolution image from that viewpoint is obtained through super-resolution (SR) reconstruction. The depth estimation results from the first step are used for the second step in two ways. First, the depth values are used to associate pixels between the input images and the latent high-resolution image. Second, the pixel-wise reliabilities of the depth information are used for regularization to adaptively control the strength of the SR reconstruction. Extensive experimental results using real images show the effectiveness of our method.

[ Publisher's site (full text PDF available) ] [ supplementary video 1 (.mov) ] [ supplementary video 2 (.mov) ]


Super-Resolved Free-Viewpoint Image Synthesis Using Semi-Global Depth Estimation and Depth-Reliability-Based Regularization

Keita Takahashi and Takeshi Naemura

Proc. Pacific-Rim Symposium on Image and Video Technology 2011, LNCS 7087, pp. 22--35, Gwangju, South Korea (2011.11)

Abstract: A method for synthesizing high-quality free-viewpoint images from a set of multi-view images is presented. First, an accurate depth map is estimated from a given target viewpoint using modified semi-global stereo matching. Then, a high-resolution image from that viewpoint is obtained through super-resolution reconstruction. The depth estimation results from the first step are used for the second step. First, the depth values are used to associate pixels between the input images and the latent high-resolution image. Second, the pixel-wise reliabilities of the depth information are used for regularization to adaptively control the strength of the super-resolution reconstruction. Experimental results using real images showed the effectiveness of our method.

[ Springer's site (full text PDF available) ] [ supplementary video (.mov) ]


Super-Resolution Plane Sweeping for Free-Viewpoint Image Synthesis

Keita Takahashi, Masato Ishii, and Takeshi Naemura

Proc. IEEE International Conference on Image Processing (ICIP) 2011, pp. 2013--2016, Brussels, Belgium (2011.9).

Abstract: Free-viewpoint image synthesis (FVIS) refers to the process of generating novel viewpoint images from a set of multi-view images. Most of the conventional FVIS methods were based on image blending, so that they are subject to a fundamental limitation in resolution: the output resolution is lower than or at most equal to that of the input images. A reasonable approach to overcome this limitation is to replace image blending with reconstruction-based super-resolution. Following this idea, we propose a new FVIS method named as super-resolution plane sweeping by extending general plane sweeping methods. We also propose an adaptive weighting scheme to make super-resolution reconstruction operate only on the pixels where it improve the quality. Experimental results with real images are presented to show the effectiveness of our method.

[IEEE Xplore's site (full text PDF available) ] [ supplementary video (.mov) ] [ supplementary video (.wmv) ]

Publications List (English only)