Interactive Media Systems, TU Vienna

Temporal-Consistent Stereo Matting for High-Quality Novel View Synthesis and Visual Effects

Research project in the area of Image and Video Analysis & Synthesis.

Keywords: image matting, special effects, 3D reconstruction, video processing, stereo analysis.

About this Project

This project develops new image processing techniques for the emerging field of 3D television. On the basis of two videos that are recorded by slightly displaced cameras, we aim to extract (1) the opacity values of individual pixels, (2) a depth reconstruction of the scene and (3) the temporal relationship between the images of a sequence. The computed information can serve as the input for an autostereoscopic display that uses these results to provide the user with temporal-smooth depth impression. Moreover, our results may offer the exciting possibility to realize free-viewpoint video where the user has control over a virtual camera and can select the preferred viewpoint. To suit the needs of these applications, our focus lies on generating results of high quality that outperform the current state of the art. Due to addressing the image matting problem, we will also be able to handle fuzzy and hairy objects that are traditionally difficult in image processing.

Funding provided by

Vienna Science and Technology Fund (WWFT)

Additional Information

The project has been funded by the Vienna Science and Technology Fund (WWTF) through project ICT08-019.

People

Principal Investigator: Margrit Gelautz

Postdoctoral Researchers: Michael Bleyer, Christoph Rhemann

Partner: Carsten Rother (Microsoft Research)

Results / Publications

2014

  • S. Ghuffar, N. Brosch, N. Pfeifer, M. Gelautz: "Motion Estimation and Segmentation in Depth and Intensity Videos"; Integrated Computer-Aided Engineering 2014. Detail information

2013

  • A. Hosni, M. Bleyer, M. Gelautz. “Secrets of Adaptive Support Weight Techniques for Local Stereo Matching”, Journal of Computer Vision and Image Understanding (CVIU) 2013. Detail information
  • A. Hosni, C. Rhemann, M. Bleyer, C. Rother, M. Gelautz: "Fast Cost-Volume Filtering for Visual Correspondence and Beyond", IEEE Transactions on Pattern Analysis and Machine Intelligence 2013. Detail information
  • M. Bleyer, C. Breiteneder: "Stereo Matching - State-of-the-Art and Research Challenges"; in: "Advanced Topics in Computer Vision" 2013. Detail information

2012

  • M. Bleyer, C. Rhemann, C. Rother: "Extracting 3D Scene-Consistent Object Proposals and Depth from Stereo Images", ECCV 2012. Detail information
  • A. Hosni, M. Gelautz, M. Bleyer: "Accuracy-Efficiency Evaluation of Adaptive Support Weight Techniques For Local Stereo Matching", DAGM and OAGM Symposium 2012 (poster presentation). Detail information
  • N. Brosch, A. Hosni, C. Rhemann, M. Gelautz: "Spatio-temporally Coherent Interactive Video Object Segmentation via Efficient Filtering", DAGM and OAGM Symposium 2012 (poster presentation). Detail information
  • N. Brosch, A. Hosni, C. Rhemann, M. Gelautz: "Interactive Video Segmentation via Efficient Filtering", International Computer Vision Summer School 2012 (poster presentation). Detail information
  • P. Kohli, H. Nickisch, C. Rother, C. Rhemann: "User-Centric Learning and Evaluation of Interactive Segmentation Systems", Int. J. Computer Vision 2012. Detail information

2011

  • L. He, M. Bleyer, M. Gelautz: "Object Removal by Depth-Guided Inpainting" , ÖAGM/AAPR 2011. PDF
  • N. Brosch, C. Rhemann, M. Gelautz: "Segmentation-Based Depth Propagation in Videos" , ÖAGM/AAPR 2011. PDF
  • C. Rhemann, A. Hosni, M. Bleyer, C. Rother, M. Gelautz: “Fast Cost-Volume Filtering for Visual Correspondence and Beyond” , CVPR 2011, Colorado Springs. Detail information
  • K. He, C. Rhemann, C. Rother, X. Tang, J. Sun: “A Global Sampling Method for Alpha Matting” , CVPR 2011, Colorado Springs. PDF
  • A. Hosni, C. Rhemann, M. Bleyer, M. Gelautz: "Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering" , PSIVT 2011. Detail information
  • M. Bleyer, C. Rhemann, C. Rother: “PatchMatch Stereo - Stereo Matching with Slanted Support Windows” , BMVC 2011. Detail information
  • M. Bleyer, C. Rother, P. Kohli, D. Scharstein, S. Sinha: “Object Stereo – Joint Stereo Matching and Object Segmentation” , CVPR 2011. PDF
  • A. Hosni, M. Bleyer, C. Rhemann, M. Gelautz, C. Rother: “Real-time Local Stereo Matching Using Guided Image Filtering” , IEEE International Workshop on Hot Topics in 3D 2011. Detail information

2010

  • H. Nickisch, C. Rother, P. Kohli, C. Rhemann: “Learning an Interactive Segmentation System” , ICVGIP 2010, Chennai . PDF; TR: PDF
  • C. Rhemann, C. Rother, P. Kohli, M. Gelautz: “A Spatially Varying PSF-based Prior for Alpha Matting” , CVPR 2010, San Francisco. PDF; Supplementary Material: ZIP (video and additional results); TR: PDF
  • M. Bleyer, C. Rother, P. Kohli: “Surface Stereo with Soft Segmentation” , CVPR, San Francisco 2010. PDF
  • A. Hosni, M. Bleyer, M. Gelautz: ”Near Real-Time Stereo With Adaptive Support Weight Approaches” , 3DPVT 2010, Paris. PDF
  • M. Bleyer, S. Chambon: ”Does Color Really Help in Dense Stereo Matching?” , 3DPVT 2010, Paris. PDF

2009

  • A. Hosni, M. Bleyer, M. Gelautz: “Image Segmentation Via Iterative Geodesic Averaging” , ICIG 2009, Xi’an, China. PDF

Downloads

1291-supp.zip 7.81 MB Zip archive Download
1291-tr-final.pdf 5.51 MB PDF document Download
1291.pdf 2.37 MB PDF document Download
cvpr11matting.pdf 2.85 MB PDF document Download
does color text 1.27 MB PDF document Download
final-iasyslearn-supp.pdf 296 KB PDF document Download
final-iasyslearn.pdf 2.15 MB PDF document Download
image segmentation via iterative geodesic averaging 2.08 MB PDF document Download
near real time stereo with adaptive support weight 620 KB PDF document Download
object stereo joint stereo matching 925 KB PDF document Download
segmentation based depth propagation 4.36 MB PDF document Download
surface stereo with soft segmentation 861 KB PDF document Download
VCIP14_supplementary_material 2.12 MB PDF document Download