filename     : Ger10.pdf
entry        : article
conference   : Eurographics 2010, Norrkoping, Sweden, 03-07 May 2010
pages        : 585-594
year         : 2010
month        : May
title        : Articulated Billboards for Video-based Rendering
subtitle     :
author       : Marcel Germann, Alexander Hornung, Richard Keiser, Remo Ziegler, Stephan Würmlin, Markus Gross
booktitle    : Computer Graphics Forum (Proceedings of Eurographics 2010)
ISSN/ISBN    : 0167-7055
editor       : T. Akenine-Möller and M. Zwicker
publisher    : The Eurographics Association and Blackwell Publishing 2010
publ.place   : Oxford, UK
volume       : 29
issue        : 2
language     : english
keywords     : Articulated Billboards, Video-based Rendering, Body Pose Estimation, Billboards
abstract     : We present a novel representation and rendering method for free-viewpoint video of human characters based on multiple input video streams. The basic idea is to approximate the articulated 3D shape of the human body using a subdivision into textured billboards along the skeleton structure. Billboards are clustered to fans such that each skeleton bone contains one billboard per source camera. We call this representation articulated billboards. In the paper we describe a semi-automatic, data-driven algorithm to construct and render this representation, which robustly handles even challenging acquisition scenarios characterized by sparse camera positioning, inaccurate camera calibration, low video resolution, or occlusions in the scene. First, for each input view, a 2D pose estimation based on image silhouettes, motion capture data, and temporal video coherence is used to create a segmentation mask for each body part. Then, from the 2D poses and the segmentation, the actual articulated billboard model is constructed by a 3D joint optimization and compensation for camera calibration errors. The rendering method includes a novel way of blending the textural contributions of each billboard and features an adaptive seam correction to eliminate visible discontinuities between adjacent billboards textures. Our articulated billboards do not only minimize ghosting artifacts known from conventional billboard rendering, but also alleviate restrictions to the setup and sensitivities to errors of more complex 3D representations and multiview reconstruction techniques. Our results demonstrate the flexibility and the robustness of our approach with high quality free-viewpoint video generated from broadcast footage of challenging, uncontrolled environments.