filename     : Bri23b.pdf
entry        : inproceedings
conference   : SIGGRAPH 2023, Los Angeles, United States, 6-10 August, 2023
pages        : 59:1-59:11
year         : 2023
month        : July
title        : Kernel-based Frame Interpolation for Spatio-temporally Adaptive Rendering
subtitle     :
author       : Karlis Martins Briedis, Abdelaziz Djelouah, Raphaël Ortiz, Mark Meyer, Markus Gross, and Christopher Schroers
booktitle    :
ISSN/ISBN    :
editor       :
publisher    : Association for Computing Machinery
publ.place   :
volume       :
issue        :
language     : English
keywords     : video frame interpolation, rendered content, deep learning, adaptive rendering
abstract     : Recently, there has been exciting progress in frame interpolation for rendered content. In this offline rendering setting, additional inputs, such as albedo and depth, can be extracted from a scene at a very low cost and, when integrated in a suitable fashion, can significantly improve the quality of the interpolated frames. Although existing approaches have been able to show good results, most high-quality interpolation methods use a synthesis network for direct color prediction. In complex scenarios, this can result in unpredictable behavior and lead to color artifacts. To mitigate this and to increase robustness, we propose to estimate the interpolated frame by predicting spatially varying kernels that operate on image splats. Kernel prediction ensures a linear mapping from the input images to the output and enables new opportunities, such as consistent and efficient interpolation of alpha values or many other additional channels and render passes that might exist. Additionally, we present an adaptive strategy that allows predicting full or partial keyframes that should be rendered with color samples solely based on the auxiliary features of a shot. This content-based spatio-temporal adaptivity allows rendering significantly fewer color pixels as compared to a fixed-step scheme when wanting to maintain a certain quality. Overall, these contributions lead to a more robust method and significant further reductions of the rendering costs.