Computer Graphics Laboratory

CGL » Research » Rendering, Images, Video

Rendering, Images, Video

Physically-based Rendering

Our goal is the efficient and physically accurate simulation of how light interacts with the world around it. Currently, our research concentrates on applying machine learning methods to improve rendering efficiency. We are also interested in developing algorithms and theoretical frameworks which enable the synthesis of images that previously were infeasible to render. Additionally, we investigate the corresponding inverse process of fabricating objects which exhibit a given desired appearance.

Topics

Efficient Rendering with Machine Learning

The interactions of light with the world can be described by the Rendering Equation. Being an infinite-dimensional integral to which no closed-form solution exists in the general case, the equation has to be evaluated using numerical methods, more specifically Monte Carlo integration. The estimates produced by these rendering methods can often be noisy. Machine learning methods can improve rendering efficiency in two aspects, either as post-processing or during rendering. We investigate both options, with image-plane reconstruction methods that produce clean images from noisy estimates, and with path guiding methods which learn the distribution of light to reduce the noise produced by path tracing.


Advanced Monte Carlo Methods for Image Synthesis

Different Monte Carlo rendering techniques, such as Path Tracing, Photon Mapping, Metropolis Light Transport, are suitable for different types of scenes in terms of efficiency. General as they might be, it could be that none of them is efficient for certain types of scenes, and a more specialized method is required. We explore improvements and extensions to existing Monte Carlo methods and develop advanced methods to efficiently simulate light transport in scenes whose simulation was very expensive or even infeasible before.


Appearance Fabrication

Appearance is an important property of real world materials. In many situations it is desirable to be able to simulate the appearance of an object on screen, or replicate the appearance of a real object. These tasks are usually performed by artists and require multiple tedious iterations of trial and error. Our goal is to automate the process of measuring, replicating and controlling the appearance of objects, both on screen and in the real world.


Image and Video

The goal of our research is to study and develop algorithms for image and video processing, editing, analysis and synthesis. Our focus lies on developing highly efficient algorithms which can be applied to real-world high resolution image and video data.

Topics

Image Resampling

Image Super-Resolution (SR) is a classic vision problem where the goal is to reconstruct a high-resolution image from its low-resolution counterpart. Image resampling is a more general problem where the input image is first warped by an arbitrary function, then new pixels are sampled to generate an output image. In addition to image SR, image resampling has many applications. For example, it may be used to correct perspective or lens distortion. It also finds use in video retargeting. Retargeting is the problem of converting video from one aspect ratio to another. For example converting old 4:3 footage to modern 16:9 footage. Due to its more general nature compared to image SR, image resampling poses many new and interesting research questions.


Frame Interpolation and Motion Estimation

Frame interpolation - synthesizing new frames in between a given sequence - is a video processing technique that can be used for various reasons, such as achieving certain artistic slow-motion effects, synchronizing content captured at different frame rates, or for reducing rendering costs and turnaround times by rendering fewer frames and interpolating to the originally intended frame rate. A crucial step in state-of-the-art frame interpolation methods is motion estimation of the scene that is used for finding correspondences in the input frames.


Image and Video Compression

With the ever-increasing image and video content that has been produced it becomes more and more important to reduce their storage needs and the Internet traffic that is necessary to make them available for the consumer. The goal of the image and video compression methods is to improve the quality/bit-rate compromise by leveraging recent development in machine learning and in particular generative models, exploring both lossy and near lossless compression.


Face Swapping

The swapping of the appearance of a target to a source actor while maintaining the target actor's performance is a longstanding and challenging problem in visual effects. The problem typically arises in cases in which a character needs to be portrayed at a younger age, when an actor is not available / deceased or when stunt scenes would be too dangerous for an actor to perform. Our goal is to produce photo-realistic, temporally coherent results at megapixel resolution for diverse facial expressions and lighting conditions.


Publications





© 2024 ETH Zurich, Computer Graphics Laboratory