Introduction
Mastering and understanding human faces generally encompasses a variety of different research challenges. We investigate novel approaches to acquire and represent human faces, and to develop algorithms that enable stunning visual effects and even medical applications.
Topics
Geometry Acquisition & Reconstruction
We design multiview face capture systems and develop algorithms that are able to recover the geometry and appearance of human faces with incredible accuracy and realism. Our acquisition technology and pioneering research in this field has been used in over 30 Hollywood movies so far and has led to an Academy Sci-Tech Oscar.
Performance Capture and Facial Animation
By leveraging our face capture technology, we further develop algorithms that can track nonlinear facial deformations during expressive facial performances. In addition to tracking the visible skin, our solutions also enable efficient and accurate tracking of other components of the face like the eyes, jaw, and teeth, for use in highly demanding production pipelines. Our solutions significantly reduce the manual burden of VFX artists and assists them in areas like facial animation, rigging, tracking, retargeting, secondary dynamics etc.
Neural Rendering
A primary drawback of traditional capture systems is their inability to model complex geometry and appearance of non-skin areas like the eyes, hair, clothing etc. In our pursuit of a photorealistic digital human, we leverage recent advances in neural networks and bridge gaps between traditional and neural pipelines in VFX with neural rendering. Our work in this area has significant benefits for data generation, the creation of digital doubles, face recognition and much more.
Deep Face Models
Furthermore, we build holistic models of facial geometry and appearance with neural networks that enables us to succinctly represent the manifold of human faces and allows for easy, semantically meaningful navigation of this subspace of shapes. Our work in this area has applications in sculpting, geometry synthesis and animation for background characters, face reconstruction and tracking, retargeting and much more.
Generative Facial Appearance Model
We work towards machine learning algorithms capable of producing high resolution generative facial appearance models. These models produce geometry and various textures representing different components such as albedo, normals and more.
The resulting methods are applicable in numerous areas including medical visualization, film and video game production, communication applications, and augmented- and virtual reality experiences, to name a few.
Digital Humans for Medical Applications
Finally, we leverage our work on human faces and digital humans for various medical applications to improve people's lives.
Examples include dental treatment simulation and outcome visualization, as well as digital modelling and treatment planning for craniofacial malformations such as cleft lip and palate.
For details please visit our page on medical research.
Publications
2024
T. N. Schnabel, Y. Lill, B. K. Benitez, P. Nalabothu, P. Metzler, A. A. Mueller, M. Gross, B. Gözcü, B. Solenthaler
Large-Scale 3D Infant Face Model
Medical Image Computing and Computer Assisted Intervention - MICCAI 2024 (Marrakesh, Morocco, October 06-10, 2024), pp. 217-227
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
L. Yang, G. Zoss, P. Chandran, M. Gross, B. Solenthaler, E. Sifakis, D. Bradley
Learning a Generalized Physical Face Model From Data
Proceedings of ACM SIGGRAPH (Denver, US, July 28-August 1, 2024), ACM Transactions on Graphics, vol. 43, no. 94, pp. 1-14
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
2023
L. Yang, G. Zoss, P. Chandran, P. Gotardo, M. Gross, B. Solenthaler, E. Sifakis, D. Bradley
An Implicit Physical Face Model Driven by Expression and Style
Proceedings of SA '23: SIGGRAPH Asia 2023 (Sydney, Australia, December 12-15, 2023), pp. 106:1-106:12
Available files:
[
PDF][
PDF suppl.] [
Video] [
BibTeX]
[
Abstract]
Y. Zhou, M. Chai, A. Pepe, M. Gross, T. Beeler
GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent Representations
Proceedings of ACM SIGGRAPH Asia (Sydney, Australia, December 12-15, 2023), ACM Transactions on Graphics, vol. 42, no. 6, pp. 267:1-267:16
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
C. Otto, P. Chandran, G. Zoss, M. Gross, P. Gotardo, D. Bradley
A Perceptual Shape Loss for Monocular 3D Face Reconstruction
Pacific Graphics (Daejeon, South Korea, October 10-13, 2023), Computer Graphics Forum, vol. 42, no. 7, 2023, pp. 1-12
Available files:
[
PDF][
PDF suppl.] [
BibTeX]
[
Abstract]
[
YouTube]
M. Kansy, A. Raël, G. Mignone, J. Naruniec, C. Schroers, M. Gross, R. M. Weber
Controllable Inversion of Black-Box Face Recognition Models via Diffusion
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (Paris, France, October 02-06, 2023), pp. 3159-3169
Available files:
[
PDF][
PDF suppl.] [
BibTeX]
[
Abstract]
[
YouTube]
M. Kansy, J. Balletshofer, J. Naruniec, C. Schroers, G. Mignone, M. Gross, R. M. Weber
Self-Supervised Effective Resolution Estimation With Adversarial Augmentations
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops (Waikoloa, USA, January 3-7, 2023), pp. 573-582
Available files:
[
PDF][
PDF suppl.] [
BibTeX]
[
Abstract]
[
YouTube]
2022
C. Otto, J. Naruniec, L. Helminger, T. Etterlin, G. Mignone, P. Chandran, G. Zoss, C. Schroers, M. Gross, P. Gotardo, D. Bradley, R. Weber
Learning Dynamic 3D Geometry and Texture for Video Face Swapping
Pacific Graphics (Kyoto, Japan, October 5-8, 2022), Computer Graphics Forum, vol. 41, no. 7, 2022, pp. 611-622
Available files:
[
PDF][
PDF suppl.] [
BibTeX]
[
Abstract]
[
YouTube]
D. Dorda, D. Peter, D. Borer, N.B. Huber, I. Sailer, M. Gross, B. Solenthaler, B. Thomaszewski
Differentiable Simulation for Outcome-Driven Orthognathic Surgery Planning
Proceedings of the 2022 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (Durham, UK, September 13-15, 2022), pp. 53-61
Available files:
[
PDF] [
BibTeX]
[
Abstract]
L. Yang, B. Kim, G. Zoss, B. Gözcü, M. Gross, B. Solenthaler
Implicit Neural Representation for Physics-driven Actuated Soft Bodies
Proceedings of ACM SIGGRAPH (Vancouver, Canada, August 8-11, 2022), ACM Transactions on Graphics, vol. 41, no. 4, pp. 122:1-122:10
(Honorable Mention)
Available files:
[
PDF][
PDF suppl.] [
Video] [
BibTeX]
[
Abstract]
P. Chandran, G. Zoss, M. Gross, P. Gotardo, D. Bradley
Shape Transformers: Topology-Independent 3D Shape Models Using Transformers
Proceedings of Eurographics (Reims, France, April 25-29, 2022), Computer Graphics Forum, vol. 41, no. 2, pp. 195-207
Available files:
[
PDF][
PDF suppl.] [
BibTeX]
[
Abstract]
[
YouTube]
2021
P. Chandran, S. Winberg, G. Zoss, J. Rivere, M. Gross, P. Gotardo, D. Bradley
Rendering with Style: Combining Traditional and Neural Approaches for High-Quality Face Rendering
Proceedings of ACM SIGGRAPH Asia (Tokyo, Japan, Dec. 14-17, 2021), ACM Transactions on Graphics, vol. 40, no. 6, pp. 223:1-223:14
Available files:
[
PDF] [
BibTeX]
[
Abstract]
[
YouTube]
2020
P. Chandran, D. Bradley, M. Gross, T. Beeler
Semantic Deep Face Models
Proceedings of the International Conference on 3D Vision (Virtual, November 25-27, 2020), pp. 345-354
Available files:
[
PDF] [
Abstract]
[
YouTube]
G. Zoss, E. Sifakis, M. Gross, T. Beeler, D. Bradley
Data-driven Extraction and Composition of Secondary Dynamics in Facial Performance Capture
Proceedings of ACM SIGGRAPH (Washington, D.C., USA, July 19th - 23rd, 2020), ACM Transactions on Graphics, vol. 39, no. 4, pp. 107:1-107:10
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
A. Gruber, M. Fratarcangeli, G. Zoss, R. Cattaneo, T. Beeler, M. Gross, D. Bradley
Interactive Sculpting of Digital Faces Using an Anatomical Modeling Paradigm
Proceedings of the Symposium on Geometry Processing (Utrecht, The Netherlands, July 6th - 8th, 2020), Computer Graphics Forum, vol. 39, no. 5, 2020, pp. 93-102
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
M. Fratarcangeli, D. Bradley, A. Gruber, G. Zoss, T. Beeler
Fast Nonlinear Least Squares Optimization of Large-Scale Semi-Sparse Problems
Proceedings of Eurographics (Norrköping, Sweden, May 25th - 29th, 2020), Computer Graphics Forum, vol. 39, no. 2, pp. 247-259
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
2019
G. Zoss, T. Beeler, M. Gross, D. Bradley
Accurate Markerless Jaw Tracking for Facial Performance Capture
Proceedings of ACM SIGGRAPH (Los Angeles,USA, July 28 - August 1st, 2019), ACM Transactions on Graphics, vol. 38, no. 4, pp. 50:1-50:8
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
P. Bérard, D. Bradley, M. Gross, T. Beeler
Practical Person-Specific Eye Rigging
Proceedings of Eurographics (Genova,Italy,, May 6-10, 2019), Computer Graphics Forum, vol. 38, no. 2, pp. 441-454
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
L. Shiming, M. Srinivasan, R. Mörzinger, M. Lancelle, T. Beeler, M. Gross, B. Solenthaler, V. Fehmer, I. Sailer
Reliability of a three-dimensional facial camera for dental and medical applications: A pilot study
Journal of Prosthetic Dentistry, Elsevier, 2019, Vol. 122(3), pp. 282-287.
Available files:
[
PDF] [
Abstract]
2018
G. Zoss, D. Bradley, P. Bérard, T. Beeler
An Empirical Rig for Jaw Animation
Proceedings of ACM SIGGRAPH (Vancouver, Canada, August 12th - 16th, 2018), ACM Transactions on Graphics, vol. 37, no. 4, pp. 59:1-59:12
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
2017
Y. Kozlov, D. Bradley, M. Bächer, B. Thomaszewski, T. Beeler, M. Gross
Enriching Facial Blendshape Rigs with Physical Simulation
Proceedings of Eurographics (Lyon,France,, April 24-28, 2017), Computer Graphics Forum, vol. 36, no. 2, pp. 75-84
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
2016
P. Bérard, D. Bradley, M. Gross, T. Beeler
Lightweight Eye Capture Using a Parametric Model
Proceedings of ACM SIGGRAPH (Annaheim, USA, July 24-28, 2016), ACM Transactions on Graphics, vol. 35, no. 4, pp. 117:1--117:12
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
H. J. Kim, A. C. Oztireli, I. K. Shin, M. Gross, S. M. Choi
Interactive Generation of Realistic Facial Wrinkles from Sketchy Drawings
Proceedings of Eurographics (Zurich, Switzerland,, May 4-8, 2015), Computer Graphics Forum, vol. 34, no. 2, pp. 179-191
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
2014
P. Bérard, D. Bradley, M. Nitti, T. Beeler, M. Gross
High-Quality Capture of Eyes
Proceedings of ACM SIGGRAPH Asia (Shenzhen, China, December 3-6, 2014), ACM Transactions on Graphics, vol. 33, no. 6, pp. 223:1--223:12
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
A. H. Bermano, D. Bradley, T. Beeler, F. Zünd, D. Nowrouzezahrai, I. Baran, O. Sorkine-Hornung, H. Pfister, R. W. Sumner, B. Bickel, M. Gross
Facial Performance Enhancement Using Dynamic Shape Space Analysis
Proceedings of ACM SIGGRAPH 2014 (Vancouver, Canada, August 10-14, 2014), pp. 13:1--13:12
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
C. Kuster, J.C. Bazin, T. Deng, C. Oztireli, T. Martin, T. Popa, M. Gross
Spatio-Temporal Geometry Fusion for Multiple Hybrid Cameras using Moving Least Squares Surfaces
Proceedings of Eurographics (Strasbourg, France, April 7-11, 2014), Computer Graphics Forum, vol. 33, no. 2, pp. 1-10
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
T. Beeler, B. Bickel, G. Noris, S. Marschner, P. Beardsley, R. Sumner, M. Gross
Coupled 3D Reconstruction of Sparse Facial Hair and Skin
Proceedings of ACM SIGGRAPH (Los Angeles, USA, August 5-9, 2012), ACM Transactions on Graphics, vol. 31, no. 4, pp. 117:1-117:10
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
B. Bickel, P. Kaufmann, M. Skouras, B. Thomaszewski, D. Bradley, T. Beeler, P. Jackson, S. Marschner, W. Matusik, M. Gross
Physical Face Cloning
Proceedings of ACM SIGGRAPH (Los Angeles, USA, August 5-9, 2012), ACM Transactions on Graphics, vol. 31, no. 4, pp. 118:1-118:10
Available files:
[
PDF] [
Video] [
Abstract]
2011
T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, M. Gross
High-quality passive facial performance capture using anchor frames
Proceedings of ACM SIGGRAPH (Vancouver, Canada, August 7-11, 2011), ACM Transactions on Graphics, vol. 30, no. 4, pp. 75:1-75:10
Available files:
[
PDF] [
Video] [
Video] [
Video] [
Video] [
BibTeX]
[
Abstract]
2010
D. Bradley, W. Heidrich, T. Popa, A. Sheffer
High Resolution Passive Facial Performance Capture
Proceedings of ACM SIGGRAPH (Los Angeles, USA, July 25-29, 2010), ACM Transactions on Graphics, vol. 29, no. 3, pp. 41:1--41:10
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]
T. Beeler, B. Bickel, P. Beardsley, R. Sumner, M. Gross
High-Quality Single-Shot Capture of Facial Geometry
Proceedings of ACM SIGGRAPH (Los Angeles, USA, July 25-29, 2010), ACM Transactions on Graphics, vol. 29, no. 3, pp. 40:1-40:9
Available files:
[
PDF] [
Video] [
BibTeX]
[
Abstract]