The Relightables

Research Paper at SIGGRAPH Asia 2019

The Relightables is a research paper presented at SIGGRAPH Asia 2019. The presented work is a 3D capture system that produces highly detailed 3D models of people in motion. In addition to the high resolution 3D reconstructed mesh and albedo textures, the system recovers photometric properties of the subject and generates normals and roughness maps. The resulting volumetric output data enables these performances to be relit by virtual light sources photorealistically, maintaining the high frequency detail in the person's clothes and skin.

I had the privilege of working with this data to prototype mobile and desktop apps with game engines. The captured performances consists of a unique mesh and image sequences for the textures per frame. I scripted a playback system syncing the meshes and texture passes for both Unity and Unreal game engines to view the data in real-time.

Creating the renders for the paper's supplemental videos was another contribution I made to the project. I played a key role in setting up the scenes, lighting, shaders, render settings, and compositing.

  • Company :

    Google

  • My Role :

    Technical Artist

  • Year :

    2019

  • Link :

    Project Page

a Relightable Running in ARCore

This is a prototype I made of a Relightable running in ARCore using Environmental HDR. The combination of the high quality model and estimated lighting makes the character integrate nicely in AR.

a Relightable in a Synthetic Scene

This shot is fully synthetic: both the Relightable and the environment were rendered offline with V-Ray.

a Relightable Composited over Real Footage

This shot is a Relightable rendered offline with V-Ray and composited over a live-action plate.

a Relightable Composited over Real Footage

This is another shot with the relightable rendered offline with V-Ray and composited over a live-action plate.

The Relightables Volumetric Capture System