DeepView: Light Field Video

Research Paper at SIGGRAPH 2020

DeepView is a research paper presented at SIGGRAPH 2020. The presented work is a method of capturing, processing, compressing, and rendering light field video on consumer grade hardware.

A light field is a capture of all the rays of light within a volume. It allows you to move a virtual camera anywhere around the captured volume. One of the use cases for light fields is to view a space in VR supporting translational motion, usually referred to as 6 degrees of freedom, or 6 DoF. The benefits of light fields can be seen when viewing shiny or semi-transparent objects, and how the reflections accurately react to the movement of a virtual camera.

My contributions to the team were pretty diverse, ranging from shooting light fields with the camera rig, writing pipeline ingest scripts, creating prototypes of viewing experiences in Unity on both desktop and mobile devices, capturing and integrating audio, as well as building the project web page.

  • Company :

    Google

  • My Role :

    Technical Artist

  • Year :

    2020

  • Link :

    Project Page

The Sentinel

The Sentinel is a portable multi-camera light field video capture rig. Consisting of 47 action sports cameras, it's relatively portable. The Sentinel has been the main workhorse for the team's panoramic light field video efforts. I had the pleasure of going out and shooting with it.

Representation of the Volume

The light field volume is made up of about 100 semi-transparent images mapped onto spheres which encompasses the depth of a given space. This method of approximating a light field builds upon previous work of multi plane images, or MPIs.

Screenshot of the Rendered Result

This frame is a rendered result of the multi spherical image, or MSI. This method of generating light field video enables high quality view synthesis, even from a sparse set of input views.

Depth Map of the Scene

This is a stylized depth map of the shot above. The algorithm produces convincing results in areas of high depth complexity, such as the chicken wire on the fence and the tree in the background.

Technical Video