Lighting and Shadows in Augmented Reality

The tech ATM

ARKit and ARCore expose two values via Light Estimation for content creators to work with: a light intensity value and an average color tone value in units of Kelvin. The best way I’ve found to approach lighting with these available values in Unity3D is to set a single real-time directional light’s intensity to the Light Estimation intensity value. After I’ve set up the project’s RenderSettings to use a gradient as the Environment Lighting, I convert Light Estimation’s color tone from Kelvin to RGB and apply the RGB color value at different percentages for the top, equator, and bottom on the gradient. This video shows the difference with and without Light Estimation applied to the render using this technique.

Forward Progress

There are a couple things we can do right now with the current light estimation values and other outside sources to help sell realism in augmented reality. Unity’s shadows under low light tend to be strong. Apply light estimation intensity to the opacity of shadows to help blend them in darker environments. Morning and evening light casts longer shadows while noon light tends to be overhead with short shadows. Update Unity’s light rotation based on the time of day to help match shadow skews to real-world conditions.

Who’s moving in this space?

Obvious players: Apple and Google.
RNDR, Umbra 3D. These are cloud-based rendering services focused on AR.
6D.AI. They claim to have highly accurate object detection and density recognition, which could play a role in detecting shadows, and if you can detect shadows and know the object’s size and shape you might accurately estimate a light source. Maybe. Probably. I just made that up so we’ll see.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jason Marziani

Jason Marziani

Habitual shaker upper. Sr. Engineering Manager of 3D Experiences @ Matterport