AR applications on Android have historically always had problems with proper depth sensing and distinguishing between foreground and background in the physical world. Whenever you'd add an AR object, it would just sit on top of the whole scenery in front of your viewfinder, regardless of whether or not something should realistically block the view. To combat this, Google today introduced the Depth API to ARCore. The new interface helps distinguish between real-world foreground and background so digital objects will be properly occluded while improving their path finding capabilities and physics.
Developers can integrate the technology into their projects starting today, and you should see the change in some of Google's own products. The API uses a depth-from-motion algorithm similar to Google Camera's bokeh Portrait Mode to create a depth map. This is achieved by taking multiple images from different angles while you move the phone, which lets the system estimate the distance to every pixel you see on the screen. At the moment, the API only relies on one single camera for that.
This is how the mapping process works in the background.
Thanks to this depth information, digital objects can be hidden or partially hidden behind real-world materials. The first Google product to be equipped with the API is the Scene Viewer, which is part of Google Search. It lets you view all kinds of animals and more right in front of your camera — just search for "cat" on your ARCore-enabled device, for example.
The Depth API will help make AR objects feel more immersive.
The depth information can also be used for improved path finding (so digital characters stop running through your furniture), proper surface interactions (so you can paint on more complex objects than walls), and better physics (when you throw a digital ball, it will be obstructed by real-world objects). With more and more cameras sprouting on phones' backs, Google also teases that the API will rely on additional depth sensors and time-of-flight lenses in the future to improve and speed up the mapping process: "We’ve only begun to scratch the surface of what’s possible."
The API will be available on all devices currently supporting ARCore, which includes most of the recent flagships and some mid-range phones. Apart from Google Search, the first product to receive an update that takes advantage of occlusion is Houzz, an application that lets you outfit your home with new furniture in AR.