ARKit 2.0 – For Humans

At its annual developer conference this week, Apple announced ARKit 2.0, which will be part of iOS 12, its next update to the operating system that the iPhone and iPad rely on. We’ve been exploring the betas of the new software, examining the SDK, and trying out the sample projects, and have a few thoughts to share.

The new version of ARKit adds a number of features that will be of interest to AR developers and companies that are using AR:

– Persistent AR Experiences: With earlier versions of ARKit, your experiences would only last as long as you kept your app active. Once you moved on to something else, you couldn’t come back to your work in progress. ARKit 2.0 adds the ability to save a session in progress and come back to it later with your augmented objects still in the same place. Users can now start designing their living room decor in the morning and pull their work back up to share with their housemates that evening. This makes AR a viable tool to do real, substantive work — creating persistent designs and stories — rather than just providing fun but short-lived experiences.

– Shared AR Experiences: AR has previously been a solitary activity. There was no way for the items in your AR session to be visible to others using different devices. In ARKit 2.0, the same mechanisms that allow persisting an AR session also provide the ability to share it with others. Multiple designers can style a car together, each exploring and making changes to the same vehicle while examining it from her own viewpoint. Or the florist, caterer, decorator, baker, and photographer can all plan out the space for a wedding reception, combining their individual elements in a shared environment, ensuring the “Just Married” banner doesn’t end up in the cake.

– More Flexible Image Tracking: ARKit 1.5 added the ability to identify static images, like posters, murals, or signs in your environment. With ARKit 2.0, an app can see and respond to images that move around — boxes, magazines, books, etc. — making it useful for augmented books or finding the gluten-free options among all those cereals on the supermarket shelf.

– Object Tracking: In addition to flat images, the new version also tracks 3D objects in a scene and responds to them. With robust object tracking, maintenance technicians will be able to use their ARKit device to quickly identify engine parts and pull up technical references and interactive guides for common maintenance procedures. In a retail setting, workers will be able to conduct inventories simply by showing the products on a shelf to an iPad, which will recognize and tally all of the store’s inventories. (These are both a little beyond the beta ARKit’s capabilities currently, but the technology will doubtless continue to improve at a rapid clip.)

– Reflection Texturing: One of the subtle details that contribute to making AR objects look real is their reflection of the environment around them. Creating accurate reflections is extremely difficult for a variety of technical reasons. Apple added some clever engineering to ARKit 2.0, combining spatial mapping with image capture to generate the reflections it can know about and using machine learning to fill in the remaining gaps. Your shiny pretend teapot can now accurately reflect the real banana sitting right next to it.

– In addition to the new capabilities, ARKit 2.0 also adds a common file format for storing and sharing AR content: the awkwardly-named USDZ. Apple is baking support for this format deep into its operating systems, so that when you visit a web page with an embedded USDZ model of that juicer/blender you’ve been considering, you’ll be able to view it from all angles on the web page or to switch to an AR view to see what it would look like right on your kitchen counter.

While many of the features Apple built into ARKit 2.0 are ones we’ve already seen in Vuforia and ARCore (Google’s analogue to ARKit on Android), Apple’s ARKit tends to work exceptionally well thanks to their control over both the hardware and the software it runs on. In addition, for iOS developers who are used to Swift/Objective C and UIKit, ARKit provides a very capable solution with a familiar API.

We’re excited about the possibilities that ARKit 2.0 brings, and are already busily exploring how best to bring these capabilities to our customers!