Apple’s AR glasses: the patents, hires and acquisitions behind the rumors

Apple is seriously interested in AR – that much is no secret. Tim Cook himself has talked about the company’s desire to build AR specs.

Apple has been working on virtual and augmented reality projects for years. It first launched ARKit in 2017, which lets developers build AR experiences for Apple’s iPhone or iPad, from brands such as IKEA and Night Sky.

But there have been countless rumours, patent applications and alleged leaks that point to Apple’s VR and AR headset plans for years now. But only recently does it seem like these products aren’t just a sci-fi pipe dream but could soon become a reality.

Let’s take a look at the the patents, hires and acquisitions behind the rumors.

Main image credit: Apple

The facts

Tim Cook has been open about his interest in AR, but has also been forthcoming about the challenges it presents.

“We don’t give a rat’s about being first, we want to be the best, and give people a great experience,” he told The Independent. “But now anything you would see on the market any time soon would not be something any of us would be satisfied with. Nor do I think the vast majority of people would be satisfied."

Leo Gebbie, a senior analyst for XR and Wearables at CCS Insight echoes this sentiment. “We believe that Apple’s most likely approach will be to wait and allow rivals to launch AR glasses first, before following with its own effort – an approach which has proven hugely successful elsewhere for the company.”

He uses the Apple Watch as an example of Apple entering a market late and yet still dominating it.

“I expect the company to tackle AR in a similar fashion,” Gebbie said.

Let’s not forget that it won’t just be exciting for Apple to bring out a product in a new category, this will also be a huge step forward for AR generally.

Eric Abbruzzese, the AR and VR Research Director at tech advisory firm ABI Research told Wareable: “While it’s not ideal to distil an entire market down to one company, Apple truly will dictate how the AR market progresses.”

The acquisitions

The Akonia headset acquired by Apple (Credit: Akonia)

Apple has been on an acquisition spree over the past few years, buying up companies that have created AR and VR products, as well as technologies that can be used in them.

This gives us some indication of Apple’s upcoming plans. Some AR software acquisitions include Metaio, an AR app that teamed up with Ferrari to give people an AR tour of a new car.

As well as Flyby Media, which created an app that lets people leave messages on real-life locations that could only be seen through the app – think writing “HAVE A NICE DAY” on a nearby bridge.

AR hardware startups including Vrvana, which created a mixed reality headset called Totem, and Akonia, a company that made lenses for AR products, were also bought by Apple.

As well as a range of other exciting startups dedicated to reading faces and expressions, including Emotient, which used machine learning to detect emotions, RealFace, which developed facial recognition technology and Faceshift, which worked with game, animation and film studios to capture facial expressions using 3D sensors and overlay them onto virtual characters.

More recent acquisitions in 2020 point to a VR focus, like NextVR, a company that brought entertainment experiences to VR headsets, as well as Spaces, which designs location-based VR experiences such as a VR Terminator event at a shopping centre.

Just because Apple acquired these companies doesn’t mean it’s incorporating their exact tech into its upcoming products, but it does mean it sees value in buying up its talent, tech or ideas for future use – all of which further point to big AR ambitions.

The hires

Andrew Kim was designer at Hololens and Tesla

Apple has, allegedly, been building an AR and VR team since 2015. According to Bloomberg, this is 1000 people strong and known as the Technology Development Group.

It’s led by Mike Rockwell, who originally worked at Dolby, and he’s hired talent from Oculus, HoloLens, Lucasfilm and more.

Some notable hires include Douglas Bowman, a top VR researcher who was a computer science professor at Virginia Tech and director of its centre for human-computer interaction. As well as Andrew Kim, a designer who worked at Microsoft HoloLens, Arthur Van Hoff, the founder of Jaunt VR, who joined Apple as a senior architect, and Brad Peebler, who co-founded computer animation and effects company Luxology.

Ex-intel Selim Ben Himane joined Apple and now works as a senior engineering manager developing 3D vision and machine learning algorithms for AR and VR systems.

And Jeff Norris joined Apple from the Mission Operations Innovation Office of Nasa's Jet Propulsion Lab where he led AR and VR teams.

The patents

Credit: Apple

Like acquisitions and hires, the patents Apple has applied for are big clues as to what it’s working on next.

With that in mind, the patents we’ve been interested in over the years are those that explicitly detail AR tech, including a patent filed in 2014 and granted in 2017, spotted by Apple Insider, which explains a mobile augmented reality system that detects your surroundings and then displays virtual images in real-time. Was this Apple’s first AR eyewear concept?

Patents for other VR and AR systems have followed, including one in 2021, which details the best design for a head-mounted display to ensure it’s comfortable and not affected by movement. This will be key if Apple wants people to wear its eyewear often – anyone with even the most high-end VR headset knows they can cause a headache after an hour of play.

There are plenty of other patents that seem to detail AR systems but rather than the head-mounted display or eyewear they’re concerned with the various bits needed to deliver better, more immersive experiences.

For example, a patent spotted by MacRumors details a lens-adjustment system for a head-mounted display. It involves different lenses that adjust to the right fit thanks to a fluid-filled chamber.

Another patent filed in 2021 explains a finger-mounted device that’s fitted with sensors and can control a mixed-reality headset, as well as detect pressure and provide haptic feedback.

There have been numerous patents concerned with eye tracking tech, which is extremely important for AR to work – the system would need to know where you’re looking to overlay virtual elements onto your surroundings so they’re believable.

One patent details a headset that can see where the user is looking so it can record video of what they experience. It’s described as being built into a pair of smart glasses, with a separate device for power and storage.

Another patent application looks at ways to detect the position and movements of a user’s eyes. It uses a combination of an eye tracking camera and infrared light to see exactly where they’re looking.

Some of the patents even hint at use cases, like one spotted by Patently Apple, which details a mixed reality headset that could double as a portable desktop, allowing users to edit documents and create things in a virtual 3D space.

Or another, detailed by Apple Insider, that explores touch detection so you can touch things in your environment that don’t contain pressure sensors. Think pressing virtual buttons or interfaces on a regular table.

The rumors

Oculus Quest (Credit: Oculus)

Based on the information above – and the thousands of rumours you can find online – we could speculate endlessly about what AR projects Apple might be working on. But what’s the most likely?

A Bloomberg report suggests the first headset will be high-end and mostly for professional use. It’s the second AR eyewear product (whether that’s a headset or specs) that’s expected to have mainstream appeal.

However, by developing a mixed reality headset first Apple can invest in all the underlying technologies, develop the marketing messages and consumer education, and build developer relations and content to make AR glasses as good as they can possibly be.

In early 2021, through reports on Bloomberg, The Information and predictions from analyst Ming-Chi Kuo on MacRumors, it looks like we can expect two products from Apple.

The first is likely to be a headset that combines VR and AR – sometimes called mixed reality. Allegedly, it’ll look a lot like a standard VR headset, similar to the Oculus Quest. It’ll have ultra high-resolution 8K displays, eye-tracking and the ability to show people wearing it video of their real environment – hence the VR and AR elements.

The second – expected later – is solely an AR device with a slimmer glasses-like design, more like Google Glass.Interestingly, Eric Abbruzzese of ABI Research tells us he doesn’t think the AR eyewear will be completely standalone, but will require an iPhone or iPad to power it.

This would mean there’d be less worry about battery within the AR specs and another smart device could do the heavy-lifting.“Positioning this product as a next generation Apple Watch is probably the best anecdote,” he explains. “Tethered to an iOS product, used for notifications, interaction, navigation, fitness, etc.

The AR version of that kind of product would add heads-up visuals for all of that, and possibly add more media and entertainment plays as well.”

The use cases

Oculus Quest (Credit: Oculus)

VR lends itself to more immersive experiences, like games, entertainment, documentaries or discovery. AR is more versatile. You could find all of those experiences, as well as plenty more, including education, health, training and productivity, like maps, virtual keyboards and visualisation tools to help with your work.

If we had to make a guess what Apple would focus on, it’d be a mix of entertainment and productivity. In an April 2021 interview with The New York Times, Tim Cook revealed details about what Apple’s AR experiences might look like.

“You and I are having a great conversation right now,” he said. “Arguably, it could even be better if we were able to augment our discussion with charts or other things to appear.”

We spoke to analyst Stephen Mears at data and research company Futuresource Consulting who told us: “I suspect that the productivity angle will be the first wave of AR from Apple, linking the glasses with their Mac ecosystem primarily, before then trying to tie it in with the phone, watch and AirPod ecosystem.”

This would fit well with some of the patents we’ve seen, which could be used to generate virtual keyboards and buttons. It also aligns with many of the more general predictions we’ve heard over the past year that suggest AR could play a huge part in the future of work.

But whatever the applications, it seems true Apple AR specs are still some way off – and it seems we'll have to wait for other companies to crack the AR Enigma code first.