My thoughts on Apple Releasing a VR headset in 2022
Apple VR’s (XR, MR, AR) Potential
The Early Days
I remember the day I got my first VR headset from Kickstarter (in 2013); I couldn’t help but see the potential but yearned for something more polished. It was easy to make the leap asking what VR would be and who would bring us there; we all wondered what it would be like if Apple jumped into the game.
Before we look forwards, let’s go back to 2010, Apple was on a high and had just released the iPhone 4. It was a slick piece of hardware. It used new materials never before seen in a phone, had a stunning design, and even had something defined as a “Retina Display.”
It was a simple dream of Steve Jobs to have the device and technology disappear behind great design. To make reading information on this little piece of glass as easy as reading on the printed page and the Retina Display would make that dream a reality. It doubled the pixel density, making text not just readable but much more comfortable to read. It wasn’t just about making something ‘functional’ but about making something for humans. To make it feel real.
Big swings like this allowed Apple to create a luxury device market worth trillions of dollars. The iPhone 4 and Apple defined a decade of technology and inspired copy-cats to design upwards.
In the early days of consumer VR, Palmer Luckey had the crazy idea of creating a cheap open headset that anyone could afford. So, he founded Oculus and launched a Kickstarter. Years later, and even additional investment later, I finally received my DK1. I wasn’t sure I would ever get one, but here it was. I received my DK1 in June of 2013 and fell in love with it. I threw parties to show it off, and so many others were amazed too.
However, my Oculus DK1 wasn’t the device to define a decade. Oculus and Facebook kept aspiring to push VR hard, but it was clear that the whole industry was much further behind. Manufacturing these devices was much more challenging than making mobile phones, even though they shared a lot of tech, and that was precisely the problem. VR just needed a lot more sophisticated hardware than mobile phones, even fancy ones.
VR would have to first grow slowly enough for the hardware to catch up to the vision and its eventual promise.
When VR launched, it reminded me of what the Atari was for console gaming or what an Apple IIc felt like for PC gaming. Sometimes I even compared it to a Sega CD/ Sega 32x, a niche product that dared an industry to dream of rich multimedia, high-quality audio, and 3d graphics. Those devices felt visionary but lacking. They missed the polish of arcades, and content was impossible to find, but my first experience playing Sewer Shark was unforgettable.
It was on my neighbor’s floor, all the kids in the neighborhood were huddled around that glowing CRT, all 5 of us, and the game just felt so much more real than anything before. None of us wanted to miss our chance at a turn, sitting shoulder-to-shoulder, trying to get a chance to go next. However, for every bit of ‘wow’ and ‘bated breath’ that followed, we would also notice bugs and quirks that were just enough that took us out of the magic. It was like a window to what the future would be, but at the same time, a clear reminder that we would have to wait for better hardware and games. It wasn’t until the PlayStation 2 that the dream of 3d games was fully realized.
VR is demanding; the hardware needed a push.
Today, we have very polished consoles. My PS4 is so good that it’s hard to find reasons to rush out to get a PS5, and my Apple iPhone 12 Pro was good enough that I didn’t upgrade to a 13 model this year. My ‘previous’ laptop had stuck around for five years, and my desktop still feels state of the art and modern at 3+ years old. If it weren’t for VR or AR, you’d think we hit the pinnacle of computing hardware. Except, VR/ AR is a very demanding technology. It has unique and unsolved user experience problems. PC and console gaming has had 60+ years to get where they are now, and VR is still a very young industry.
Early VR headsets took available mobile phone displays and wedged some decent lenses between your face and the display, and that got us ‘Good Enough.’ As you would have it, a version ZERO, the V0 — the Oculus DK1.
Yet, You can’t just take a high res mobile phone display and expect it to still look high-res a few inches from your face, and then zoom in to a majority of your field of view and expect it not to pixelate … but that’s precisely what the first few generations of VR hardware tried. Heck, at launch, we didn’t even have the graphics horsepower to render content at 120hz. That doesn’t even consider all of the optical physics problems to make the displays stress-free for your eyes. My DK1 was fun, but I could feel my eyes getting stressed after 15–20 min.
In 2021, it was reported that Facebook/ Meta had turned a corner and sold 10 million of 12.5 million VR headsets. Proving that VR is now ready to start moving out of the innovator / early adopter phase and into the early mainstream segment of the marketplace. Companies on the VR train can now grow with the market. This is a critical inflection point in the industry. Hardware manufacturing is starting to lean into VR and develop products specifically for VR.
When Apple broke into the mobile phone business, the Palm Treo was a device for early adopters that wanted an internet-connected phone. There were apps, and you could look stuff up on the internet and send and receive emails. It was life-changing, but Apple improved on the UX and ditched the keyboard for a ‘larger display.’ It was a considerable risk, but it paid off. Apple has since dominated the mobile market for two decades.
So here comes Facebook/ Meta, and they just hit a similar growth curve that the Palm Treo had before Apple entered the market. It certainly fits Apple’s MO to join a growing demand, selling millions of devices and growing. At Apple’s size, they can’t create complex luxury devices that sell thousands of units. It’s not their business model. So it makes sense that they will enter the market this year or next; however, what will Apple do differently? How will they make an Apple-esq VR headset feel?
Breaking Down the Rumors of Apple VR
I’ve heard lots of rumors that the ‘Apple View’[a potential name floated around, UPDATE: or Apple Reality (AR), or even Apple Reality One] will have very high-resolution displays, more cameras than anyone else, and will be the lightest high-resolution headset on the market.
Weight — 350g?
Some rumors even claim it might be so lightweight that it competes with the ‘smart glasses’ category that Snapchat, Facebook, and Google have been trying to define. A 350g headset would be much lighter than the 550g industry average for headsets released in 2021. I wonder if this is a reasonable expectation, though; batteries have a finite weight — so any more powerful device will need a bigger, heavier battery to get decent battery life. Phones and watches need to work all day without charging, but it’s typical for a gaming laptop to go only a few hours between charges. How long do portable VR headsets need to hold a charge? Is it more like a gaming laptop? Or more like a mobile phone? The new M1 Macs can run all day for light workloads freeing you from your charging adapter and the wall. For reference, my iPhone 12 Pro with a case is 217g, and my Quest 2 is 528g and is very comfortable.
I hope Apple targets 4hrs of battery life or 2x 2hr VR sessions.
Processing Power — M1 or A15/A16?
Rumors have also speculated that Apple will go with an M1-level chip, but I think this is not quite right. The M1 is a 14W chip which is impressive for a laptop, tablet, or an internet appliance, but for a VR chip that’s 40% more watts than the Qualcomm Snapdragon XR2 powering the Oculus Quest 2 and the HTC Vive Focus 3, even though it’s got just over 2x the GFLOPS of the XR2 (GFLOPS — Giga floating point operations per second ).
For Apple to win on processing power, weight, and battery length, they will need to be more efficient than the M1. Some have speculated that Apple might ‘borrow’ processing from another nearby device, or even possibly from the cloud, but that takes power too. High bandwidth WIFI or 5G takes a lot of energy to maintain that connection. Off-device computing isn’t without consequence for the battery. Off-device computing also adds latency, affects comfort, reduces fidelity, and increases UX complexity.
Apple has yet to use cloud rendering for anything, I doubt they would start on a new device. By comparison, NVIDIA and Microsoft have used cloud rendering for GeForce NOW and Azure / Xbox, respectively. Instead, Apple likes to do ‘hard things’ on other products as a trial run and build up manufacturing supply. The cloud also has privacy concerns, which Apple tends to be more on the consumer side than Facebook, Amazon, and Microsoft.
The M1 was released in 2020 on a 5nm process for computers, not ultra-mobile devices like phones and watches. If Apple used a newer process, they could save some power, maybe generously 20–30%. Except, I also think that their VR chip will be more mobile-focused — so perhaps the newer A15 Bionic or something laser-focused on VR. The A15 might also be a miss, even though it’s only a 6W part, almost half the watts of the Snapdragon XR2, but its performance is virtually the same. Apple wouldn’t launch something with the same performance that Facebook launched in 2020.
UPDATE 10/7/2022: The iPhone 14 Pro launched with an A16 Bionic on a 4nm process, so it would be fair to estimate that the Apple Reality One might launch with an A16 or better CPU.
This leads me to the hunch that the Apple View will use an Apple Silicon chip customized for this application based on the 5nm (UPDATE: 4nm) or a newer process.
How many cameras, and for what purpose?
If the Apple View has nine(9) Cameras, like the rumor, they could be used in several configurations.
The Quest 2 and Vive Focus 3 each use 4x tracking cameras that do double duty, also passing through a grayscale representation of your room. This is a kind of Mixed Reality lite. By comparison, the Vive Pro 2 has 2x dedicated color passthrough cameras.
Honestly, if the rumors are true, Apple could really offer a lot of innovation here. The pass-through camera options out there create a flat environment sphere of your space, flattening your room into a wallpaper applied to the inside of a virtual globe. It’s good enough, but it’s by no means magical, and it’s hard for full AR-type experiences because developers don’t have a depth map to play with. Quest 2 does some edge and shape detection, but it’s too weak to build a game or experience off of.
A few years ago, Apple added depth map sensing camera systems using 2–3 sensors to the iPhone and iPad. So it might be possible that they plan to use that here. It’s been part of their strategy to work out tech on other products, and that might be precisely what they have been doing with iPhones and iPads, building up expertise with AR point mapping and object detecting.
2x Camera Systems, with three(3) sensors each equals six(6) color depth cameras+ four(4) tracking cameras equals ten(10) total cameras, so with some magic, maybe they found a way to use nine(9) cameras to accomplish tend(10) cameras worth of work, or perhaps the rumors are wrong, and there are a different number of camera sensors.
Another option might be to have four(4) tracking cameras and two(2) color pass-throughs, and then you’d still have three(3) cameras left to do eye tracking (one for each eye) and lip tracking. No standalone headset today offers this, and Apple seems on track to be able to deliver this. On top of that, Meta has teased launching a “high-end headset” project, Cambria (Oculus Pro or Quest Pro?), in 2022, which will have full face tracking and color passthrough. Project Cambria seems directly targeted to a 9-camera Apple rumor.
Even with a 9-camera system, VR headsets still have blind spots, so an extra few cameras in the back of the head could be fantastic. I’m not really sure what the ‘sweet spot’ will be for VR, but Apple might try to rush to that standard so that they can have a seamless UX platform to build on, kind of like how they ditched the keyboard before people were ready.
I feel like nine(9) camera sensors are on the low end or what Apple might launch. I think they could create custom silicon that gives them an edge to process more camera information from more sensors; they have a tremendous amount of expertise here to flex.
Controllers?
“Look, ma, no (hands)controllers.”
Apple might try to ditch the controllers altogether, or at least they won’t be included. Facebook and the rest of the industry have been talking about moving to hand tracking, so why wouldn’t Apple do it first?
The Apple TV remote is the smallest remote I’ve seen, the iPhone ditched the keyboard, and honestly, VR controllers suck. If you want to use a device with a keyboard, you have to blindly set them down and pick them up; if you want to use a mouse, the same thing happens. They are bulky and make a ‘mobile’ device more like a portable device. Ultimately they are just uncool. Everyone in the industry is trying to ditch the controllers. So Apple could really win here with perfect hand tracking and flawless hand gestures. Apple led the industry in multi-touch and pinch-to-zoom for nearly two(2) decades. Apple could define 3d gestures. Apple defined an industry with its multi-touch, and they could do it again with 3d gestures.
Maybe Apple could even create a device that can read sign language. …. Ok, now I’m getting ahead of myself.
If Apple skips on the controllers, they save on manufacturing complexity and cost and push that towards a more flexible interface, your hands. This might allow them to spend more on the headset and save money on the controllers. This just feels like the Apple way. They did this with the iPhone, Apple Watch, and many other features. Less is more with Apple.
The View’s Display
People also expect Apple to wow on the display, and the rumors seem all over the place here. They seem to diverge a bit and defy physics if you combine them. To compare, Meta (Facebook) is messaging that their “Cambria” headset will have “pancake” optics. The only thing we know about these “pancake optics” is that they will be flatter than the current Fresnel Optics in the Quest 2 and most headsets on the market in the summer of 2022. Some rumors even point to Apple ordering from the same manufacturers as Meta for ‘pancake lenses.’
On the high end, the Varjo XR-3 performs some neat optical tricks that make it the most visually impressive headset on the market. The XR-3 uses multiple displays per eye, with smaller and higher-resolution displays that holographically float in front of the larger peripheral displays. This is the sort of thing people would describe as a ‘Retina Display’ in the VR space. Some have speculated that the Apple View might cost $3000, which would put it in the price range of the cheaper version of the XR-3, the VR-3, with the XR-3 costing upwards of $5000.
Let’s remember that Apple is in the market of selling ‘millions of devices,’ so I don’t think a $3k device is going to be their starting point. Apple will pick a middle point to establish value, then work up and down.
Apple, on the whole, will want to be better than Meta’s new flagship, but they won’t do it by sacrificing form factor, and the Varjo XR-3 and VR-3 headsets are heavier than most headsets on the market now.
Combining the XR-3 displays with foveated rendering would create the impression of life-like environments, virtual high-resolution desktop replacements, and the ability to read text with limited eye stress. The XR-3 definitely sounds like where Apple would want to be, but it also sounds more like someplace Apple might depend on a laptop or desktop to provide the massive CPU/ GPU power needed to render all of those pixels at 120hz or “pro-motion” speeds. I think the XR-3 will remain upmarket and un-sexy by Apple design standards.
Conclusion and final bets
I’m very excited to see what Apple finally does, but without knowing a price point or a form factor, it’s probably impossible to guess what Apple might do. They have dominated the mobile space launching everything from watches, and phones, to tablets and more. They offer devices that seem to overlap with only screen size differences. Their products range from the Apple Watch SE at $279 to the $5000 Apple Pro Display XDR. It’s certainly in their power to launch a VR headset or line of headsets within that price range.
Apple also has a habit of providing just enough magic but also leaving achievable performance on the table to hit a yearly cadence of innovation. Keep early adopters replacing their gear and recycling and repurposing old hardware for a new lower price point. I wouldn’t doubt that they have prototypes all over that spectrum and are watching the market to see where they can maximize their impact in making a “luxury VR/ AR” headset. Whatever they do, they will want to sell millions of units in their first year to make it worthwhile.
With all of that said, my prediction is a device that has no controllers, hand tracking, face tracking ( eye and lips ), foveated rendering with one(1) display per eye, “pancake” style optics, but a resolution closer to the Vive Focus 3/ Pro2 or the Varjo Aero ( 2448 x 2448, 2880 x 2720 “5K÷2” ). I don’t think they will launch with a magical XR-3 display in ‘pancake form’ and will leave that for a later Pro form factor in years 2 or 3, or 5. I also don’t think Apple will have some overly innovative holographic display using waveguided lenses because the manufacturing pipeline isn’t there.
I think we are looking at an A16X-style chip specialized for VR; I expect it to be faster than, say, a theoretical Qualcomm Snapdragon XR3 since the XR2 was released two(2) years ago in Q1–2020. It will have specialized cores like the A15 has ‘bionic,’ but it will be for VR-dedicated tasks like tracking, eye movement, facial expressions, foveated rendering, etc. …
When Apple unveils the device, it should feel ahead of its time, magical, but won’t be groundbreakingly advanced. It will feel smaller and lighter than expected, but it won’t be in a different category than the Meta Cambria. Apple won’t go after HoloLens customers or Magic Leap — the sales numbers are not there yet.
Its battery might last longer than expected, maybe 3–6 hours, but it won’t feel like it’s breaking the laws of physics. I don’t expect anything wild, like light field displays or tech still early in its manufacturing/development stages. The Quest 2’s battery is too short.
Nonetheless, it will be awesome, and I bet every VR enthusiast will want one. I expect the specs will make it better than anything on the market, probably even a bit better than Facebook’s / Meta’s project Cambria.
Interesting Apple View Rumors
Kuo: Apple AR/VR headset will charge using the same 96 W power adapter as MacBook Pro
Report: Apple Won’t Join the Metaverse Hype With Its Headset
Apple’s VR/AR headset will allegedly be focused on ‘bursts of gaming, communication, and content consumption.’
Report: Apple’s upcoming AR/VR headset will feature an ‘innovative three-display configuration.’
Oculus Quest Pro and Quest 3 release dates and new display tech leaked.
