Imagine a world without phones or laptops, a world in which the only device you really need is a pair of glasses — acting as your computer, your screen, your sole companion to the digital world. You would work in immersive digital environments, teleconference with 3D avatars of colleagues in overseas offices, all realistically projected in three dimensions. You could grab virtual objects with your hands and manipulate them like you would in the real world, play immersive games without a bulky VR headset or virtually fit clothes, in the comfort of your home.
This image of the future might still sound far-fetched at the moment, and it probably will be for at least a couple of years. During the last few months however, there have been increasing rumours about Apple releasing an Augmented Reality headset; Apple Glass. I believe that the introduction of such a device will eventually have a significant impact on how we live and work.
What is AR?
Augmented Reality (AR for short) is the experience of a real-world environment enhanced by computer-generated perceptual information. Devices such as tablets or smartphones can deliver this experience by projecting data (a 3D model or the user interface of an app, for instance) on top of the image recorded by a camera. AR glasses would be another way to provide such an experience, by projecting the computer-generated imagery directly in front of the eyes — or even onto the retina — overlaying images on top of our surroundings. At the moment, there are several Augmented Reality glasses available, but none of them have had the appeal or functionality to sell in great numbers. I believe Apple would change this by entering the market with a breakthrough device.
Example of an AR headset, the first product by Magic Leap. Creating a lightweight, attractive & functional AR headset has proven to be a hard challenge. Photo by Bram Van Oost on Unsplash
The Final Days of the Screen
To understand why Apple would launch an AR product, we need to look at how we currently use computers and smartphones in our daily lives. Since the onset of widespread computer use, we have become accustomed to consume visual information from 2D screens. However, we normally experience the world around us in 3 dimensions. We are all born with two eyes that enable us to perceive depth through [stereo vision, parallax and convergence.](https://www.reallusion.com/crazytalk/help/crazytalk6/09_3D_Vision/The_Concepts_of_Stereo_Vision.htm#:~:text=Parallax,of depth in the media.) 3D vision lets us accurately estimate how far away objects are and our orientation in space, which impacts our mood and behaviour.
So far, 2D displays have worked pretty well for most uses. There are a couple of disadvantages as well, however. For instance, what can be shown on a computer screen is always restricted to its size. The growing size of smartphone screens over the last few years shows that many prefer larger display sizes. At some point, phone screens can’t grow anymore before the hardware becomes too big. The same applies for desktop and laptop screens; large displays are at the expense of desk space and portability. Foldable screens are one possibility to partly overcome these limitations. However, foldable screens are complex and expensive to produce. In addition, a 2D screen can never convincingly show the third dimension. We have developed ways to trick our mind into believing what we see on the screen is 3D, but the imagery remains in 2D.
Limitations
What if we could overcome these restrictions and experience our data in 3 dimensions, free from the limits of screen size? This is where Virtual Reality and Augmented Reality headsets come in. With an AR headset, comprising a head-mounted display and some sensors, it is possible to view data in true 3D within a large field of view of the user. By unlocking the third dimension and unlimited screen space, AR and VR change the way we interact with computers. When combined with hand tracking, you could interact with a computer using your hands, manipulating objects like you would in the real world. This enables new ways of digital collaboration, playing games, creating and using content. Realistic 3D training environments and virtual classrooms would reshape education. Creation of digital art would be taken to a whole new level.
Creating such a product is not easy, however. A lot of technology needs to be crammed into a very small space. The device would need to project images realistically on top of the environment, it needs to process and transfer information and most of all: it needs a decent battery life. All while keeping the form factor sleek and attractive.
Why Apple Glass?
Apple’s mission is to deliver the best user experience through innovative hardware and software. What does this have to do with AR? As I discussed, AR glasses would change the way we interact with computers. It opens up a lot of exciting possibilities for new kinds of software (read: new markets). Apple has a laser-like focus on improving their user experience and attractiveness, resulting in products like the iPhone and iPad. The user experience of computers has changed dramatically compared to the 80s as a result of technologies like the mouse and the touchscreen, both introduced to the masses in their most user-friendly form by Apple. I expect they will try to do the same thing with Augmented Reality.
At the moment of writing, no AR headset exists that is both lightweight, compact, with a wide field of view, a high-resolution screen and an attractive appearance. Most current headsets are clunky, unattractive, offer limited functionality or don’t directly work out-of-the-box with a multitude of user-friendly apps.
These factors are all critical to Apple; they are the benchmarks Apple sets for their products. Given their mission, history and almost unlimited resources, I think Apple is the one company who could solve the technological problems and bring an appealing product toe the masses.
I don’t expect Apple to deliver a perfect product, however. Some functions might be missing, like the first Apple Watch did not have all the functions compared to the current generation of Apple Watches. They will try to get the form factor right with a pleasing user experience and just the right amount of functionality, before building out the technology bit by bit.
A track record to be proud of
Apple has an amazing track record with releasing revolutionary products. For example, look at the iPhone: it was not the first phone with a touchscreen, but it determined the shape of all subsequent smartphones. Competitors like Nokia were suddenly years behind. Only after a few years, competitors were able to introduce alternatives that would come close to the user-friendliness of the iPhone.
In addition, Apple has a lot of loyal users and ardent supporters who like to invest in their latest products, even imperfect ones: the archetypal Early Adopters. This user group is essential to launch innovative products like AR headsets successfully among the world’s population. The iPhone was initially popular amongst the early adopters, before it started attracting a wider audience. Nowadays, owning an iPhone is nothing special — but only because a small group was wiling to buy a breakthrough yet immature product.
All things considered, I expect Apple to be the company bringing an Augmented Reality-based form of computing to the masses.
What steps has Apple already taken towards AR glasses?
According to an increasing number of leaks and tech journalists, Apple has already taken a number of important steps to arrive at an Augmented Reality headset. Patents by Apple describing AR and VR techniques have been surfacing for a decade — getting increasingly frequent during the last four, five years. Let’s walk through the leading indicators of what’s coming.
A growing number of acquisitions is the first sign that Apple is working on something. For example, they bought a company developing lenses for Augmented Reality headsets (Akonia Holographics). Other acquisitions included Vrvana (VR glasses), Metaio (AR software), SensoMotoric Instruments (eye tracking), IKINEMA (motion tracking), and very recently NextVR (Live VR events).
ARKit is another important ingredient for the roll-out of AR glasses. Launched in 2017, ARKit is an Augmented Reality platform for iOS devices that enables developers to create AR apps on iOS devices. It is a substantial element of the ecosystem for Apple’s AR product, as it is already being used by many developers. This is crucial for the success of Apple Glass, as the more apps are available on their AR device, the more attractive the product will become for the customer and the higher the chance of success of Apple’s headset will be.
Laserbeams
Furthermore, Apple has prepared a crucial hardware element for its future glasses: a LIDAR sensor. LIDAR (LIght Detection And Ranging) is a technique that maps the surroundings in 3D through pulsed laser beams and a sensor. It is a vital component for any AR glasses, because the 3D images must accurately interact with the surrounding physical environment. By incorporating a LIDAR sensor into their new iPad Pros and potentially future iPhones, Apple is preparing both the industry and iOS developers for what’s coming.
It makes a lot of sense to incorporate such a crucial sensor in existing products like the iPad, since they are produced in extremely high volumes. This reduces costs, speeds up further development and enables further miniaturisation — essential for keeping Apple Glass small and attractive.
The new iPad Pro features a LIDAR sensor to map the environment for a better Augmented Reality experience. Photo by Daniel Romero on Unsplash
What happens next?
If Apple successfully releases its headset, we might experience another transformation in how we use computers. In a few years’ time, the traditional display as we know it might disappear. Instead, we would wear an AR and/or VR headset and carry around a computer that performs the calculations needed to show the 3D visuals (probably our smartphone). Eventually, all processing would occur on a chip within the headset. Computers in their current form will disappear, at least for day-to-day tasks; the AR headset would be our all-in-one computer solution. Browser windows, text editors and videoconferencing, photo and video editing programs can all be used entirely in three dimensions. The office of the future would look quite different; why sit at a desk when you can stand or even walk on a treadmill while doing your work?
New hardware accessories would appear for specific users, making their work in 3D more pleasant and ergonomic. Lifting your hands for every gesture quickly becomes tiresome, after all. I’d expect dials (such as the Microsoft surface dial), wireless keyboards and touch pads to still be used. A virtual keyboard and speech might be sufficient for most people, however.
Telepresence would be another significant opportunity. Realistic, personalised 3D avatars will replace 2D videoconferencing, enabling even more forms of virtual collaboration. Remote working will become more personal and effective; some professions might not require face-to-face contact so often.
Of course, the competition will rapidly match the level of user-friendliness provided by Apple’s glasses. Numerous Chinese manufacturers will produce cheaper variants with same overall functionality. In a few years, the entire market will have changed. Hunching down towards the glow of our smartphone will be a thing of the past.
A new era
The launch of Apple Glass could have an impact on many elements of our life. The societal changes that have been caused by smartphones have been profound — I expect this technology to be equally transformative. Entire industries will emerge while old ones will have to adapt. Remote working will be more attractive and effortless. Our reality would take another leap to a futuristic ‘science fiction-like’ world, hopefully for the better. The question remains: how soon will this happen? Will Apple really pull it off? The future will tell.
///
I do not own stock in Apple Inc.
Ruben Hanssen works as a consultant in Civil Engineering, developing stakeholder management tools utilizing technologies like Virtual Reality, Augmented Reality and 3D animation, as well as a freelance photographer & videographer.
Fraai stuk Ruben!
Dank je Peter!