A World of Digital Ghosts
Reality is a funny thing. As Michael Abrash stated in his Keynote from F8 2015: “Our reality is nothing more or less than the sum of the conclusions reached by a variety of unconscious processors driven by a body’s-worth of sensors”. We see, hear, smell, feel, taste the world and the brain interprets these signals into a more or less comprehensive image of “reality”. However this image is very rudimentary and based on a lot of assumptions and guesswork by the brain. As it turns out, every sense can be easily fooled, tricking the brain into accepting artificial things as “real”. Not only are we as humans incapable of objectively seeing the world, it seems we’re actually pretty bad at it.
Mixed Reality will be the new Smartphone
The big improvement with the current wave of Augmented and Virtual Reality technologies is that they have become good enough to fool the brain into accepting their virtual input as reality on an instinctive level. And this is why we see those sometimes funny, sometimes scary videos where people have extreme reactions within AR or VR.
As a result we will soon be able to create a believable Mixed Reality by injecting purely digital dimensions seamlessly into the physical world. With Mixed Reality we can still see the real world or at least its reconstruction around us while adding digital input. We will be able to control the level of digital input, ranging from no digital augmentation to fully replacing all input as AR and VR both have this as their end game, they just approach it from different ends of the spectrum.
Until now, screens were our primary “window” into the digital world. Traditionally these are either big and fixed in place or they are small and mobile. However with MR we will be able to create screens of any size whenever, wherever we want. Essentially screens will follow us, conveniently placed, extending and changing as needed. This will continue to the point where they are not screens anymore, but fully volumetric augmented experiences, which will seamlessly integrate into the real world environment.
Compare this continuous digital layer to the mobile phone, which we use in a similar way: We take it out when we want to enrich our daily routine with information, communication, media and other things. But instead of having an extra screen annotating our experience, imagine the phones output as an overlay within our natural vision. Essentially MR is a smartphone, but with a different kind of interface.
As mobile phones became ubiquitous in our life, MR will do the same thing: Fully untethered, able to run a full day, ergonomically enough to actually wear them on out body all the time. Hence Mixed Reality devices offer the potential of disrupting every piece of digital technology in our lives, much like smartphones did. In other words: The long sought after “iPhone killer” is not another smartphone, it will most likely be the first light-weight, fashionable Mixed Reality device.
Granted, the hardware is not exactly consumer ready right now. Currently, the most advanced Augmented Reality kit, the Microsoft HoloLens, looks more like industrial protection goggles. Its battery lasts for about 2.5h, the field of view (the actual size of the “window” into the digital dimension) is rather small and weighing 580 Grams it’s not exactly comfortable to wear either. While all those features are actually amazing technological achievements that push the limits of what’s currently possible, the device is not something a normal user can wear and use all day. The same is true for the Oculus Rift and other Virtual Reality sets, which are usually tethered to big machines and only usable at room scale at most.
But I assume that a lot of the technological challenges will be solved within the next couple of years. Looking at the 10 year evolution of the iPhone, the A10 Fusion chip in the current iPhone 7 is 120 times as fast as the chip in the first iPhone, the graphics chip is 240 times faster and the screen evolved from 320 × 480 pixel resolution at 163 ppi to 1920 x 1080 Pixel at 401 ppi (iPhone 7 Plus), all while keeping roughly the same weight.
As for HoloLens or Oculus, at this point they represent the first glimpse we have into this future. They are not the first iPhone, but rather the equivalent to the Nokia Communicator, showcasing the rough capabilities, not necessarily the final form factor, interface or experience.
Interacting with Ghost Dimensions
In a way, Mixed Reality is something very private. As an example, MR glasses augment the visual input of the wearer, so they are the only ones that can see this specific view of the world. Physical and digital input creates a personalized version of reality, making users the inhabitants of their own private world. Everybody else by default only sees part of this reality, the shared physical one. But it being a mixed reality with digital dimensions has a couple of implications.
As processing power, storage and bandwidth increases, there are less limits at how many dimensions can exist at the same time.
This means that there will be not only one mixed reality, but many parallel ones. While the physical reality stays (more or less) fixed, the digital overlay can change as easily as switching TV channels. Some of those channels might be educational, some inspirational, some might feature entertainment, others would be for work and so on.
Depending on the currently selected channel, the usage, behavior and capabilities of physical locations might change as well. A chair is a chair and not a chair as it might be augmented or adapted to be something else entirely in some digital dimension. Access to capabilities might be controlled via interfaces in a digital dimension that only authorized users can see and inhabit.
As digital systems are connected, dimensions can be freely shared.
Most digital systems today are multi-user systems. We use them to communicate, collaborate or for entertainment. So it seems natural that these digital dimensions will be shareable as well.
The best analogy we have for this are Massively Multiplayer Online Games (MMOG) like World of Warcraft or EVE Online as well as Virtual Worlds like Second Life. They simulate a virtual world and track individual objects plus their states spatially within that world. Users can log in as avatars and interact with the world, objects and other avatars according to the rules defined in that space. Mixed Reality will be the same.
As a shared experience, MR is not private at all. Instead your avatar (your digitally simulated representation) might appear in everybody else’s digital dimension, transporting our virtual avatars across digital dimensions in different locations. If we consider the option to change reality completely for each individual, we can only differentiate between “Forcefully shared realities” (= physical reality), “Acknowledged shared realities” (= multi-user MR) and “Personal versions of reality” (= single user MR). All of them can be manipulated digitally at will.
As more physical things will have digital interfaces, the digital dimensions will have an increasing influence on the real world.
The Internet of Things is about adding digital capabilities and ultimately behaviors to physical things. This will include every thing that can be in any way, shape or form be digitalized. Physical objects, no matter how mundane, will have digital capabilities and behaviors such as being able to communicate or react to us in a personalized way. As these objects become increasingly digital, centralized control units will emerge that add an additional layer of convenience and intelligence. Already available examples would be Amazons Echo or Google Home that can be used to control smart environments. As these control units evolve, they will integrate themselves into Mixed Reality scenarios, bridging the gap between the physical world and digital dimensions.
That means that actions within digital dimensions will have direct influence on the physical world as well, from opening doors, controlling lights and other environmental controls to moving mobile, autonomous things within the physical space.
All this enables an experience comparable to living in a hyper personalized theme park where every experience is catering to each individuals needs, feelings and dreams while connecting them to others around them, either through magical physical things or through purely virtual magic. A world where we can be virtually omni-present magicians, able to see ghost worlds and use their powers to manipulate matter.
Some Poltergeists will emerge
If we believe in those assumptions, it will lead to another set of implications, this time on a social / society level.
We will have to re-evaluate our notion of “property”.
Let’s assume you are walking down a street, which contains a billboard or poster of some kind, showing an ad. Your current digital reality channel however hides this ad and replaces it with different information. For example your #CatsNotAds channel would just show you cat images instead. This will spark the same discussion as with digital ad blockers, only bigger. We are used to be able to control what we regard as our physical property, however some digital reality channels will purely exist to avoid or even deface it. Your home won’t be your castle anymore.
Advertising will not be a problem, branded experiences will.
If one ad on a billboard is replaced with another in real-time, it might be a problem for our sense of “spatial ownership”, but the real issue emerges if you think about branded experiences. Currently, branded content is a form of advertising where the brand generates content for one specific channel that is somewhat similar in appearance to the content normally found there. The intent is to “blend in” so people will not notice that they consume branded and thus biased content. Now imaging your whole world view as a branded experience. Neutral or conflicting content (objects, people) are replaced with preferred equivalents to create a branded world view according to brand values or intentions, combining the implications of branded content with those of a filter bubble.
Statekeepers will become hyper powerful.
As these digital objects within virtual dimensions interact with the physical world, everything needs to be tracked in all dimensions (physical and digital). This calls for systems that keep a record of every object in existence (real or virtual), including its behaviors, capabilities and states and locate that on a map of the world as well assigning it to a specific channel. These systems would provide consistency and concurrency of the digital channels as well as ensuring constant synchronicity between the physical world and the digital channels.
Ultimately, these systems will emerge as the infrastructure of the world. And as such, they will shape society. Intentionally or unintentionally, the systems will influence how people behave. And this makes providers and operators of said systems very, very powerful.
People not able to see digital dimensions will be handicapped.
Imagine you are blind. The infrastructure of the world is largely designed for sighted people and while there are some instances that are universally designed with accessibility in mind, it unfortunately is not the norm. Now imagine that you have perfect eye sight, but are exactly in the same situation that you just can’t see. People not having the ability to see and interact with digital dimensions will encounter problems navigating everyday life. This will range from “not being able to see signage or information” to “not being able to interact with physical things as their interfaces reside in a digital dimension”.
A spiritual journey
While you can’t ignore these implications for society, I still remain optimistic that this is the right way to go. As stated, screens are wrong. The smartphone is wrong. We need to break out of that window frame and weave technology into our environment. Because this is what we collectively want: Technology that adapts to us, that aligns with our physiology and our psychology. We want the world itself to be better through technology, not have technology that only annotates the world. Over the last decades this has been firmly established in our collective minds through books, paintings, movies and everyday designs. In those visions, life is more open. More inclusive. More mindful and social. Effortless and always available for everyone.
I believe this can happen if we have those discussions around emerging ethical challenges and agree as a society on how we want to experience this future. Well, it will happen anyway, but I’d rather prefer we clear a couple of those issues mentioned above. And we need to think carefully about scenarios and experiences we want to enable. But I believe these super powers are worth having some debates.
References
- For an anthropological perspective on smartphone usage and how we intentionally and unintentionally adapt our behavior to compensate for their shortcomings, see Mobile Ordinary Gestures by Nicolas Nova.
- For more about how we perceive “Reality”, watch Michael Abrash’s Keynote from F8 2015.
- For more about Augmented VR, watch Michael Abrash’s Talk at the Oculus Connect 3 Event.
- More on Mixed Reality being a Massively Multiplayer Game, see Raph Koster’s articles “AR is an MMO” and “I really did mean “MMO”.