Author Yiliu Shen-Burke

Published 14 January 2022

Updated 19 April 2022

The idea of displaying digital information spatially is almost as old as computing itself. But the technical requirements of virtual and augmented reality are high, and there have been many false starts over the decades. In 2012, Oculus proved that progress on smartphone hardware had inadvertently created most of the missing pieces for consumer virtual reality, and its subsequent $2 billion acquisition by Facebook kicked off an industry-wide race to own a piece of this new pie.

Spatial computing is utterly transformative. Virtual and augmented reality make it possible to free the digital from our flat little screens and release it into the world around us—or even to create worlds entirely from scratch. You could wear a pair of comfortable smartglasses all day long. For an industry that’s addicted to selling our attention to advertisers, the prospect of a technology that would let it mediate our very reality is impossible to ignore.

Ten years after its Oculus acquisition, Meta is spending $10 billion each year to dominate this nascent space with its vision of "the metaverse”. It’s almost certain that its rivals are spending many billions more. It looks like these efforts are starting to bear fruit, and once Apple openly enters the fray, spatial computing will have completed its decades-long journey from visionary pipe dream to fact of everyday life. But its ultimate scale—is this the next smartphone, or the next smartwatch?—will depend on how much people need it.

What do we need virtual and augmented reality for? Not much, at the moment. Hardware and software are a chicken-and-egg problem: self-sustaining growth in one requires the other to reach a high level of compelling-ness. That’s why in addition to investing billions to solve tricky technical challenges specific to a computer that you wear over your eyes, companies and investors are also spending billions to kickstart (and control) the spatial software market.

Some think we’ll really love playing immersive social video games. Others believe this is the future of personal fitness. Some are betting that headsets will replace Zoom and let anyone meet with anyone else, from anywhere. Others are hoping it’s soft-skills training, or athletic training, or architectural visualization, or industrial troubleshooting, or battlefield intelligence, or selling fancy cars. And a few are dreaming really, really big.

All these angles have smart people and smart money behind them. I don’t doubt that many of them will work out. But I think there’s one set of paths in the spatial computing idea maze that is being under-invested in, relative to how much impact they could have: general-purpose spatial tools for thought. This phrase needs a bit of unpacking, so consider the question: What would be virtual or augmented reality’s equivalent of the spreadsheet? (By which, obviously, I do not mean literal cells in a 3D grid.)

Using a paper spread-sheet for accounting was cumbersome, error-prone, and costly. Their utility was limited, as was the number of practitioners who could justify the hassle. In 1979, VisiCalc made spreadsheet operations automatic, accurate, and instantaneous. This created a whole new method of quantitative reasoning, which became accessible to a broad audience. The spreadsheet lets us solve knowledge problems that would be difficult or impossible to otherwise. Although it has origins in accounting, it’s useful for number-crunching in any domain. And it only makes sense within the context of personal desktop computers.

Analogously, Softspace’s mission is to build a general-purpose tool that harnesses spatial computing to help us solve problems that would be difficult or impossible to solve otherwise. The fact that there doesn’t seem to be as much work being done on such tools as there is on games, or fitness, or social metaverses probably says at least as much about incentives and imagination as it does about relative merits. We’re doing this both because we see an open opportunity in the market, and because improving our ability to think is one of the highest-impact kinds of work we believe we can do.

Unfortunately, it’s not enough to look at a shiny new technology, conclude breathlessly that it’s going to change everything, and profit (though many appear to favor this approach). We need a theory of how this change will happen, and a plan for validating and updating it in the face of empirical findings. Which concrete problems could a spatial tool for thought help us solve? What real benefits or functionality does spatiality bring that don’t exist in flat interaction paradigms? What should our process of discovery and invention be?

Our starting point is the observation that when people need to make sense of a lot of information, they instinctively want to lay it out in front of and around themselves. This technique of information spatialization can be seen in methods both formal (mind maps, flow charts, mood boards) and informal (walls full of stickies, browser tabs open on multiple monitors). What is it about spatialization that’s so helpful for sense-making?

One possible answer is that information spatialization creatively leverages our highly-evolved abilities to perceive, think about, and manipulate things in space. These powers have helped us survive and thrive in our physical environments for millions of years. Some, like spatial memory, are so startlingly effective that they might be better described as latent superpowers. By substituting ideas for objects, we can apply these superpowers to knowledge instead.

Another reason might be that the geometric properties of space are particularly well-suited to sense-making. Linear perspective makes it possible to see a lot of information from any single point, effectively increasing our working memory capacity by an order of magnitude or two. It’s also possible to encode various meanings in position, rotation, and scale, which then lets us perform complex semantic operations just by moving things around. In a sense, we can harness Euclidean space itself to the task of remembering and reasoning about ideas.

A deeper explanation might be that sense-making is actually equivalent to deciding how to place things within a wider context and relative to each other, either literally or abstractly. Our experience of the physical world could be so dominant that our brains are wired to use spatial metaphors—maps, trees, charts—to understand everything. This is the strongest potential claim: that without the ability to “position” disparate ideas, we cannot relate them as pieces of a greater whole.

To “make sense of information” has an intuitive ring to it, but it’s nebulous. We’re actually interested in a concrete kind of knowledge work, one with a specific structure and objective. It takes as its input a complex collection of data, and produces as its output an actionable distillation of that data. Designers do it when they turn walls of project research into a pitch for a new design. Entrepreneurs do it when they condense piles of market research into a business plan. Researchers do it when they develop an elegant, powerful explanation that binds together a myriad of observations about the world.

It’s not self-evident that these examples are instances of the same kind of problem-solving. But our intuition—developed both through first-hand experience and conversations with users—is that it’s possible to identify a common pattern of thinking at the heart of each: synthesis. The goal of synthesis is to develop a framing idea into which all the pieces of a puzzle fit, so that a messy, intractable problem reveals itself to be a cleaner, tractable one. A great synthesis is simple, seems obvious in hindsight, and suggests effective actions for achieving desired outcomes.

Synthesis is one of the most important kinds of thinking that we do. Even the smallest corner of the universe is filled with infinite complexity. The ability to encapsulate that complexity within a narrative, or diagram, or heuristic, makes the world knowable, and gives us agency in it. Within a simplistic model where creative knowledge work consists of collection → synthesis → production, synthesis occupies the crucial, magical step between that which is, and that which is yet to be.

Synthesis is also hard. It’s an uncertain, iterative process that demands a mix of logical reasoning, trial-and-error, and intuitive leaps. Combinatorial complexity arises from the many ways data and ideas can relate to each other, and this complexity makes synthesis exponentially harder as the amount of input information increases. Unfortunately, the challenges we face have more moving parts than ever, and the quantity and quality of information we have about everything is only growing. How do we keep up in this accelerating, bewildering world?

To improve our ability to synthesize, or to do any kind of knowledge work, we can upgrade our brains or our tools. The latter is easier to do than the former. Much of our civilization is only possible because of remarkable tools for thought that make calculation, visualization, simulation, coordination, communication, and other demanding cognitive tasks easy, fast, and repeatable.

If spatialization is a core technique for knowledge synthesis, then spatial computing represents a significant expansion of the opportunity space. Before virtual and augmented reality, there was a hard tradeoff between tools which are spatial, and those which are digital. Analog methods, like a wall covered in text and images, could far better engage our spatial superpowers than a desktop or mobile app. But any digital tool is superfluously superior at storing, transmitting, and processing data. Spatial computing collapses that tradeoff.

Even the builders and users of flat software are starting to explicitly recognize the value of working with information spatially. Miro, Muse, and Milanote are examples of desktop- and tablet-based tools in which information is displayed on a plane. Personal note-taking apps, like Roam, Obsidian, and Walling, have incorporated “graph” views to visualize notes and their interconnections within a knowledge network. The existence and growing popularity of these spatial interfaces is evidence that people are seeking something beyond the conventional (linear) document.

But spatial computing isn’t only about seeing things in stereoscopic 3D (although this already opens up significant new brain-computer bandwidth). Virtual and augmented reality headsets have spatial tracking abilities that enable much more subtle and powerful interactions than those a phone or laptop can support. Head tracking lets you look around and move through information. Hand tracking restores the connection between thinking and doing that the mouse-and-keyboard paradigm so drastically narrows. I believe that a natively spatial tool for sense-making could be dramatically better than anything that runs on a flat screen.

In progress...