Virtual reality opens the doors to a new era for user interface design. Oculus VR speaks to Develop about its opportunities

Virtual reality doesn’t present user interface design with its first opportunity for transformation.

The dawn of 3D long ago afforded games makers the prospect of moving beyond flat heads-up-displays and conventional menus. And when mobile gaming finally realised its potential with the arrival of smartphones, those charged with implementing UI had a chance to establish the new standards of the virtual gamepad. 

Neither presented an easy transition, and for many releases orthodox menu design remained the best option. Mobile games makers wrestled with how to balance UI button placement with the problem of players’ fingers obstructing the screen, while those crafting 3D console games faced the difficult task of rewriting the broader grammar of user interfaces.

Those are challenges yet to be fully answered, and now – predominantly thanks to the arrival of Oculus Rift – a vast new opportunity is here with the advent of a new kind of VR. Of course, virtual reality has lurked in the badlands of the technology industries for decades, but with Oculus Rift, for the first time it looks set to make its way to consumers in a meaningful way. The technology has arrived and it’s already won the support of big names, such as John Carmark. So it is VR games that will walk hand-in-hand with Oculus Rift as it saunters into the public’s collective conscience.

CHANGE IS ON THE MENU

The fact is that VR games need a new type of UI, as they are delivered in a new way. The challenge is similar to that faced by film and game makers as the recent wave of new stereoscopic 3D technology arrived in homes and cinemas. And equally, it is also a substantial opportunity.

Two games industry professionals that know that opportunity are Oculus VR’s CEO Brendan Iribe and vice president of product Nate Mitchell. Both previously served at Scaleform, the provider of the industry’s most prolific UI middleware, which was eventually picked up by tools giant Autodesk.

“I think one of the most impressive things about VR is just the ability to have a sense of presence in a gameworld; that feeling that you are really there,” says Mitchell as he begins to explain how UI may evolve with Oculus Rift.

“Until now players have seen HUDs and flat menus, and I think the that opportunity with virtual reality here is evident already in the best early VR UI today, where they are really taking cues from the real world, or perhaps the dreams of the future.”

According to Mitchell, games makers only need to look to Google and Microsoft’s work with augmented reality, and the UI already being conceived for unusual hardware like the spectacle-shaped Glass. Those developments, say Mitchell, are serving as an inspiration to the first wave of games developers building titles for Oculus Rift.

“Developers are starting to look at UI in VR spaces in terms of how they would want to interact with that world in real life, and take information from objects in that world,” Mitchell adds.

VIRTUALLY REAL

Virtual reality is, of course, defined by its ambition of simulating real life, which is exactly where developers should look for inspiration when designing UI for Oculus games, suggests Iribe. There is arguably no UI in real life, but still, we garner information from the world around us through signage, digital displays and the devices that let us interface with our environment, and games developers now have the opportunity to reflect that in games.

“When you suddenly have VR available to you, you quickly have so much more area to play with,” explains Iribe. “You’re no longer confined to a small 2D screen anymore, and you also have a number of new mechanics to work with.

“It’s no longer about just a keyboard and mouse or gamepad; there’s this new headtracking mechanic available. You really do have to think about all of that when you design a user interface, and suddenly you have to think about how you would interact with the user interface if it existed in the real world.”

Suddenly, viewed through an Oculus Rift headset, 2D HUD elements may be intrusive? Surely it is better to glance down at your gun to check its ammo than look to an arbitrary sign floating in your field of view?

It’s something many console developers have already tried implementing in traditional 3D console games, and titles like Dead Space were among the first to use the technique with particular grace. And with VR, as players can really move their head to look down, those techniques can be used with even greater effect.

But not all UI can be placed on in game objects; targeting reticules and destination markers must exist in the game volume, while certain elements such as player instruction text and subtitles may best serve their purpose as 2D assets.

But should they lock to the players’ head movements? Or should they instead affix themselves to a point in the gameworld? Does screen edge even have meaning in a VR environment? What will feel natural and comfortable to glance at when the head and neck are serving as an input? The questions are endless, but sources of inspiration are plentiful, and one in particular is likely to have much appeal for games developers interested in the tech.

“Think of how people have imagined UI in science fiction, which is somewhere Hollywood has done a great job,” offers Iribe. “That is now where many people are starting to look for inspiration, and those kind of ideas are how a lot of people are going to realise UI in VR games.”

A NEW GRAMMAR

The challenge for conceiving how UI should work in a VR space is a more academic matter. Developers are going to do more than just learn new methods, for they must liberate themselves from the previous rules of UI design. It’s not an easy task, but an essential one, argues Mitchell.

“I think the biggest challenge at the onset will just be for developers to break down their preconceptions and get away from the conventions of user interface up until now,” he says.

“They need to begin again with what works for VR. I’ve already talked to a number of developers at game studios that are already doing that. A lot of UI and UX designers are now sitting down with a Rift for the first time, and they are integrating it with their work very quickly and jumping in, and realising the challenge and opportunity of user interfaces in virtual reality.”

Of course, if those designers are successful in integrating a new kind of UI into a virtual reality, as they strive to ape the ‘user interface of real life’, some speculate UI as a distinct discipline may disappear, at least in part. After all, if in-game objects alone can communicate needed information to the player, then surely UI is just another task for those conceiving the likes of weapons and gameplay items.

“I don’t think we’ll see UI disappear, but it will become far more integrated into a gameworld,” offers Mitchell on the matter.

“What we’re already seeing is that a lot of developers are pursuing that type of experience, because it is more immersive, and more effective in not seeing the player be taken out of a game experience to use interfaces. Rather than have those ammo-counts floating in space, and being reminded you’re in a game, with VR UI a player can keep moving through the world, just glancing down in-game to see their remaining ammunition.”

DEEPER THOUGHT

Player immersion is one of games design’s ultimate goals. If a consumer strapped into a Rift forgets they are playing a game, the developer behind that game has perhaps triumphed. And it is that that VR offers UI designers: a chance to take player engagement to a new level.

In fact, according to Iribe, VR provides what UI as a discipline has been longing for; a place where user interfaces can escape the arbitrary forms that have for decades constrained the potential for communicating with the player.

“VR is absolutely an opportunity for UI design, and really what user interfaces have been waiting for,” claims Iribe.

“VR takes UI to a whole new level, which is just super-exciting for games developers, and especially UI developers. They’re now able, to me at least, to realise UI in more compelling and innovative ways. UI is no longer a 2D element; it’s not going to just be about pieces of text and icons anymore.

“UI was never something particularly exciting for the user, and menu screens in particular. Now it can be a very exciting part of the game. In VR UI can be the CG graphics we’ve [seen] in Minority Report and District 9; that’s now something we’ll see in games.”

Such optimism is hard not to warm to, but as team sizes, budgets and time available across the industry continue to feel the force of a changing sector, it’s important to consider how realistic it might be for teams to embrace UI’s new VR era.

Fortunately, most VR games present 3D worlds technically similar to those that have long beguiled traditional console games, and as such, most existing UI middleware is already fit for purpose.

“There’s a great middleware technology called Scaleform,” says Brendan with a smile.

“With Scaleform, and other technologies like Scaleform, you can map the 3D surfaces and project out into the 3D world, which will be very useful for realising UI in VR games.

“Those kind of abilities have only got more popular and people continue to use those kind of features, and now you’ll see that go to a whole new level. The functionality of tools is already there in a number of UI systems – although not all UI systems, as many still focus on 2D overlays. Certainly with Scaleform developers are already able to do a lot of fun stuff, such as texture mapping onto 3D objects, which will be perfect for UI for VR.”

Games development is never easy as such, but relative to the task of building interactive realms, with the tools already available and familiar, implementing innovative UI in VR looks set to be technically well within the realm of most studios’ abilities.

With the development tools in place, and VR finally ready for consumers and developers, user interface design looks set for another new transformation. There are indeed challenges, but those are vastly outweighed by the new opportunities, and who knows?

We may even see the traditional game menu disappear from games altogether. Exactly what direction UI design will take in VR is now down to the developers, and it is that that makes it such a fascinating field.

Written by: Will Freeman, Develop (via Presence)
Posted by: Situated Research