Project Natal brings natural user interfaces the office

Mundie: The Desktop Of The Future Is A Room
Microsoft’s Craig Mundie demonstrates natural user interface technologies (photo credit: Robert Sorbo/Microsoft)

Via: Silicon.com – Microsoft’s Gesture Recognition Office ‘Not That Far Away’

While gesture recognition, such as that seen in Project Natal can help gaming, Microsoft’s Craig Mundie showed yesterday at Microsoft’s Annual Financial Analyst Meeting in Redmond, Wash., how it will also transform the office.

In a demo, Microsoft’s top research and strategy officer showed how the desktop computer of the future will use an entire office as both display and input device, with voice and gestures augmenting a number of touchscreens.

His demo included hologram-like video conferencing, a virtual digital assistant, and multiple surface computers along with voice, touch and gesture recognition. The desk was a multitouch surface computer, and the office’s walls were also a display that could easily switch from being a virtual window and collection of digital photos to being a corkboard of sticky notes to various workspaces.

In an interview earlier this month, Bill Gates told CNET News that Microsoft plans to use Natal far beyond the Xbox, including with Windows.

Here are the excerpts:

Gates talked about a world in which depth-sensing cameras such as the one Microsoft is adding to the Xbox allow people to “control their PCs, game devices, and televisions.”

Speaking about all of the technology Microsoft has been cooking in its labs, Gates said that it was not just for games, “but for media consumption as a whole, and even if they connect it up to Windows PCs for interacting in terms of meetings, and collaboration, and communication.”

Using your body to control devices makes a lot of sense, Gates said. “I think the value is as great for if you’re in the home, as you want to manage your movies, music, home system type stuff, it’s very cool there,” he said. “And I think there’s incredible value as we use that in the office connected to a Windows PC. So Microsoft research and the product groups have a lot going on there, because you can use the cost reduction that will take place over the years to say, why shouldn’t that be in most office environments.”

Gates actually dropped the first hint of Natal during his joint appearance with Steve Jobs at the D: All Things Digital conference in 2007

“Imagine a game machine where you’re just going to pick up the bat and swing it, or the tennis racket and swing it,” Gates said.

Moderators Walt Mossberg and Kara Swisher mocked Gates, saying such a technology already exists and it’s called the Wii. But Gates disagreed. “No, that’s not it. You can’t pick up your tennis racket.”

He later added, “You can’t sit there with your friends and do those natural things,” he said. “That’s a 3D positional device. This is video recognition. This is a camera seeing what’s going on.”

About Project Natal

You can control an Xbox 360 video game through your own motion. Using a 3-D depth camera, it captures the movement of your whole body.

Project Natal is the code name for a “controller-free gaming and entertainment experience” by Microsoft for the Xbox 360 video game platform.

Based on an add-on peripheral for the Xbox 360 console, Project Natal enables users to control and interact with the Xbox 360 without the need to touch a game controller through a natural user interface using gestures, spoken commands, or presented objects and images.

The project is aimed at broadening the Xbox 360’s audience beyond its typically non-casual gamer base and was first announced on June 1, 2009 at E3 Expo 2009.

Microsoft said that over a thousand SDKs began shipping to game developers that same day.

Though Microsoft has not officially announced any price or release date projections for Project Natal, it is expected to be released in late 2010.

Project Natal Technology – The End of Game Controllers

Natal utilizes a motion sensor device that eliminates the need for a traditional controller.

Project Natal Sensor features an “RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software”, which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities.

Industry sources are saying that the technology behind Project Natal was based on ZCam, a time-of-flight camera product from 3DV Systems. This company was bought by Microsoft in March 2009, just before the announcement of Project Natal in E3. The software which runs the Zcam was in turn developed internally by Microsoft. The software enables gesture recognition, facial recognition, and voice recognition.

A prototype of 3DV Systems’ motion-sensitive video camera – the technology that is most likely behind Microsoft’s Project Natal.

The motion sensor device contains these components – a CMOS video camera, an infrared source, a multi-array microphone and a custom processor that utilizes proprietary software for this application.

The video camera and infra red source are used to detect motion and measure distance. The multi-array microphone allows the device to do voice recognition and acoustic localization.

There is nothing new in these technologies but the magic lies probably in the software that Microsoft developed for this application.

Written By: Ina Fried
Posted By: Situated Research