Tuesday, May 10, 2011

Kinect + Bing Maps–the gestures

by vbandi via VBandi's blog on 5/10/2011 9:18:37 PM

Note: If you don’t want to read this much, just jump to the end of the article, where you can find a video of the application with all the gestures.

As I said in my introductory post, with the Kinect Maps solution we wanted to discover a new set of gestures to control the map. We had pretty ambitious goals that – as far as we know – no other solution has solved yet:

• Focus on an exhibition setting instead of a productivity scenario – so, a gesture set that makes people tired is not a problem, as long as they have fun. After all, they have to give their place to the next person in line…
• Controls must be easy to learn – either by observing the previous user, with the help of the on-screen instructions or the hostess
• With practice, the controls must be precise enough to find any city, street or (where available) 3D building, and examine it in as much detail as the map allows
• It must be engaging and entertaining to watch somebody else using the application

We decided to have two modes for the application. One is the “Map mode” where the entire Earth can be seen, panned and zoomed in. The other one is what we called “Fly mode”, where users can fly around 3D cities or landscapes. The mantra for this mode was “Superman”, and we knew we were on the right track when our users actually started to mention superheroes describing the experience they had.

We have also added a menu for some of the most interesting places in Bing Maps, such as New York, the Grand Canyon, Mount Everest, Las Vegas, etc. The exhibition’s visitors used these “bookmarks” a lot.

Map Mode

In map mode, we are using the right hand to pan the map, while the right leg can be used to zoom in and out. The further you step with the right leg, the faster the zooming happens. This combination proved to be very effective: it is possible to precisely zoom in to street level with one continuous zoom, while the extended right hand keeps the target in the center of the screen. You can see this in the video below starting at 0:48, when I zoom in to Manhattan from space.

Flight Mode

Once you get to the altitude of 100 meters, the application switches to fly mode. In fly mode, there are other gestures that you can use:

• Stepping forward or backward with the right leg moves forward or backwards.
• Extending your arms horizontally keeps the flight at level.
• Raising your arms slightly increases the altitude, while lowering your arms decreases it. The more you raise or lower your arms, the faster the altitude changes. However, if you go beyond 45 degrees, the vertical movement stops, so you can rest your hands when needed.
• If you go 10,000 meters above the ground, the application switches back to Map Mode.
• Bending your torso forward makes the camera look down. Again, the angle of the camera is proportional to the angle of your bending, but the effect is multiplied so that you can still see the screen while looking straight down at skyscrapers. Similarly, bending your torso backward (think Limbo), makes the camera look up
• Finally, bending your torso sideways makes the camera turn. You don’t have to keep your hands out for this, but most people do, remembering when they played airplanes as kids. Or, in case of children, they are playing airplanes now…

Favorite places

In both Map Mode and Flight Mode, you can reach high up with your right hand, and invoke the favorites panel. Here, you can select one of the 8 favorites. This works similarly to the Xbox menu with Kinect – move a hand cursor over the picture of the place you want to fly to, and keep it there for a few seconds. This panel is created with the Kinect WPF Toolkit.

Here is a video of me showing off all the gestures. If you have read this far, you may want to actually see it:

While there is always room for refinement, we are very happy with the results. We have successfully found a control method that is both precise and fun to use. We showed the visitors (and ourselves) that Kinect can be used beyond games, and most of them did leave with a smile on their face – feeling like Superman (or Supergirl).

Stay tuned for more posts on the details of designing the experience, lessons learned, and so on. Please let me know what you would like to learn about, and I will try to answer those questions in the next post!