A number of startups are working on this tech, with an aim to convince VR gear manufacturers such as Oculus Rift and HTC Vive to incorporate the feature in a next generation device. They include SMI, Percept, Eyematic, Fove and Eyefluence, which recently allowed USA Today to demo its eye-tracking tech.

“Eye-tracking is almost guaranteed to be in second-generation VR headsets,” says Will Mason, cofounder of virtual reality media company UploadVR. “It’s an incredibly important piece of the VR puzzle.”

At present, making selections in VR or AR environments typically involve moving the head so that your gaze lands on a clickable icon, and then either pressing a handheld remote or, in the case of Microsoft’s HoloLens or Meta 2, reaching out with your hand to make a selection by interacting with a hologram.

As shown in Eyefluence’s demonstration, all of that is accomplished by simply casting your eyes on a given icon and then activating it with another glance.

“The idea here is that anything you do with your finger on a smartphone you can do with your eyes in VR or AR,” says Eyefluence CEO Jim Marggraff, who cofounded the Milpitas, Calif-based company in 2013 with another entrepreneur, David Stiehr.

“Computers made a big leap when they went from punchcards to a keyboard, and then another from a keyboard to a mouse,” says Marggraff, who invented the kid-focused LeapFrog LeapPad device. “We want to again change the way we interface with data.”

Eye Tech Not Due for Years

As exciting as this may sound, the mainstreaming of eye-tracking technology is still a ways off. Eyefluence execs say that although they are in discussions with a variety of headset makers, their tech isn’t likely to debut until 2017. Other companies remain largely in R&D mode, and Fove has a waitlist for its headset’s Kickstarter campaign.

The challenges for eye-tracking are both technological and financial. Creating hardware that consistently locks onto an infinite variety of eyeballs presents one hurdle, while doing so with gear that is light and consumes little power is another.

And while a number of companies in the space have managed to land funding – Eyefluence has raised $21.6 million in two rounds led by Intel Capital and Motorola Solutions – some tech-centric VCs are sitting on the sidelines while they wait for the technology to mature and for headset makers to make their moves.

“What eye-tracking will do will be powerful, but I’m not sure how valuable it will be from an investment standpoint,” says Kobie Fuller of Accel Partners. “Is there a multi-billion-dollar eye-tracking company out there? I don’t know.”

Among the unknowns: whether the tech will be disseminated through a licensed model or if existing headset companies will develop it on their own.

Still, once deployed eye-tracking has the potential to revolutionize the VR and AR experience, Fuller expects.

Specifically, eye-tracking will “greatly enhance interpersonal connections” in VR, he says, by applying realistic eye movements to avatars.

Facebook founder Mark Zuckerberg, who presciently bought Oculus for $2 billion, is banking on VR taking social interactions to a new level.

“The most exciting thing about eye-tracking is getting rid of that ‘uncanny valley’ (where disbelief sets in) when it comes to interacting through avatars,” says Fuller.

Less Computing Power

There are a few other ways in which successful eye-tracking tech could revolutionize AR and VR beyond just making such worlds easy to navigate without joysticks, remotes or hand gestures.

First, by tracking the eyes, such tech can telegraph to the VR device’s graphics processing unit, or GPU, that it needs to render only the images where the eyes are looking at that moment.

That means less computing power would be needed. Currently, a $700 Oculus headset requires a powerful computer to render its images. Oculus’s developer kit with a suitable computer costs $2,000. “If you can save on rendering power, that could significantly lower the barrier to entry into this market for consumers,” says UploadVR’s Mason.

And second, by not just tracking the eyeball but also potentially analyzing a person’s mood and logging in details about their gaze, AR/VR headsets are in a position to deliver targeted content as well as give third-party observers insights into the wearer’s state of mind and situational awareness.

Police Use

The former use case would appeal to in-VR advertisers, while the latter would come in handy for first responders.

“Police and paramedics are looking for an eyes-up, hands-free paradigm, and eye-tracking can bring that,” says Paul Steinberg, chief technology officer at Motorola Solutions, an investor in Eyefluence.

Steinberg sketches out a scene from what could be the near future.

A police officer on patrol has suddenly unholstered his gun. Via his augmented reality glasses with eye-tracking, colleagues at headquarters are instantly fed information about his stress level through pupil dilation information.

They can then both advise the officer through a radio as well as activate body cameras and other tech that he might have neglected to turn on in his stressed state. What’s more, another officer on the scene can instantly scan through a variety of command center video and data feeds through an AR headset, flipping through the options by simply looking at each one.

“We would have to work with our (first responder) customers to train them how to use this sort of tech of course, but the potential is there,” says Steinberg. “But we’re not months away, we’re more than that.”

Demo Shows Off Ease of Use

An Eyefluence indicates that eye-tracking technology isn’t a half-baked dream.

Navigating between a dozen tiles inside a first-generation Oculus headset proves as easy as shifting your gaze between them. Making selections – the equivalent of clicking on a mouse – is also equally intuitive. At no time does the head need to move, and hands remain at your side.

After about 10 minutes in the demo, it feels antiquated to pop on a VR headset and grab a remote to click through choices selected with head movements.

Marggraff says Eyefluence’s technical challenges included making technology that could respond in low and bright light, accounting for different size pupils and ensuring that power consumption is minimal.

But, he adds, his team remains convinced of the inevitability of its product: “Just like when we started tapping and swiping on our phones, we’re going to eventually need a better interface for AR and VR.”

Written by: Marco della Cava, USA Today (via Presence)
Posted by: Situated Research

The post Next Big Thing for Virtual Reality: Lasers in Your Eyes appeared first on Situated Research.

“It’s a privilege to be a part of this project as we explore how this exciting new technology might be incorporated into the everyday care of our patients,” said Dr.Christopher Kaeding, the physician who performed the surgery and director of sports medicine at Ohio State.  “To be honest, once we got into the surgery, I often forgot the device was there. It just seemed very intuitive and fit seamlessly.”

Google Glass has a frame similar to traditional glasses, but instead of lenses, there is a small glass block that sits above the right eye.  On that glass is a computer screen that, with a simple voice command, allows users to pull up information as they would on any other computer.  Attached to the front of the device is a camera that offers a point-of-view image and the ability to take both photos and videos while the device is worn.

During this procedure at the medical center’s University East facility, Kaeding wore the device as he performed ACL surgery on Paula Kobalka, 47, from Westerville, Ohio, who hurt her knee playing softball.  As he performed her operation at a facility on the east side of Columbus, Google Glass showed his vantage point via the internet to audiences miles away.

Across town, one of Kaeding’s Ohio State colleagues, Dr. Robert Magnussen, watched the surgery his office, while on the main campus, several students at The Ohio State University College of Medicine watched on their laptops.

“To have the opportunity to be a medical student and share in this technology is really exciting,” said Ryan Blackwell, a second-year medical student who watched the surgery remotely.   “This could have huge implications, not only from the medical education perspective, but because a doctor can use this technology remotely, it could spread patient care all over the world in places that we don’t have it already.”

“As an academic medical center, we’re very excited about the opportunities this device could provide for education,” said Dr. Clay Marsh, chief innovation officer at The Ohio State University Wexner Medical Center. “But beyond, that, it could be a game-changer for the doctor during the surgery itself.”

Experts have theorized that during surgery doctors could use voice commands to instantly call up x-ray or MRI images of their patient, pathology reports or reference materials.  They could collaborate live and face-to-face with colleagues via the internet, anywhere in the world.

“It puts you right there, real time,” said Marsh, who is also the executive director of the Center for Personalized Health Care at Ohio State. “Not only might you be able to call up any kind of information you need or to get the help you need, but it’s the ability to do it immediately that’s so exciting,” he said.  “Now, we just have to start using it. Like many technologies, it needs to be evaluated in different situations to find out where the greatest value is and how it can impact the lives of our patients in a positive way.”

Only 1,000 people in the United States have been chosen to test Google Glass as part of Google’s Explorer Program. Dr. Ismail Nabeel, an assistant professor of general internal medicine at Ohio State applied and was chosen. He then partnered with Kaeding to perform this groundbreaking surgery and to help test technology that could change the way we see medicine in the future.

Broadcast quality video and high-definition photos available for download: http://bit.ly/16jXc6c

Written by: The Ohio State University (via Presence); for details about the first international Google Glass surgery, in June 2013, see Clinica Cemtro; for a report about early reactions from those testing Glass see NPR

Image: Dr. Christopher Kaeding, an orthopedic surgeon at The Ohio State University Wexner Medical Center is shown wearing Google Glass

Posted by: Situated Research

The post Ohio State Doctor Shows Promise of Google Glass in Live Surgery appeared first on Situated Research.

SANTA CLARA, Calif. — Reality isn’t what is used to be. With increasingly powerful technologies, the human universe is being reimagined way beyond Google Glass’ photo-tapping and info cards floating in space above your eye. The future is fashionable eyewear, contact lenses or even bionic eyes with immersive 3D displays, conjuring up a digital layer to “augment” reality, enabling entire new classes of applications and user experiences. 

Like most technologies that eventually reach a mass market, augmented reality, or AR, has been gestating in university labs, as well as small companies focused on gaming and vertical applications, for nearly half a century. Emerging products likeGoogle Glass and Oculus Rift’s 3D virtual reality headset for immersive gaming are drawing attention to what could now be termed the “wearable revolution,” but they barely scratch the surface of what’s to come.

The wearable revolution can be traced back to Ivan Sutherland, a ground-breaking computer scientist at the University of Utah who in 1965 first described a head-mounted display with half-silvered mirrors that let the wearer see a virtual world superimposed on the real world. In 1968 he was able to demonstrate the concept, which was dubbed “The Sword of Damocles.”

His work was followed up and advanced decades later by researchers including the University of Toronto’s Steve Mann and Columbia University’s Steven Feiner. In the second decade of the 21st century, the technology is finally catching up with their concepts.

The necessary apparatus of cameras, computers, sensors and connectivity is coming down in cost and size and increasing in speed, accuracy and resolution to point that wearable computers will be viewed as a cool accessory, mediating our interactions with the analog and digital worlds.

Augmented Reality past and future

“You need to have technology that is sufficiently comfortable and usable, and a set of potential adopters who would be comfortable wearing the technology,” said Feiner at the gathering of the fledgling AR industry at the Augmented Reality Expo here Wednesday. “It would be like moving from big headphones to earbuds. When they are very small and comfortable, you don’t feel weird, but cool.” He added that glasses with a “sexy lump of bump” with electronics and display could also be cool to the early adopters, especially the younger generation that has grown up digital. However, he didn’t have any prediction for when wearable computer would reach a mass market.

In the last decade, AR has been primarily focused on immersive gaming that teleports users to another world and on vertical applications, such as tethered, interactive 3D training simulations.

But now augmented reality is about to break out into free space. “AR will be the interface for the Internet of things,” said Greg Kipper, author of “Augmented Reality: An Emerging Technologies Guide to AR.”

“It is a transition time, like from the command line to graphical user interface,” he said. “Imagine trying to do PhotoShop in a command-line interface. Augmented reality will bring to the world things beyond the graphical user interface. With sensors, computational power, storage and bandwidth, we’ll see the world in a new way and make it very personal.”

Will Wright, creator of the popular The Sims family games, likened AR to having super-sensory abilities, like flipping a switch to see what is underground, beneath your feet. “It’s not about bookmarks or restaurant reviews…it’s something that maps to my intuition.” He hoped that instead of augmenting reality, the technology could “decimate” reality, filtering out even more information than the brain already does to engage reality with less cacophony.

Steve Mann, who is rarely seen without one of his wearable computing rigs and is considered the father of AR, views the wearable revolution as a benefit to society. Quality of life can be improved with overlays of information, adding and subtracting it to facilitate improved “eyesight,” he said. “The first purpose is to help people see better,” he said during his keynote at the expo.

Just as the smartphone is compressing a lot of the function from antecedent computing devices into a single product, wearable computing will eventually make the handheld smartphone irrelevant.

“The value proposition of digital eyewear is having all devices in one, with a camera for each eye representing full body 3D, and the ability to interact with an infinite screen. We are architecting the future of interaction,” said Meron Gribetz of Meta, a Ycombinator startup working on a new operating system and hardware interface for augmented reality computing.

“There is no other future of computing other than this technology, which can display information from the real world and control objects with your fingers at low latency and high dexterity. It’s the keyboard and mouse of the future,” he claimed.

Atheer, a Mountain View, Calif.-based AR startup, is developing a platform that will work with existing mobile operating systems, such as Google’s Android. “We are the first mobile 3D platform delivering the human interface. We are taking the touch experience on smart devices, getting the Internet out of these monitors and putting it everywhere in physical world around you,” said CEO Sulieman Itani. “In 3D, you can paint in the physical world. For example, you could leave a note to a friend in the air at restaurant, and when the friend walks into the restaurant, only they can see it.”

The company plans to seed its technology to developers this year and have its technology embedded in stylish, lightweight glasses with cameras next year.

The transition to touch and gesture interfaces doesn’t mean that the old modes of human-computer interaction go away. Just as TV didn’t replace radio, augmented reality won’t obliterate previous interfaces. The keyboard might still be the best interface for writing a book. Nor is waving your hands in front of your face all day a good interface.

“Holding hands out in front of self as primary interface is the ‘gorilla arm’ effect,” said Noah Zerkin, who is developing a full-body inertial motion-capture system for head-mounted displays. “You get tired. We need to have alternative interfaces. If not thought-based, it needs to be subtle gestures that don’t require that you to wave hands around in front of your face.”

3Gear Systems is working on technology that allows 3D cameras mounted above a keyboard, like a lamp, to detect smaller gestures just above the keyboard, such as pinching to rotate an object on a screen, and can use input from all 10 fingers with millimeter-level accuracy.

Some companies are taking less radical approaches, focusing on inserting a layer of digital information into scenes via smartphones. Par Works, for example, image recognition technology makes it possible overlays digital imagery on real world data, such as photos and videos, with precision. A person looking for an apartment takes a picture of a building with a smartphone and the app overlays information on the image, or a shopper will see coupons or other information for various products on a shelf in a drug store.

Brands are adopting AR technology to increase performance of ads and sales. Several companies provide ways to turn a print ad into an interactive experience just by pointing the camera at the paper or an object with a marker. Blippar, for instance, recognizes images by pointing a phone camera at ads or object with its mark and inserts virtual layers of content.

The future of augmented reality

And where is all this heading over the next few years? It’s beginning to look like a real business, just as mobile did nearly a decade ago. Mobile analyst Tomi Ahonen expects AR to be adopted by a billion users by 2020. Intel is betting that AR will be big. The chip maker is investing $100 million over the next 2 to 3 years to fund companies developing “perceptual computing” software and apps, focusing on next-generation, natural user interfaces such as touch, gesture, voice, emotion sensing, biometrics, and image recognition.

Apple isn’t in the AR game yet, but the company has been awarded a U.S. patent, “Synchronized, interactive augmented reality displays for multifunction devices,” for overlaying video on live video feeds.

Eyewear will evolve over the next year with comfortable stylish glasses with powerful embedded technology. They will range from Google Glass-style glance-at displays that also replace the phone to stereoscopic 3D-viewing wearables for everyday use.

“You’ll get 20/20, perfectly augmented vision by 2020, with movie-quality special effects blended seamlessly into the world around you,” said Dave Lorenzini, founder of AugmentedRealityCompany.com and former director at Keyhole.com, now known as Google Earth. “The effects will look so real, you’ll have to lift your display to see what’s really there. There’s more of the world than meets the eye, and that’s what’s coming.”

He cautioned that the growth of the AR industry could be slowed by a lack of standards to connect disparate players and their formats for bringing a 3D digital layer to life. “The AR industry has to get together to power the hallucination of what’s do come,” Lorenzini said. He added that a key turning point will be the availability of the WYSIWYG (What You See Is What You Get) real-world markup tools needed to bring this digital layer to life.

When the AR industry does take off, Lorenzini envisions a trillion dollar market for animated content, services and special effects layered into the real world. “Imagine people tagging friends with visual effects like a 3D halo and wings, or paying for a face recognition service to scan and add a floating name tag over the head of everyone in a room,” he said. “AR will grow from specific vertical uses to mass market appeal, driven by young, early adopters.

“Anyone reviewing devices like Google Glass needs to take it to their kids’ school before they pass judgement,” Lorenzini added. “This is not a device from our time, it’s from theirs. They love it, use it effortlessly, and are totally unfazed by ad targeting or privacy concerns. It will be be a natural part of who they are, how they learn, connect and play.”

Eventually, wearable technology will become more integrated with the human body. With advances in miniaturization and nanotechnology glasswear will be replaced with contact lens or even bionic eyes that record everything, make phone calls and allow you to use parts of your body, or even your thoughts, to navigate the world.

“Contact lenses are difficult now but the bionic eye will become commonplace and AR will just be a feature,” Kipper said. “Some may choose to have eyes in back of their heads, and some won’t. Some will want to be cyborgs. We will always use tools as advanced as they can be to help ourselves.”

Brian Mullins, CEO of Daqri, an augmented reality developer of custom solutions, went even further in melding humans and technology. “Thinking is the future of AR,” he said. Mullins talked about measuring “thought intensity” with EEG machines and focusing the mind to manipulate objects during a panel discussion at the Augmented Reality Expo.

Of course, the technical challenges are accompanied by issues of social etiquette and privacy. Smartphones are now a well-accepted part of daily life in most countries, but issues around data ownership and access to the data abound. The subtlety and potentially always-on capacity of wearable technologies will create more privacy concerns and challenges to acceptance.

Feiner acknowledged that it’s “scary” in terms of the information available, especially when billions of people with cameras and microphones can capture anything in public. “There are no laws against it,” he noted.

He gave Google some compliments for not overloading Glass with features. “It not suffering from doing too much too soon,” he said. Whether Google Glass is the tip of the spear for the mass adoption of far more powerful AR is uncertain, but it is doing a good job of surfacing the issues around the introduction of a disruptive, new way of computing.

Nicola Liberati, a Ph.D. student in philosophy at the University of Pisa studying the intersection of humans and technology, suggested another line of thinking about AR in his presentation at the expo. “We should not focus our attention only on what we can do with the such technology, but even on what we become by using it.”

So, when you are strolling down the street wearing the latest digital eyewear from Google, Apple or some as yet unformed or now early-stage company, with your continuous partial attention on the 3D holographic screen feeding you all kinds of personalized information about the environment around you, zeroing in on the people and places in your field of view or piped in remotely from around the real and virtual worlds, and spaces in between, think about what we have become.

It all depends on your perspective.

Written by: by Dan Farber, CNET (via Presence)
Posted by: Situated Research

The post Beyond Google Glass: The Evolution of Augmented Reality appeared first on Situated Research.

Experience Media Studios’ 3DPOV^® system enables the capture of a three-dimensional visual and auditory experience from the first-person perspective. 3DPOV^® media delivers a higher level of sensory engagement than virtual reality that replicates a true-to-life binocular and peripheral visual field and a stereophonic auditory experience. The system also captures GPS coordinates and altitude information to further augment reality.

“Modern audiences demand more of their media experiences,” said Michael-Ryan Fletchall, CEO of Experience Media Studios. “With more control over how and when they consume media, audiences want new and individualized experiences offering deeper levels of engagement. 3DPOV delivers an experience that goes far beyond just watching.”

Immersive media quickly absorbs the viewer into the experience, providing implications for critical skills training, simulations, and experiential learning environments. Experience Media Studios formally launched 3DPOV^® in conjunction with the Military and Government Summit at the National Association of Broadcasters (NAB) Show. Today’s announcement underscores the value of 3DPOV^® in these key segments where small details not available in virtual reality are integrated to assess and teach armed forces critical life-saving, decision-under-pressure skills through rapid processing and reaction according to policies and protocols.

“In developing this media for military and government blended learning simulations, we immediately recognized the opportunity to apply the technology to our wheelhouse of entertainment and advertising,” said Fletchall.

Experience Media Studios is currently in pre-production with Possessed Soul, its upcoming feature length horror “experience” shot entirely using 3DPOV^®technology. The project is partially financed through pre-sales to the horror-genre fan community using the Indiegogo crowdfunding platform. In 2012, Experience Media Studios released the Josh Hutcherson drama, The Forger.

The 3DPOV^® system also features a cloud-based digital delivery platform, connecting affiliated media production companies with 3DPOV^® technology to build a high quality digital asset inventory for worldwide distribution to private and public end users via 2D and 3D televisions, personal computers, and mobile devices.

“Our goal was to build a complete front-to-backend solution for creating and directly distributing unique 3DPOV^® content,” said Fletchall. “We have an exclusive content platform for creating, cataloging, managing and distributing experience-driven 3DPOV^® assets through an industry-leading pipeline with a user-friendly interface.”

Experience Media Studios will roll out the consumer subscription service component of 3DPOV.com with limited content later in 2013.

Written by: PR Newswire (via Presence); more images available at the Experience Media Studies website
Posted by: Situated Research

The post New Media Capture and Delivery System Gives Users Immersive “Experiences” appeared first on Situated Research.

Such a display comes in the form of futuristic glasses that allow wearers to see virtual maps, drawings or other images — up to 3 feet (1 meter) in size — projected in front of their eyes. A chip smaller than half the size of a postage stamp can detect the wearer’s eye movements so that they just need to glance at an arrow key to turn a page in a virtual instruction manual or book.

“The data glasses allow us to see the real world in the normal way, while at the same time registering our eye movements with the camera,” said Rigo Herold, project manager at the Fraunhofer Center for Organics, Materials and Electronic Devices Dresden in Germany.

Such eye movement control frees up the hands of the glasses wearers entirely so that they can focus on their real-world work — whether they’re U.S. military mechanics trying to fix armored vehicles or hospital surgeons doing a marathon operation.

The hands-free technology for the augmented reality glasses may also represent the future direction for Google Glasses. That head-worn display made by the Internet search giant was made public as something that requires hand control, but TechRadar noticed that Google has since patented eye-tracking technology for its glasses.

“Despite the fact that Google’s data glasses, for instance, might be a little more stylish in appearance, navigating through the menu still requires using joysticks, whereas our glasses do not,” Herold said.

Researchers plan to exhibit the technology at the Electronica 2012 trade fair in Munich from Nov. 13-16. But buyers will have the choice of ordering a computer with the system or installing the device’s software on their own computer. The system can run on Windows or Linux.

The system’s hardware and software came from German researchers at the Fraunhofer Institute for Optronics, System Technologies and Image Exploitation IOSB, while the company TRIVISIO produced the actual eyewear.

Written by: TechNewsDaily (via Presence)
Posted by: Situated Research

The post Your Eyes Can Control Augmented Reality Glasses appeared first on Situated Research.

If you’ve ever seen a child interact with an iPad, you’ve seen the power of the touch interface in action. Is this a sign of what’s to come — will we be touching and swiping screens rather tapping buttons? I reached out to Josh Clark (@globalmoxie), founder of Global Moxie and author of “Tapworthy,” to get his thoughts on the future of touch and computer interaction, and whether or not buttons face extinction.

Clark says a touch-based UI is more intuitive to the way we think and act in the world. He also says touch is just the beginning — speech, facial expression, and physical gestures are on they way, and we need to start thinking about content in these contexts.

[Image: Screenshot from Apple’s trackpad tutorial.]

Clark will expand on these ideas at Mini TOC Austin on March 9 in Austin, Texas.

Our interview follows.

Are we close to seeing the end of buttons?

Josh Clark: I frequently say that buttons are a hack, and people sometimes take that the wrong way. I don’t mean it in a particularly negative way. I think buttons are an inspired hack, a workaround that we’ve needed just to get stuff done. That’s true in the real world as well as the virtual: A light switch over here to turn on a light over there isn’t especially intuitive, and it’s something that has to be learned, and re-learned for every room we walk into. That light switch introduces a middle man, a layer of separation between the action and the thing you really want to work on, which is the light. The switch is a hack, but a brilliant one because it’s just not practical to climb up a ladder in a dark room to screw in the light bulb.

Buttons in interfaces are a similar kind of hack — an abstraction we’ve needed to make the desktop interface work for 30 years. The cursor, the mouse, buttons, tabs, menus … these are all prosthetics we’ve been using to wrangle content and information.

With touchscreen interfaces, though, designers can create the illusion of acting on information and content directly, manipulating it like a physical object that you can touch and stretch and drag and nudge. Those interactions tickle our brains in different ways from how traditional interfaces work because we don’t have to process that middle layer of UI conventions. We can just touch the content directly in many cases. It’s a great way to help cut through complexity.

The result is so much more intuitive, so much more natural to the way we think and act in the world. The proof is how quickly people with no computing experience — people like toddlers and seniors — take so quickly to the iPad. They’re actually better with these interfaces than the rest of us because they aren’t poisoned by 30 years of desktop interface conventions. Follow the toddlers; they’re better at it than we are.

So, yes, in some contexts, buttons and other administrative debris of the traditional interface have run their course. But buttons remain useful in some contexts, especially for more abstract tasks that aren’t easily represented physically. The keyboard is a great example, as are actions like “send to Twitter,” which don’t have readily obvious physical components. And just as important, buttons are labeled with clear calls to action. As we turn the corner into popularizing touch interactions, buttons will still have a place.

What kinds of issues do touch- and gesture-oriented interfaces present?

Josh Clark: There are issues for both designers and users. In general, if a touchscreen element looks or behaves like a physical object, people will try to interact with it like one. If your interface looks like a book, people will try to turn its pages. For centuries, designers have dressed up their designs to look like physical objects, but that’s always just been eye candy in the past. With touch, users approach those designs very differently; they’re promises about how the interface works. So designers have to be careful to deliver on those promises. Don’t make your interface look like a book, for example, if it really works through desktop-like buttons. (I’m looking at you, Contacts app for iPad.)

So, you can create really intuitive interfaces by making them look or behave like physical objects. That doesn’t mean that everything has to look just like a real-world object. Windows Phone and the forthcoming Windows 8 interface, for example, use a very flat tile-like metaphor. It doesn’t look like a 3-D gadget or artifact, but it does behave with real-world physics. It’s easy to figure out how to slide and work the content on the screen. People figure that stuff out really quickly.

The next hurdle — and the big opportunity for touch interfaces — is moving to more abstract gestures: two- and three-finger swipes, a full-hand pinch, and so on. In those cases, gestures become the keyboard shortcuts of touch and begin to let you create applications that you play more than you use, almost like an instrument. But wait, here I am talking about abstract gestures; didn’t I just say that abstractions — like buttons — are less than ideal? Well, yeah, the trouble is you don’t want to have the overhead of processing an interface, of thinking through how it works. The thing about physical abstractions (like gestures) versus visual abstractions (like buttons) is that physical actions can be absorbed into muscle memory. That kind of subconscious knowledge is actually much faster than visual processing — it’s why touch typists are so much faster than people who visually peck at the keys. So, once you learn and absorb those physical actions — a two-finger swipe always does this or that — then you can actually move really quickly through an interface in the same way a pianist or a typist moves through a keyboard. Intent fluidly translated to action.

But how do you teach that stuff? Swiping a card, pinching a map, or tapping a photo are all based on actions we know from the physical world. But a two-finger swipe has no prior meaning. It’s not something we’ll guess. Gestures are invisible with no labels, so that means they have to be taught.

In what ways can UI design alleviate these learning issues?

Josh Clark: Designers should approach this by thinking through how we learn any physical action in the real world: observation of visual cues, demonstration, and practice. Too often, designers fall back on instruction manuals (iPad apps that open with a big screen of gesture diagrams) or screencasts. Neither are very effective.

Instead, designers have to do a better job of coaching people in context, showing our audiences how and when to use a gesture in the moment. More of us need to study video game design because games are great at this. In so many video games, you’re dropped into a world where you don’t even know what your goal is, let alone what you’re capable of or what obstacles you might encounter. The game rides along with you, tracking your progress, taking note of what you’ve encountered and what you haven’t, and giving in-context instruction, tips, and demonstrations as you go. That’s what more apps and websites should do. Don’t wait for people to somehow find a hidden gesture shortcut; tell people about it when they need it. Show an animation of the gesture and wait for them to copy it. Demonstration and practice — that’s how we learn all physical actions, from playing an instrument to learning a tennis serve.

How do you see computer interaction evolving?

Josh Clark: It’s a really exciting time for interaction design because so many new technologies are becoming mature and affordable. Touch got there a few years ago. Speech is just now arriving. Computer vision with face recognition and gesture recognition like Kinect are coming along. So, we have all these areas where computers are learning to understand our particularly human forms of communication.

In the past, we had to learn to act and think like the machine. At the command line, we had to write in the computer’s language, not our own. The desktop graphical user interface was a big step forward in making things more humane through visuals, but it was still oriented around how computers saw the world, not humans. When you consider the additions of touch, speech, facial expression, and physical gesture, you have nearly the whole range of human (and humane) communication tools. As computers learn the subtleties of those expressions, our interfaces can become more human and more intuitive, too.

Touchscreens are leading this charge for now, but touch isn’t appropriate in every context. Speech is obviously great for the car, for walking, for any context where you need your eyes elsewhere. We’re going to see interfaces that use these different modes of communication in context-appropriate combinations. But that means we have to start thinking hard about how our content works in all these different contexts. So many are struggling just to figure out how to make the content adapt to a smaller screen. How about how your content sounds when spoken? How about when it can be touched, or how it should respond to physical gestures or facial expressions? There’s lots of work ahead.

Are Google’s rumored heads-up-display glasses a sign of things to come?

Josh Clark: I’m sure that all kinds of new displays will have a role in the digital future. I’m not especially clever about figuring out which technology will be a huge hit. If someone had told me five years ago that the immediate future would be all about a glass phone with no buttons, I’d have said they were nuts. I think both software and context and, above all, human empathy make the difference in how and when a hardware technology becomes truly useful. The stuff I’ve seen of the heads-up-display glasses seems a bit awkward and unnatural. The twitchy way you have to move your head to navigate the screen seems to ask you to behave a little robot-like. I think trends and expectations are moving in an opposite direction — technology that adapts to human means of expression, not humans adapting to technology.

This interview was edited and condensed.

Written by: Jenn Webb, O’Reilly Radar (via Presence)
Posted by: Situated Research

The post Buttons Were An Inspired UI Hack, But Now We’ve Got Better Options appeared first on Situated Research.

Speaking in a video interview to promote next month’s b.tween 3D event in London, SCE Studios exec Mike Hocking explained that Sony’s recently announced HMD device could represent the future of 3D gaming by allowing users access to a full ‘virtual’ world.

“Another thing that’s coming back which I’m very excited is the notion of virtual reality,” he explained.

“So we have our new HMD – or head-mounted display – which was announced at CES earlier this year, and you can see that we can now get back to where we really wanted to get with virtual reality in the ’80s.

“We’ve now got the power to do it, we’ve got the screen resolution to do it, we’ve got the processing power to update fast enough so we can have very immersive experiences on head-mounted displays in gaming in the not too distant future.

“Being in a virtual world where I can see my virtual hands or a virtual gun with all the things we can do in the gaming world is going to be absolutely amazing,” he added.

There’s no release date set yet for Sony’s HMD or even a firm set of tech specs. Have a look at the clip from Sony’s CES show to see what the gizmo looks like.

Staring even deeper into his crystal ball, Hocking also explained that he’d love to see AR-enabled contact lenses that would super-impose information about anything in your line of sight, Terminator-style.

“Most of the really exciting stuff is out there in the R&D area at the moment – like having contact lenses with cameras and sensors built-in so that everything you see can be augmented with useful data,” he said.

Written by Fred Dutton, Eurogamer (via Presence)
Posted by: Situated Research

The post Sony Predicts Return of Virtual Reality appeared first on Situated Research.