AirPiano | How haptics enhances the music playing experience in VR
Dr. Orestis Georgiou, Director of Research, Ultrahaptics, writes about how researchers used Ultrahaptics’ mid-air haptic technology to create a virtual piano.
Listening to music is an auditory experience. But playing music is multi-modal. A piano player pressing the keys, or a guitarist plucking strings, are fusing visual, auditory and haptic inputs and outputs. It is nearly impossible to play an instrument without the sense of touch – although, with recent advances in prosthetics and robotics, there can be more than one way to achieve this.
In VR, immersive environments can easily be created to facilitate educators with an empty canvas on which lessons can be drawn up, tested, and evaluated. Students can virtually be transported to museums in other continents, communicate in virtual learning spaces with fellow students in Beijing or Sydney, or attend a lecture at a prestigious university thousands of miles away.
VR YOU CAN TOUCH
However, until now most of these experiences relied on audio and visual content delivery alone. That is, you could only see and hear in VR – not touch, smell or taste. The lack of haptics, in particular, has limited the ability of VR to become a skill-learning medium. You may be able to experience a world-class concert in VR, but actually learning to play a musical instrument virtually (and transfer those skills back to the real world) is another matter.
Rapid advances in haptic technologies and their adoption by the VR community is starting to change this, however. Dr. Jin Ryong Kim and his colleagues at Electronics and Telecommunications Research Institute (ETRI) in Korea, for instance, recently presented their latest development, AirPiano at the IEEE World Haptics Conference 2017 at Munich.
AIRPIANO | VIRTUAL PIANO
AirPiano is a virtual piano that incorporates tactile feedback. The feedback is created on the user’s bare hands, using Ultrahaptics’ ultrasonic mid-air haptic technology. The user’s fingers are tracked by a Leap Motion® camera module as they press the piano keys. Then Ultrahaptics’ technology is used to create concentrated pockets of air pressure targeted at the user’s fingertips. These mimic the experience of pressing on actual piano keys.
“We designed two haptic rendering schemes to investigate whether it is possible to recreate the resistance force of real piano keys effectively with the mid-air vibrotactile stimulus. The first scheme applied a constant feedback intensity and was designed to provide a short, concentrated pulse of intense feedback. The second scheme applied an adaptive feedback that was designed to follow the dynamic pressure changes associated with the pressing of real piano keys.” – Dr. Inwook Hwang.
Through user studies involving three simple musical pieces, AirPiano’s results showed that incorporating mid-air haptics into educational VR experiences can significantly improve the user experience. Specifically, the second scheme with adaptive feedback increased the overall clarity, reality, and enjoyment of the experience.
MUSICAL VR EXPERIENCE
This is the first musical VR experience that has used mid-air haptics in such an innovative way, opening up numerous new directions for research and development in this area. Through this research, the feasibility of finger interaction and tactile substitution was explored without the need of wearing additional cumbersome gloves or holding bulky instruments.
We are very excited about these new possibilities and look forward to seeing other virtual instruments that naturally integrate visual, auditory and tactile cues to improve VR experiences.