4D glasses—New Technology in Mixed Reality

“A touching image”: far from being a metaphor, this expression will soon become a reality. The new technology will create the impression of being touched by an object emerging from a screen. The purpose of this invention is to reduce and optimize the synchronization of 7D cinema components that increase the feeling of immersion within the universe of a movie: VR glasses, environment simulators and interactive devices.

The development of this technology is based on a neuroscience study conducted by a team of researchers from San Diego’s University of California and San Diego State University. Research in neural computation, brain imaging and experimental psychology helped to combine optical and tactile illusions in order to simulate the sensation of a virtual object approaching and touching the viewer’s face.

Experimental Stage—The Study of Multi-Sensory Cues

Cette étape permettra non seulement de comprendre certains aspects des mécanismes sensoriels et neuronaux de la perception multisensorielle, mais aussi d’initier une base scientifique pour créer la prochaine génération de médias et de réalité virtuelle et augmentée (la réalité mixte). Les lunettes 4D créées par l’équipe dans le cadre de l’expérimentation peuvent être connectées à plusieurs contenus divertissants comme les jeux vidéo, la musique, les films et la réalité virtuelle afin de vivre une expérience immersive multisensorielle et d’accentuer l’effet de présence au niveau du visage.

A study combining the analysis of subjective interpretations and maps of the brain areas receiving tactile and visual signals from an object was conducted to understand the perception system of a moving object. Indeed, according to the main author of this study, Ruey-Song Huang, perceiving an object coming in our direction generates visual, tactile and audio signals. All these signals are not perceived at the same time and have different origins and duration. The design of mixed reality technology must take into consideration the spatiotemporal reception of these signals by the viewer.

Psychophysical Experience

Les chercheurs ont effectué des observations d’imageries fonctionnelles et des tests de perceptions subjectives d’une balle en mouvement sur un écran 2D, représentée par des logiciels de modélisation 3D et de l’air soufflé dans la même direction. Les stimuli tactile et visuel ont été déclenchés avec différents décalages temporels et suivant les directions frontale et oblique. Onze femmes et neuf hommes âgés de 19 à 23 ans ont participé au test psychophysique. La neuro-imagerie a permis d’observer l’activité cérébrale de onze participants du même groupe percevant les mêmes stimuli du test subjectif.

Participants evaluated the synchrony between the ball and the pulsed air. When they were triggered at almost the same time (with a delay of 100 milliseconds), the air was perceived as completely out of sync with the object. With delays of 800 to 1000 milliseconds, both stimuli were perceived as being in sync. It felt like the object passed close to the face, generating a little air. This first test showed that the best time to trigger the tactile stimulus is when the ball approaches the face.

Functional Magnetic Resonance Imaging (fMRI)

The active areas of the brain indicated by fMRI are the cerebral hemispheres that govern the reception of sensory information. The analyzes supported the psychophysical experiment findings and confirmed the importance of spatiotemporal alignment and accuracy in time coordinates of multi-sensory stimuli.

This study entitled “Spatiotemporal integration of looming visual and tactile stimuli near the face,” co-authored by Ruey-Song Huang, Ching-fu Chen and Martin Serenowas, was published on February 6, 2018, in Human Brain Mapping.