Project summary

The main objective of this Integrated Project (IP) IMMERSENCE is to enable highly realistic multi-modal interactive immersion into virtual and augmented reality environments.

Its focus will be on visual, haptic and auditive sensory components, while addressing an even broader range of human senses. While aiming at full multimodal feedback of all relevant information, the project will focus on hand-based (i.e. manual) tasks when dealing with interaction, in order to keep the related problems tractable. For the same reasons we will not cover all facets of information exchange which can emerge in an interactive situation, especially semantics related issues such as verbal communication will be ignored, while explicitly addressing emotional aspects. Haptic enhancement of the multimodal environment will be a special focus of the project, in order to compensate for the relatively limited efforts spent on this area up to now, as compared to visual or auditory components.

Three driving scenarios will cover the full spectrum of problems arising when implementing interactive environments: manipulation or exploration of an object by a human; direct contact between persons e.g. during a handshake or social dancing; and object mediated collaboration between humans, like jointly carrying a heavy object. Multimodal interaction is the key issue to all of them.

Three major areas have to be investigated in detail in order to create a solid scientific and technological basis for any future development on the field:

• The indispensable scientific foundation will be laid by basic research into the psychophysical and neurological mechanisms underlying multimodal interaction. It necessarily has to cover the entire spectrum of human sensory modalities, while haptics will play a distinguished role as related neuroscientific understanding is largely missing.
• A novel, data-driven algorithmic paradigm will be investigated for the generation of sensory feedback. During a first recording stage all relevant data characterizing interaction are collected by multimodal sensing suites, to be developed during the project. The recorded data will be processed in order to convert the raw signals into abstract descriptors. This abstraction will not only enable to faithfully replay the interactive session using the collected data while replacing the real counterpart by a virtual one, but will provide the fundament for prediction schemes to create sensory feedback for interaction which never has been observed before.
• A major research effort is needed to characterize and measure immersion and presence, in order to provide design guidelines for the technological development and methods for the systematic evaluation of different approaches. The project will base its development on the combination of two recently developed tools, providing subjective (based primarily on questionnaires, on-line ratings, interviews, or post rating of video-taped scenes) and objective (like psycho-physiological or eventually even fMRI or EEG-based) measurements

Numerous immediate benefits of this integrated project are foreseen in basic science, technology, and applications. We will gain new insight into the perceptional and neural mechanisms involved in multi-modal interaction, resulting in a deeper understanding of presence. Numerous haptic interfaces and demonstrative multimodal integrated systems will be developed including novel sensors and actuators. New methods will be created to acquire geometric components of multi-modal, haptic-enhanced interaction in 3D together with the visual appearance of the involved components. Issues in medical training and diagnosis, tele-cooperation in industrial or architectural construction and entertainment, and collaborative task solving in augmented reality environments will be addressed.