Max-Planck-Institut Tübingen: Critical role of vision in adaptation to asynchronous multi-modal information

When combining real and virtual cues, imperfect synchronization will greatly diminish a user’s feeling of presence. Since perfect synchronization is not achievable, the MPS planned to act on the human perceptual system in order to accustom it to these limitations. Given the correct conditions, perceived synchrony can be an adaptive process. Until now, the flexibility of the perceptual system has been studied only for audio-video interfaces, and the knowledge applied to television broadcasting and movie theatres. Here the researchers investigated adaptation to multimodal information, including haptics. In an experiment MPS investigated the mechanism underlying temporal adaptation. The figure below (left) shows the experimental paradigm used: after repeated presentation of asynchronous audio-visual signals, the PSS (Point of Perceived Simultaneity) was determined in three modality pairs (audio-visual, audio-tactile, and visuo-tactile). The right side of the figure summarizes the results obtained: positive values of PSS difference indicate that the adaptation procedure produced a perceptual outcome. Adapting to audiovisual asynchrony produced not only an effect on audiovisual timing, but also on visual tactile timing. This finding suggests a critical role of the visual modality in the adaptability to asynchronous multimodal information. It appears that the storage of visual information can have a flexible temporal storage, depending on the adaptation condition.     (left): Experimental paradigm. (right): Shift in perceived synchrony in three modality pairs after repeated presentation of asynchronous audiovisual signals. MPS further investigated this critical factor by running a pilot experiment that recorded reaction time to the detection of signals after adaptation to the same repeated asynchronous signals. Preliminary results indicate a different reaction time to the visual signals, but equal for auditory and tactile signals. If this result will be corroborated, it will confirm the flexibility of the visual modality in the recalibration of asynchrony and extend the results above by proving that the effect is based on low-level sensory processes. In a second series of studies, MPS investigated the native tolerance of the perceptual system and its intrinsic strategies to cope with asynchronous information by exploring a new type of temporal adaptation that has to do only with tactile information at two distant body locations. The researchers addressed the resulting tolerance to the different runtime that causes asynchrony in the arrival of information to the brain. The perceptual system is expected to have to compensate for this delay to preserve the sensation of a synchronous intra-modal event. In one study, they presented asynchronous signals at distant body locations in order to lower the sensitivity to this asynchrony. Results of such manipulation indicate that adaptation in this modality does not have lasting perceptual consequences. This is most likely because the tactile modality does not have to deal with asynchronies during everyday activities, therefore no dedicated mechanism for temporal recalibration is in place.

Related deliverables:

Related papers:

>>back to overview