Multisensory scientific visualization for multidimensional scalar fields data analysis in particular visual-Auditory case

Authors: Malikova, E., Pilyugin, V., Adzhiev, V., Pasko, G. and Pasko, A.

Journal: Scientific Visualization

Volume: 8

Issue: 4

Pages: 15-27

eISSN: 2079-3537

Abstract:

A well-known definition of visualization is the mapping of initial data to a visual representation, which can be perceived and interpreted by humans. Human senses include not only vision, but also hearing, sense of touch, smell and others including their combinations. We discuss in this article multisensory scientific visualization, in other words scientific visualization extended with sound, haptic and other sensory stimuli, related fields and concepts such as visualization, sonification and perceptualization, and geometric modeling using real functions. The formalization of the multisensory analysis process and particularly of establishing correspondences between the initial data and multiple sensory stimuli (mapping process) is an open research question. In this article, some generalizations based on using real-valued vector functions for solving data analysis problems by means of perceptualization in the area of multisensory mapping are proposed. These generalizations might be considered as a formalization of the correspondence between the initial data and different sensory stimuli. A formalization of different sensory stimuli analysis and interpretation of the results according to initial data is also an open research issue. In this article, some results were obtained for audio analysis interpretation of scalar fields, and approaches mainly based on musical theory and used by musicians for musical composition analysis were proposed for the case of visual-Auditory multidimensional scalar fields analysis. A special case study of a scalar field analysis using scientific visualization extended with such sensory stimuli like sound is presented. Both mapping to sound and further ways of sound analysis by researchers for this case are described in detail.

Source: Scopus