Design of a multimodal environment

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Paper I Evaluating environmental sounds for virtual reality applications, EURASIP Journal of Audio, Speech and Music Processing, Special issue on Environmental Sound Synthesis, Processing and Retrieval, 2010. This paper describes the design of a multimodal environment to provide users with IBR (Image Base Rendering) visual information and various levels of auditory information: ego-motion (footsteps), static sounds, dynamic sounds and music. It starts by describing the need for high quality audio in multimodal virtual environments and the notion of presence in such environments. A review of the state of the art research on presence and auditory feedback is presented. As described in Chapter 4, research on presence and auditory feedback has focused mostly on sound delivery methods and their role in creating sense of presence. Some work has also investigated the role of sound content in creating sense of presence [33, 37, 38]. However, to our knowledge nobody has previously investigated how interactive auditory feedback affects presence in virtual environments. The paper presents a large study whose goal is to investigate the role of different sound delivery methods in enhancing sense of motion in virtual environments, and investigates to what degree such environments create a sense of presence. The research was first motivated by previous investigations where subjects were asked to visit a virtual place where visual feedback was rendered by using IBR techniques, displayed by using an Head Mounted Displays. In such simulated environments, where scenes were extremely static (subjects could control only the point of view of the environment, but there was no actual action or moving objects in the environment) it was observed that users moved very little, and were not motivated to explore the environment. To overcome this limitation, other forms of feedback which could vary over time were investigated, with the purpose of creating a more immersive experience. We hypothesized that auditory feedback was an ideal time-varying feedback which could allow subject to engage and move in the environment. To investigate not only the presence or absence of auditory feedback, but also its quality, several soundscapes were designed. Such soundscapes, six in total, ranged from a piece of music and environmental sounds, both static and dynamic. The more complete solution, which was called Full in the paper, consisted of rendering soundscapes in 3D with moving sound sources, as well as rendering interactively footsteps sounds. Such sounds were obtained by creating some custom made sandals enhanced with sensors. Such sensors detected the act of users walking in the environment, and triggered a real-time footsteps synthesizer which simulated the sound of walking in the virtual surface depicted. In the six different conditions to which the subjects were exposed, the body motion was tracked. Results show that interactive footsteps sounds, together with the corresponding soundscape, significantly enhanced the motion of subjects in the environment. Rather surprisingly, the sense of presence, measured by using a presence questionnaire, was not significantly enhanced in the conditions with interactive auditory feedback and soundscape combined. This can be due to the difficulties of subjects to interpret the questions in the presence questionnaire. 6.2 Paper II R. Nordahl, S. Serafin and L. Turchet. Sound synthesis and evaluation of interactive footsteps for virtual reality applications. Proc. IEEE Virtual Reality, 2010. In this paper, a study whose goal is to evaluate ability of subjects to recognize synthetic footsteps sounds in both an interactive and an off-line context is presented. Subjects' perception in terms of quality of the developed system and realism of the simulation is also investigated through an psychophysical test. The ultimate goal is to develop a high quality footsteps synthesizer where users can wear their own footwear and which is efficient enough yet of high quality to be used in virtual reality applications. While walking, the net force F exerted by the foot against the ground can be represented by a time varying spectrum F(a; t), having components tangential and normal to the ground surface, where a denotes angular frequency and t is time. The term Ground Reaction Force (GRF) is often used to refer to the low frequency information in F, below about 300 Hz. The GRF is essentially responsible for the center of mass movement of the individual. It is approximately independent of footwear type, but varies between individuals or walking styles [51]. Higher-frequencies components of F(a; t) can be attributed to fast impacts between heel or toe and ground, sliding friction and contact variations between the shoe and the ground [52]. Unlike the GRF, these components can depend on footwear, on ground sur-face shape and material properties. They give rise to remote signatures in the form of airborne acoustic signals, seismic vibrations of the ground, and vibrations transmitted through the shoe to the foot, which have been studied in prior literature on acoustic [52,53] and vibrational [54]. signatures of human walking. These signals vary with the local material and spatial structure of the ground and with the temporal and spatial profile of interactions between the foot of the walker and the ground surface. The system proposed in this paper is based on microphones, which capture the sound of a person walking and from such sound extract temporal information concerning the step. More specifically, the GRF is extracted from the acoustic waveform. Our goal is to extract the users' contribution to the step, i.e., the GRF, and simulate the floor's contribution, i.e., the material on which the person is walking. The ultimate goal is to create the sensation of walking on a different surface than the one subjects are actually walking upon. Experiments on recognition show that subjects are better in recognizing sounds created with the interactive system developed compared when listening to prerecorded footsteps. A Chi-square analysis, however, shows that such recognition is significant only for two of the simulated surfaces, namely dry leaves and metal. This means that no real conclusions can be drawn on the better ability of subjects to recognize sounds in an interactive rather than off-line scenario. However, the developed system is promising since results show a high recognition rate. Moreover, informal interviewes performed after the recognition test showed that subjects appreciated the quality of the system and the fidelity of interaction. One advantage of the system used in this research, as compared to other systems used in previous research (such as the shoes enhanced with sensors) is the fact that it can be used by subjects wearing their own footwear. However, one disadvantage is the fact that the microphones capture any environmental sound, preventing such system to be used with other external sound sources rather than footsteps, or using it with soundscapes which would add a context to the virtual scenario, and are essential in experiments testing sense of presence. 6.3 Paper III R. Nordahl, S. Serafin and F. Fontana. Exploring sonic interaction design and presence: Natural Interactive Walking in Porto. Proceedings of Presence 2009. In this paper, two different interfaces to control synthesize footsteps are compared, their role in simulating the sensation of walking in a specific place, in this case the city of Porto is described. 6.3. PAPER III 39 The first interface, developed at the University of Verona, is a pair of shoes enhanced with sensors. Specifically, the shoes are a pair of sandals with two force sensitive resistors, one for each shoe. The shoes are used to control synthesized sounds of footsteps on different surfaces. The second interface consists of four microphones which are placed on top of a solid surface in a square configuration. The microphones detect footsteps of a person walking on the surface. An algorithm extracts in realtime the ground reaction force from real footsteps, and uses it to control the temporal variations of synthesized sounds of footsteps on different surfaces. The two developed interfaces are analyzed in terms of their ability to be used in a multimodal virtual environment, and advantages and disadvantages of the two devices are discussed. The ultimate goal is to design an interface which can be utilized in multimodal environments (complemented with haptic and visual feedback) where presence studies can be performed. The shoes enhanced with sensors have the advantage of being portable. Their main advantage, however, is the fact that they can be used also when other sounds are present in the environment, being those soundscapes synthetically generated or unwanted sounds existing in the environment where the installation is setup. However, the main disadvantage of this setup is the fact that users need to wear custom made shoes. Moreover, the setup is wearable, meaning that all the processing happens on a laptop placed on a backpack carried by the users. This fact has the advantage that users can walk in an unlimited space, but the disadvantage that their way of walking is not natural, likely preventing sense of presence. The main advantage of the configuration with microphones is the ability of subjects to wear their own footwear. This is an advantage from the point of view of naturaleness of the interface and the interaction. However, the system does not work when other sounds either than footsteps are present in the environment. We conclude that neither of the systems is optimal to perform presence studies in a multimodal context, and new solutions are been currently investigated. This paper was presented at the Presence 2009 conference which took place in Los Angeles in November 2009. The audience at the conference admired especially the combination of custom made de-vices with studies examining sense of presence. Traditionally, the presence community infact performs her studies using devices which are available in the market. Paper IV R. Nordahl et al. Sound design and perception in walking interactions. International Journal of Human-Computer Studies, 2009. This paper is a collaboration between several authors. Most of the authors are partners in the European project Natural Interactive Walking. The main goal of the paper is to outline all the different issues which are important to be researched when designing multimodal interfaces, in this case based on walking. It is beyond the possibility of a single researcher to be able to cover all those competencies in depth, this is why an interdisciplinary collaboration between different skills is desirable. The paper starts with motivating the importance of a multimodal approach when studying walking. From a sensory standpoint, in addition to vision, the pedestrian receives sound information via the auditory channel, vibrational information via the tactile (touch) sensory receptors in the skin of the feet, and information about ground shape and compliance via the proprioceptive sense (the body's ability to sense the configuration of its limbs in space). Proprioception, vision, and the vestibular (balance) sense are integrated to inform the pedestrian about his motion in space. As can be seen from the forgoing description, walking generates a great deal of multisensory information about the environment. Prior research has emphasized the influence of visual, haptic, vestibular, and proprioceptive information on planning and control of locomotion over flat surfaces (e.g., [55]). In two respects, these studies provide a limited account of the complexity of walking in real word environments. Firstly, they have not addressed the range of ground surfaces and materials met outside the lab (e.g., to our knowledge, none has investigated locomotion on gravel before). Secondly, they ignore the information contained in sounds generated by walking on real world surfaces (e.g., acoustic information about the gender of a walker [53]). These limitations are addressed in human perception studies presented in this dissertation. Notably, in VR contexts, when such layers of perceptual information are available, they are likely to contribute to a heightened sense of presence in a virtual environment. The paper starts by reviewing the literature on walking in virtual environments. It then outlines the different issues which need to be taken into account when designing and evaluating interfaces based on walking. First of all, the paper discusses several interfaces which have been built to track a person walking. In the context of the Natural Interactive Walking project, we focused both on haptic floors, by enhancing the interaction of the floor with haptic feedback, and on shoes enhanced with both sensors and actuators. The auditory feedback designed is also discussed, together with the physically based and physically informed algorithms designed to simulate walking interactions. What the author found particularly valuable when collaborating on this paper was to realize how such a simple everyday task like the act of walking becomes extremely complex when the goal is to faithfully reproduce it in a virtual environment. Although different solutions for haptic, auditory and also visual feedback are proposed, together with methods for evaluating such solutions, an overall evaluation of the advantages and disadvantages of each solution is not provided, and is still an open research issue. 6.5 Paper V R. Nordahl. Sonic Interaction Design to enhance presence and motion in virtual environments. Proc. CHI 2008. This paper was presented at the Computer Human Interaction (CHI) 2008 conference, in a special session dedicated to sonic interaction design. The paper describes a large study whose goal is to investigate the role of different forms of auditory feedback in enhancing motion of subjects in a virtual environment. The motivations behind this work are the results of the BENOGO project, an European project investigating the reproduction of physical places in the virtual world using image based rendering (IBR) techniques. In the BENOGO project, real places were photographed with a machine rotating around itself in 360 degrees. Such places were virtually reconstructed using IBR, in order to give to users the impression of navigating in the place, i.e, being there without going (hence the name BE-NO-GO). One of the peculiarities of this project is the fact that, in order for the IBR technique to succeed, no moving objects need to be present in the environment. This means that, from the visual point of view, the visited environment is extremely static. From the point of view of presence research, i.e., investigating if subjects felt present in the reproduced place, the project was rather unsuccessful, since subjects were feeling extremely bored in visiting such a static environment. The goal of this paper is therefore to provide a temporal dimension to the IBR based simulations provided by the BENOGO project. By using auditory feedback, which is notoriously evolving over time, we hypothesize that subjects will become more interested in visiting the virtual environment. We therefore tracked subjects while visiting the environment, in this case a botanical garden in the city of Prague, and we provided six different kinds of auditory feedback. The auditory feedback ranged from a static soundscape, to a piece of music to interactive shoes enhanced with sensors, which control synthesized footsteps. Results show that the auditory environment where interactive footsteps and 3D sound is rendered sig-nificantly enhance the motion of subjects in the virtual environment. 6.6 Paper VI R. Nordahl and D. Korsgaard. Distraction as a measure of presence: using visual and tactile adjustable distraction as a measure to de-termine immersive presence of content in mediated environments, Virtual Reality, Springer, November 2009. This paper presents several studies whose goal is to evaluate the role of distractors in determining presence in two kinds of mediated environments: movies and games. The paper is an elaborated journal version of the research described in papers VII and VIII. Specifically, we presented several studies where we propose the use of the adjustable distractor method as a way to evaluate presence in mediated environments. The method has been applied to videogames and movies, and implemented both using visual and tactile feedback. The main assumption behind the method is that presence is as strong as the minimum amount of stim-uli required to break it. Therefore the method proposed follows some ideas from the secondary task reaction time technique [20], trying to overcome its limitations. The main characteristic of an adjustable distractor is that it is a signal which can be perceived, and var-ies in strength. In the first described experiment, the method was used to evaluate presence in the context of a videogame. In this experiment, the adjustable distractor is visual, and in the form of a circle placed outside the player' s region, in a black frame surrounding the game's space. In the second experiment the distractor is tactile, in the form of a signal provided to users when watching a movie. The first experiment showed that the adjustable distractor method is not intrusive in the context of a game. This makes the method more interesting compared to similar techniques such as the secondary task reaction time. However, a player's experience of intensity while playing a game is not proportional to the minimum amount of visual distraction needed to attract the player's attention, so one of our original hypotheses had to be rejected. The second experiment, where tactile feedback is used, does not show conclusive results on how such feedback can be used to evaluate presence in movies. We believe this might due either to the fact that subjects perceived the tactile stimuli differently. 6.7 Paper VII R. Nordahl and D. Korsgaard. On the use of adjustable distraction as a measure to determine sustained attention during movie clips. Proceedings of Presence 2009. This paper presents research which is a continuation of the one described in Presence 2008. Precisely, we adapted the adjustable distractor method and we compared it to two other methods for measuring presence, specifically heart rates and subjective intensity rates. The field of research of this paper is examining presence and immersion in movies. In particular, it is the author's interest to investigate whether the adjustable reaction method can overcome the drawbacks which appear in other techniques to measure presence in movies, such as the use of secondary task reaction time (STRT). The adjustable distraction was in this case provided by using a vibration generated by a small speaker. The results of the experiments show similarities between results from questionnaires and heart rate measurements, but the adjustable distraction method reported different results. Great variance was measured in the recorded vibration of the different subjects. This problem might be overcome by having a pre-screening of the subjects, to ensure uniform perceived threshold. Another solution could be to average the results of each subject based on their previously screened perceived threshold. 6.8 Paper VIII R. Nordahl and D. Korsgaard. On the Use of Presence Measurements to Evaluate Computer Games. Proceedings of Presence 2008. In this paper it is described a novel methodology to measure user experience in computer games and interactive environments in general. The main contributions of this paper are a novel evaluation technique to measure experience in games. Until recently many or even most of the evaluation methods available to the presence community have been so-called pen-and-pencil-methods. They offer researchers the possibility to allow test subjects to record results, or rather their personal impressions, of the experiences. Drawbacks of such methods are stated in literature and in the paper: common issues are the issues of accuracy (how much can we trust what test subjects report since this is not an actual measurement), paper-and-pencil methods records the experiences retrospectively. As such it is not informa-tion from the actual experience that is recorded, but rather the memory of the experience from the individual test subject, that has had time (and is even asked to further process this experience by the mere act of filling out a questionnaire) Typical interactive experiences such as computer games are of their nature not truly linear ex-periences. Therefore each test-subject will have had a, at least slightly, different path through the computer game scenario. Test-subjects may be positively or negatively biased (of many different reasons). Issues such as feelings, moods, personal prejudices etc. may all affect the way the test-subject chooses to answer the questionnaire. In this paper, the goal was to develop a novel alternative that would allow measurements of experiences in computer games. The method is thought as a tool for developers of computer games that need to input during the development period of a product (concerning immersion, engagement and presence). The novel technique proposed was coined adjustable distraction. The paper sought to com-bine data recorded through a computer application with data from more traditional questionnaires. The technique proposed consists of using a distracter which becomes increasingly noticeable and asking subjects to press a button when the distracter is noticed (and the immersion in the computer game is thereby broken). Specifically, the screen was divided into two sections, one being a black frame containing a white dot. The dot's colors changes from black to gray and represents the distracters. A novel adaptive algorithm was also proposed which allows changing the size of dots according to users' responses. This is in order to measure several levels of presence, instead of simply measuring the ON/OFF possibility of the button being there or not. Finally a computer game was build to use as basis for running the experiments. Papers I, II, III, IV and V examine how presence can be adapted to the field of sonic interaction design. Specifically, novel techniques to adapt presence research to the design and evaluation of walking interfaces are proposed. The papers are organized in anti-cronological order, from the newest published to the oldest. In paper V the design of a photorealistic virtual environment that users can visit by using an head mounted display is described. The environment consists of a reproduction of the botanical garden in Prague. The visual feedback was enhanced by using different combinations of auditory feedback. Results show that interactive auditory feedback significantly enhances the motion of subjects in the environment. Paper IV describes different issues which need to be taken into account when designing walking interactions. The paper is of interdisciplinary nature, and co-authored by different members of the FET Open EU project Natural Interactive Walking. Expertises of the different authors covered in the paper range from the design and implementation of floors which provide haptic feedback, to the design of shoes enhanced with sensors, to sonic feedback to simulate sensation of walking on different surfaces and evaluation of the different systems, both in terms of perception and in terms of presence. The author contributed mostly to the evaluation part in the paper, which includes both basic psychophysical experiments, and which evaluate the affective content of footsteps sounds and presence and immersion studies. Paper III explores the use of the developed walking interfaces and sound synthesis algorithms in the context of the city of Porto. A workshop is described where interactive footsteps driven by the users are combined with the soundscape of the city of Porto, in such a way to be able to recreate in a laboratory the sensation of walking in the city. Paper II explores basic evaluation of the interactive footsteps synthesizer described in Paper III. The paper proposes and justifies the design of different alternatives to synthesize and control footsteps. Such design is tested in the paper. Finally, Paper I investigates in details sense of presence and self-motion in a walking interface, extending the investigations started in Paper V. All papers have in common the design, implementation of alternative interfaces to simulate walking in virtual reality, and the evaluation of such interfaces from a human centered perspective. The papers present an evolution in the development of the technology used. Paper V presents studies performed with shoes with only one pressure sensors. An improved design is presented in papers III and IV. Paper II describes basic evaluation techniques of the technology developed, such as psychophysical experiments which assess ability of subjects to recognize the simulated surfaces they are stepping upon. Finally, paper I reports the results of an in depth presence and immersion studies conducted with the technology developed. The second set of papers examines how presence studies can be adapted to the evaluation of computer games. The papers examine how distractors can be used to evaluate presence in computer games and movies. Paper VIII presents an experiment where a a visual distractor is adopted. A similar technique is also used to evaluate presence in the context of a movie clip, as described in Paper VII. The advantage of using a movie clip as opposed to a videogame is the fact that the linearity of the media allows a better control to the stimuli subjects are exposed to. Finally, paper VI presents a large study which combines studies presented in Papers VII and VIII, and extends them by also using a tactile distractor. Both sets of papers examine how presence methodology can be adapted to be used in evaluation of novel media technology. In both sets of papers, custom made interfaces are designed to support the investigation of presence. Specifically, in the first set of papers custom made shoes with sensors are designed, while in the second set of paper a vibrator to induce tactile distraction is designed.