Spatial Mapping for Microsoft’s HoloLens

ABSTRACT

Mixed reality (MR) is a growing technology that is expected to provide major advancements to the future of how people and businesses interact. Mixed reality is a division of virtual reality (VR) and augmented reality (AR) that combines the real-world with the virtual world. The main objectives of this study was to explore how mixed reality (MR) accomplishes this to create a believable experience for users. Microsoft’s HoloLens is a leading example of how this is achieved. Our results show how Microsoft’s HoloLens uses spatial mapping to allow developers to create a three-dimensional map of the surrounding environment that allows the HoloLens and the user to interact with the real world.

Keywords: Mixed Reality (MR), Virtual reality (VR), Augmented reality (AR), HoloLens, Spatial mapping.

INTRODUCTION

Mixed reality is the result of blending the physical world with the digital world. Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems. The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays.” Their paper introduced the concept of the virtuality continuum and focused on how the categorization of taxonomy applied to displays. Since then, the application of mixed reality goes beyond displays but also includes environmental input, spatial sound, and location. With our project we aim to find the most efficient user interface to interact with this technology of the future.

Get Help With Your Essay

If you need assistance with writing your essay, our professional essay writing service is here to help!
Find out more about our Essay Writing Service

1.1 Literature Reviews

An extensive search on spatial mapping, Microsoft’s HoloLens, or mixed reality shows that there are a vast number of articles dedicated to similar applications. Many of these articles research the use of spatial mapping in the medical industry or related fields [2, 3, 4, 5]. These articles demonstrate the true potential that mixed reality and spatial mapping has in various uses.

Although these articles are vast, there are very few that are dedicated to spatial mapping and the application into Microsoft’s HoloLens mixed reality headset. Bajura, Fuchs, and Ohbuchi, in a paper titled “Merging virtual objects with the real world: Seeing ultrasound imagery within the patient”  stated that mixed reality is an environment envisioned to provide an ideal virtual space with reality essential for communication. [6] This is brought together by spatial mapping. Spatial mapping provides “applications with the opportunity to present natural and familiar forms of interaction to the user”. [9] The use of special mapping in mixed reality has led authors to a very comprehensive and useable research. Laying out detailed objectives, methods, and conclusions.

2. METHODS

2.1 Apparatus

Our research interface consisted of a few software systems to accomplish a mixed reality environment using spatial mapping. The specification for each component is listed below.

Microsoft HoloLens (development edition)

A mixed reality head-mounted display that allows the user to run and build Windows Mixed Reality platform using Windows 10.

Microsoft Visual Studios SDK

A software development kit (SDK) from Microsoft that contains documentation, libraries and other tools to develop in the .NET framework. Allowing our team to develop using the C# language.

Unity3D Editor

A cross-platform game engine providing tools for our team to develop a three-dimensional mixed reality environment for Microsoft’s HoloLens.

Computer 1

• MSI Intel Core i5-8300H CPU
• RAM 8.00GB
• Intel UHD Graphics 630

Computer 2

• VAIO Intel Core i3-2370M CPU
• RAM 4.00GB
• Intel UHD Graphics 630

2.2 Procedure

Our team gathered various research papers on the topics of mixed reality, Microsoft HoloLens, and spatial mapping. As we gathered information, our team eliminated unnecessary information and compiled only pertinent information to correlate with our intended purpose. Our intended purpose was to describe and illustrate the importance of spatial mapping in a mixed reality environment. Specifically, Microsoft’s HoloLens.

2.3 Participants

No participants were used in our study and research. As our study was better suited without participants. However, the youngest age for use as recommended by Microsoft is 13 years old and the oldest age recommended for use without potentially needing special adjustment is 50 years old. Thus the ideal participants would be men and women between the ages of 13 to 50. The statement of participants is to reflect the health and safety in the use of Microsoft’s HoloLens.

2.4 Build

After our team gathered data, our focus shifted to an actual build to demonstrate spatial mapping on Microsoft’s HoloLens. Using the described software above, our team  developed our own mixed reality environment to engage a user and illustrate the importance of spatial mapping in the development of our user interface. Our HoloLens prototype allows a user to engage in the surrounded environment and uses spatial mapping to detect and place the user interface in the best possible location for interaction creating a beautiful mixed reality.

3. RESULTS

Before beginning our research and collected information, it was vital to begin with a simple question:

“What allows Microsoft’s HoloLens to achieve a mixed reality environment?”

After going through an extensive amount of research, the answer was obvious, spatial mapping. The use of spatial mapping allows the user to experience a true mixed reality experience. Mixed reality is the union of three primary factors: the human, the computer, and the environment. These factors create various connections amongst each other to create mixed reality.

Figure 1. Diagram of mixed reality­

Spatial mapping allows the user to engage and place objects into real world surfaces and environments. This is considerably important for the user interface and experience since a mixed reality environment harnesses a user’s visual surrounding and is able to generate a virtual world mapped onto it. The process of spatial mapping on the HoloLens begins by initially scanning the environment to learn about the surfaces and the environment within it.

Our research indicates that there are four user scenarios that are most important when using spatial mapping:

• Placement
• Occlusion
• Physics
• Visualisation

These four elements are crucial to a great user experience in a mixed reality world. Placement is vital to creating a great user experience. Spatial mapping does this by taking into account not just direction of an object but more importantly the distance. These details guide the hologram to interact with its surroundings. At times a hologram may need to be hidden behind the physical world. This is called Occlusion, the ability for the hologram to collide, hide, or use a surface to create additional information to the user. The way HoloLens moves and adjust to look like real-life objects is through its physics. This is typically done through the Unity 3D Editor by adding such actions as movement and collision to create natural objects we might see in the real-world.. Finally, visualization creates the backdrop for a truly appealing mixed reality environment. This is accomplished by creating things such as shadows and light sources. This truly gives the user a more realistic visualization of Microsoft’s HoloLens.

Although there are various common usage scenarios that deal with a mixed reality environment. They tend to me more vital to the user experience. But what about the key components to make spatial mapping work. Research suggests that there are four elements that are vital to the development of spatial mapping in Microsoft’s HoloLens.

• The surface observer
• Mesh caching and processing
• Rendering
• Ray casting and collision

The surface observer is the first point of contact with spatial mapping. The surface observer creates spatial volumes and data. It creates a specific coordinate system and changes the data points as new information arises.

Next, mesh caching and processing is particularly important in spatial mapping. When the surface observer scans a room, it creates triangular meshes that structures the surfaces in the room. These are stored by the mesh cache. Then to help performance and computational tasks, these meshes are sent to the mesh processor to process the data and account for changes and updates to incoming data from the surface observer and the mesh cache.

Spatial mapping meshes are then needed to be rendered. The rendering of these meshes combined with mesh processing creates a smooth operation so that users can engage with the environment with no lagging.

Finally, the physics and visualization of the HoloLens comes together by ray-casting and collision. Which takes the meshes from the mesh cache and sends them to the physics generator to create visual shadows, lights, and collisions to complete the spatial mapping.

As we finished and compiled our research, it was clear that spatial mapping takes into consideration a great deal of factors. From common user scenarios to development features, all these factors together create an immersive mixed reality experience in Microsoft’s HoloLens.

4. CONCLUSION & DISCUSSION

4.1 Conclusion

In this paper,  the work that has went into researching and implementing a user interface design when it pertains to mixed reality systems, namely for this project the Microsoft HoloLens. We can conclude that mixed reality user interfaces must be user friendly by taking into account the real world around a user. We also have to take into account the differences in gestures that different users might find intuitive in interacting with the interface. Finally, we have to understand the fact that this is a new technology and we have to help users adapt to a completely new interface type then the ones they have been accustomed to.

The result of this research is seen as the interface created for the Microsoft HoloLens that that the team created. It was developed taking into account what was learned via research. Considering the fact that it was the first time for all team members developing for augmented reality and HoloLens, the implementation might seem slightly rudimentary, however it still tries to implement as many user interface design principles into the first prototype as possible.

4.2 Future Work

While there is no current plans to continue working on this project in the near future, there is definitely room to further expand and work on this project. The research and work accomplished can be used to further improve user interfaces of augmented reality devices especially, but not limited to, the Microsoft HoloLens. Further work can be done such as gathering more user input to further refine the created interface and further refining the user interface and the spatial mapping.

5. REFERENCES

[1]   Milgram, P., & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information Systems. 77(12), 1-15.

[2]   Lis-Marciniak, A., Tomiakowski, J., & Kapusta, P. (2018). Design Rules, Implementation and Testing of User Interfaces for Mixed Reality Applications. 2018 11th International Conference on Human System Interaction (HSI).

[3]   W. Si, X. Liao, Q. Wang and P. Heng, (2018). Augmented Reality-Based Personalized Virtual Operative Anatomy for Neurosurgical Guidance and Training. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
[4]   Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, and Pattie Maes. (2016) Procedurally generated virtual reality from 3D reconstructed physical space.  Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (VRST ’16), 191-200.

[5]   Tamura, H., Yamamoto, H., & Katayama, A. (2001, November/December). Mixed reality: Future dreams seen at the border between real and virtual worlds. IEEE Computer Graphics and Applications. 21(6), 64-70.

[6] Bajura, M., Fuchs, H., Ohbuchi, R. (1992). Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. Proceedings of the 19th annual conference on Computer graphics and interactive techniques.

Computer Graphics,  26(2). 203-210.

[7] Christian Zimmer, Michael Bertram, Fabian Büntig, Daniel Drochtert, and Christian Geiger. (2017). Mobile augmented reality illustrations that entertain and inform: design and implementation issues with the hololens. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. 23, 7 pages.

[8] Mohamed Handosa,Hendrik Schulze, Dennis Gracanin, Mattew Tucker, Mark Manuel. (2018). An Approach to Embodiment and Interations with Digital Entities in Mixed-Reality Environments. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2.

[9] Microsoft Documents Website. Available: http://www.docs.microsoft.com/en-us/windows/mixed-reality