DeveloLens 3 - HoloToolkit Spatial Mapping

<< DeveloLens 2 - HoloToolkit-Unity Init

HoloToolkit-Unity Spatial Mapping

So you've created a UWP app or two for your HoloLens, but you have an idea for an app that utilizes whatever room the user is in. There are a few tutorials in the Microsoft Academy that cover the concepts, and a few that show you how to use a version of the UnityToolkit that no longer exists. Unless you can reverse engineer the code they provide, and find new versions of classes they tell you to use, it can set you back in understanding how easy it is to use.

Here we will cover the latest version of the toolkit, and try to explain how it's used well enough so that it will remain relevant even as development continues. This will be about how to use the code they provide when you require specific functionality, and specifically spatial mapping for this post.

welcome, to the real world...

Egg Roll

Let's say we want to see what it would look like to have an egg roll around our floor. After creating a new scene, and saving it wherever you prefer, go ahead and add the toolkit. If you are not familiar with adding HoloToolkit to your Unity project, see my previous post on doing so. Since we'll be creating our own scene, we will not need the Examples or Tests folders.

If you're wanting to run this locally in Unity before deploying, you will need your room model. You can record your own from the device portal, or you can use the Test model currently provided in HoloToolkit-Tests > SpatialMapping > Prefabs which is usually nicer than the rooms I find myself developing in.

Remember that starting with a fresh project and scene, you must always run the HoloToolkit > Configure menu settings to make sure your scene is setup and the project is configured correctly for your app.

Delete the default the components, and then add the following components which are currently located in the following locations.

• HoloLensCamera: HoloToolkit > Input > Prefabs > HoloLensCamera.prefab
• SpatialMapping: HoloToolkit > SpatialMapping > Prefabs > SpatialMapping.prefab

Now we need a placeholder which I'll call Managers to add our, well, managers. Specifically the following managers.

• WorldAnchorManager: HoloToolkit > Utilities > Scripts > WorldAnchorManager.cs
• Wrapper around world anchor store to streamline some of the persistence api busy work.
• GesturesInput: HoloToolkit > Input > Scripts > InputSources > GesturesInput.cs
• Input source for gestures information from the WSA APIs, which gives access to various system-supported gestures. This is a wrapper on top of GestureRecognizer.
• InputManager: HoloToolkit > Input > Scripts > InputManager.cs
• Input Manager is responsible for managing input sources and dispatching relevant event to the appropriate input handlers. 
• RawInteractionSourcesInput: HoloToolkit > Input > Scripts > InputSources > RawInteractionSourcesInput.cs
• This input source only triggers SourceUp/SourceDown and SourceDetected/SourceLost. Everything else is handled by GesturesInput.
• Just remember this script is necessary to fill a gap for InputManager.
• GazeManager: HoloToolkit > Input > Scripts > Gaze > GazeManager.cs
• The gaze manager manages everything related to a gaze ray that can interact with other objects

These scripts will give us the functionality we need to use Spatial Mapping, and we can add just a little bit of code to roll our egg into it. Let's add our own script to our Managers called EggManager.

In a nutshell, this script enables the GestureRecognizer input functionality, which we then assign an event method to. This event is just going to place our egg at a predefined distance away from the user.

The important part of this is the public prefab that we call fill with any prefab we want, and the InputClicked event we will use to drop that prefab on the user click. This will give a project that looks like so:

I created an almost egg prefab by creating a sphere and altering it slightly, but you can drop anything like it's hot as long as it contains a rigidbody with the 'Use Gravity' enabled.

You can now build your Visual Studio solution from the HoloToolkit menu to whatever folder you'd like. Then open it up in Visual Studio for deployment to the emulator or the device.

Lets watch these eggs roll.

DeveloLens 2 - HoloToolkit-Unity Init

<< DeveloLens 1 - Introduction

Developing with HoloToolkit-Unity

You built a neat 3D model in Unity and deployed it to your HoloLens, but it just floats somewhere in the distance with no clue it's supposed to be an augmented reality app. How do you put that model on your couch? How do you hang it from your ceiling? How do you make that character talk when you tap him?

Init

Let's get you setup. You can download theHoloToolkit-Unity-vx.x.x.x.unitypackage from their releases page, import said package into Unity and profit. But this is only the dependencies you'll need for actually publishing. For us, school is in session and we need all the sources. 

You can also download the zip from the releases page. But if you want to stay up to date and you have Git installed already, I recommend cloning the repository.

This gives us access to the all the content in the Assets folder, which contains a very important Examples folder.

From this point, you can either copy the folders you want into your project or you can create a custom package out of them for fun and profit. You can open the project in Unity and use the export function

to select relevant folders you'd like to reuse. The HoloToolkit folder is the only folder your project needs, but we will be including the Examples and Tests folders for learning purposes.

Now you're a HoloToolkit distribution boss with ready access to all the goods. Import this package into your project and you are good to go.

Settings

After adding HoloToolkit-Unity to your project you will notice a handy menu option that shows up in Unity. Instead of having to worry about searching every nook and cranny of Unity to set the necessary requirements for a HoloLens app, you are able to check off these requirements with a few menu selections. 

Keep in mind, after setting the project settings option you will be asked to reload the project. Also remember you only need to check off the capabilities options that your app will actually need. So if you're doing spatial mapping, you'll need the spatial perception capability among some other capabilities.

Examples

There is so much functionality in this project that even covering it briefly would be a lengthy process. The Examples folder contains scenes that cover common scenarios using multiple functionalities. The Tests folder contains scenes that portray a single functionality. 

Keep in mind the folder structure and naming is bound to change. But the following pieces are fundamental and it's doubtful they're going anywhere. 

There's a few common functions that help with development that are worth explaining.

Cursor

There are several cursors available to us here, but loading the test scene HoloToolkit-Tests\Input\Scenes\Cursor.unity will show you how to use a cursor prefab and related scripts necessary to add a cursor to your HoloLens application and understand the input it provides from the user. The pleasant surprise is being able to simulate user input while debugging in Unity! 

Attached to the InputManager > EditorHandsInput game object you'll find the Manual Hand Control script. This class allows for manually controlling simulated hands while running inside an editor. You hold 'Shift' to enable moving the user's hand and pressing 'Space' will simulate a tap. The ManualHandControl class is destroyed when it detects that it's not running inside the Unity editor.

Spatial Mapping

You may already be aware of how spatial mapping and understanding works on HoloLens, but working with it in Unity is key to streamlining your development experience. I won't get into working with these scenes yet, but there is another big helper here that more developers should be aware of. 

If you've been inside the Device portal, you've probably already noticed that you can save a model of the room your HoloLens is scanning at the time.

So go ahead and do that one time, I'll wait until you get back...

Okay, you didn't go do that, and that's fine because HoloToolkit provides you with a couple models of rooms with different characteristics (and scanned better than you would anyways). There is a prefab located in HoloToolkit-Tests\SpatialMapping\Prefabs\ which has a nice room attached already. 

You can also find meshes of a few different rooms in the sibling folder Meshes to see how your project fairs in different environments. Pretty cool, huh? Not unless you have something do with it?

Go ahead and load the scene at HoloToolkit-Tests\SpatialMapping\Scenes\PlaneFinding.unity for a look at how these models can be used in Unity to prevent you from having to do an actual deployment to see how your application deals with a dynamic environment.

The PlaneFinding scene uses the built in room model. But the other scenes in this folder show how you would use the SpatialMapping prefab to load whatever room model you make or find. In these other scenes you select the SpatialMapping prefab to load any room model into the ObjectSurfaceObserver script. 

Now you may want to record your room and load it into one of these test scenes just to see how easy it is.

DeveloLens 3 - HoloToolkit Spatial Mapping >>

DeveloLens 1 - Introduction

HoloLens Development Impressions

The inspiration for the DeveloLens blog series came from my finding a lack of practical examples of how to utilize HoloLens abilities despite the wealth of functionality provided for it. I am not a professional blogger, but wanted to help others getting into this space. HoloToolkit-Unity is a great project that was put out there for all AR developers wanting to kick it into next gear. My goal is to provide some practical examples for others to see how easy it is to use. I welcome any and all feedback.

Background

In November my company, Illumination Works, purchased a few Development Edition HoloLenses. Being the most experienced with Microsoft development and Azure technologies, I was asked to familiarize myself with the device and development experience. 

I found that there is information out there for us when it comes to basics like setup, understanding what the HoloLens does conceptually, and "Hello World" examples. But there are not as many resources to get you from there to Young Conker. The good news is Microsoft has given us a really great lever for HoloLens functionality, the bad news is many will reinvent the wheel before they uncover these hidden artifacts. This is my "holo brain-dump", which will hopefully prevent a holographic arrow to the knee.

Install the moving target

If you have the device already, then you're familiar with the Install the tools page MS put up for us. This will get you going, but there are new versions of these tools coming out regularly. Even with all the stars aligned, there's a lot of monkeys throwing wrenches here. Maybe you're a wizard with requirements, but just in case you're a mere mortal like myself I wanted to share a few caveats.

The Emulator

If you don't have the device, this is pretty important piece to have. Even if you do, it's pretty handy when you're just trying to debug your application. If it's not deploying correctly from Visual Studio, then luckily you get to learn a lot more about the Hyper-V Windows feature that the emulators use. The unfortunate reality is, "run as admin" or not,  I still get this half the time:

But the other half of the time it's a very convenient feature.  

Having my environment on a company issued machine, and being on my company's network added a few kinks. So if possible, I recommend a dedicated machine that's without network admin concerns. The same kind of trouble comes with deploying to the actual device, as it's a breeze on my home network but Visual Studio never finds my device on the office network and I have to connect through USB.

The Windows 10 SDK

The hurdle that took me the longest time to figure out was the Windows 10 SDK versioning mismatch. When you log into your shiny new HoloLens, it wants to start updating, and you want it to start updating. But keep on eye on the sdk versions installed in the installation of Visual Studio you will be deploying with.

HoloToolkit

From here on out I'm going to assume you know Unity and can deploy a floating cube or maybe some Origami to your emulator or device. You're ready to have everyone think you're a wizard because you've mastered spatial mapping, and can make the HoloLens say their name. Welcome to the HoloToolkit.

Academy and HoloToolkit

The Academy lessons by Microsoft are helpful in understanding the capabilities of the HoloLens device. People come to the device with some extreme over-estimations of what it can do, so it's good to watch these videos and get a grasp of what it's actually doing. 

The problem I encountered was when I delved into the coding sections. They introduce you to the fantastic work being done by Microsoft developers called HoloToolkit, unfortunately they introduce you to what it was in November 2016. Much of toolkit has changed since then, for the better I might add. But this is a problem for beginners like me who didn't have much else to draw from.

HoloToolkit & HoloToolkit-Unity

Which one do you need to get started with? You can develop a HoloLens application without Unity, and that is where HoloToolkit comes into play. I assume it's for anyone wanting to build with a platform other than Unity, or some wizard who can visualize 3D objects through code and wants to build only in Visual Studio. 

But luckily for mortals and wizards alike, there is HoloToolkit-Unity. When you look at the two projects, you can see there is a lot more functionality and related examples here. It's comforting to see this big project with a lot of active development going on, but you have to stay up to date with all of this content if you want to use the latest and greatest.

Developing with HoloToolkit-Unity 

What is it and how do I use it? That's what the rest of this series is for. When you read the overviews in GitHub it can come off as daunting, you can quickly fall into a rabbit hole of descriptions with no practical application. Keep coming back to learn more about what you can do with this great project.


DeveloLens 2 - HoloToolkit-Unity init >>