Holographic development with Unity

Holographic development with Unity

Thursday, May 14, 2015 Josué Yeray unity 2 Comments

In January of this year Microsoft made public their most innovative and disruptive product in quite some time called HoloLens, an augmented reality headset that combines breakthrough hardware, input and machine learning so that you can bring mixed reality experiences to life using the real world as your canvas. These are not just transparent screens placed in the center of a room with an image projected on them but truly immersive holograms that enable you to interact with the real world. This is a truly innovative product with a rich set of APIs that enable you to develop Windows Holographic applications that will blur the line between the real world and the virtual world.

As impressive as this may sound, Microsoft has been very quiet about this technology; only allowing a few videos and bits of information to be released. But at the most recent Microsoft Developer Conference//Build 2015, they allowed a select group of people, around 180 people in total including me, to try out this new technology.

Creating a 3D World... in the real world

One of the strengths of HoloLens is the ability to blur the line between the real world and the virtual world. To make this possible and get the best immersive experience, you need to be able to define 3D objects that can be placed in the real world so that you can interact with them. To accomplish this, you need something more specialized than Visual Studio because rather than require a new development environment for their new hardware platform Microsoft opted to use a tried and true software development environment called Unity3D.

Unity3D allows you to develop 3D scenes and games for a lot of different platforms, like iOS, Android, Windows Store, Windows Phone, Windows Desktop... and on April 29th the new HoloLens Platform.

Microsoft and Unity worked closely together to enable developers to quickly and easily develop for the HoloLens platform. In fact, with only minimal changes to your scenes, you can quickly render them in Hololens. So let's take a look at how this is done.

When you create a new Scene in Unity, you will initially have two elements: the scene's main camera and the scene's main directional light:

Those are the most basic objects any virtual world needs. The camera is the viewport we are going to use to actually "see" the world objects. The light is needed to interact with the objects and enable the audience to see more than a black scene.

Initially by default the world you can see is totally empty of any "Assets". Since Unity has a lot of default Assets you can choose from for your creations, you only need to go to the Assets menu, Import package submenu and select from one of the options listed to add: characters, vehicles, particle effects, etc...

In the example above, you could note that there is no "world" where your assets are positioned but in fact, there is one, the real world because when doing Holographic programming you need to take into consideration that the objects will be positioned in the real world, so you don't need to create a complete scenario for them. Also you need to keep in mind that the viewer uses the camera to adjust their position so the view is not fixed. Working with Holograms, the user is in control of the camera position which is used to adjust their real position.

Using all of the assets, you create your scene. You can also control their behavior, events, animations, etc... using scripts you can write in C#. These scripts are .NET classes, based on the MonoBehavior base class:

using UnityEngine;
using System.Collections;

public class EmptyScript : MonoBehaviour {

 // Use this for initialization
 void Start () {
 
 }
 
 // Update is called once per frame
 void Update () {
 
 }
}

An empty script like the one above has two methods by default:

• Start is executed when the object is created and loaded in the scene.
• Update is executed one time per frame.

You are going to use both of these methods a lot when developing with HoloLens so let's take a look at how this would be done.

By default Unity opens the scripts in MonoDevelop environment, but you can change it to open using Visual Studio by simply going to the Preferences option in the menu. In the preferences window, go to the External tools option in the list of options on the left. One of the available options is the External Script Editor:

Open the External Script Editor dropdown and click on browse. Go find the folder were you have Visual Studio installed and find the devenv.exe executable file. Now the next time you open an asset, it will be opened in Visual Studio.

Note: the first time you do this it may be a bit slow.

By way of this brief introduction, you now have all the information you need to start working with Unity, not only for Holograms, but for any type of project.

Next, let's take a look at what HoloLens from Microsoft will bring to Unity to make holograms come alive in our real world.

HoloLens

All the information we have on the HoloLens product comes from private sessions with Microsoft held during the //Build 2015 conference in San Francisco from April 30th to May 1st. Of the 5000 attendees to the conference, only one hundred were invited to attend the 3 hour hands on lab for the current internal SDK for Unity and real HoloLens devices and I was lucky enough to be among those selected for the "Holographic Academy":

So what is needed to make a standard Unity 3D scene work with HoloLens and how do you add the HoloLens SDK to Unity? You may be surprised to learn that the answer to these two questions is: very little!

The most complicated thing is to make a switch in your head about what you are going to do, and how it is going to be drawn. In most games the camera is a fixed point that may follow a character or a vehicle and normally the user can't control it directly, unless you create a First Person Shooter (FPS).

You need to think about your camera when working with HoloLens because it was designed for an FPS and the user is the camera. As the user moves in the real world, the camera moves around your scene, keeping it in sync with the user movement to create the hologram that are integrated with the rest of the real world viewed by the user.

So the first thing you need to know is how positioning and units are related to the user and real world. HoloLens maps Unity measurement units to meters in the real world. Positioning objects using X,Y,Z values of 0,1,1 will put the object centered horizontally with the user, 1 meter over the floor and 1 meter ahead of the user. This works also for the rotation and scale of the objects.

To be able to see your scene in HoloLens, you will need to replace the standard Unity camera included in your scene with a HolographicCamera that is included with the HoloLens SDK assets added to Unity. With this simple change you are ready to deploy your first app to HoloLens.

But wait there's more. HoloLens SDK adds more than just a custom camera to Unity. In the scripts side, HoloLens adds new namespaces that allows you to access HoloLens headset information like head position and the vector for line of sight so that you can obtain the object the user is looking at.

It also adds gestures APIs to detect an "Air Tap" which is the gesture created by Microsoft that HoloLens recognizes as a tap/click over the scene or selected object.

Last but not least, it adds voice commands. You can define a text, associated to a method. When the text command is recognized, the method is launched.

By far the most impressive part is the way you can make your scenes collide and interact with the real world. Instead of implementing this manually, it is designed to handle this automatically using the HoloLens SDK. You only need to place a mesh that is included with the Holographic Assets added to Unity and all objects with physics and gravity will collide with the mapped surface of the real world. The mapping is very fast and is updated every frame, so if you launch a sphere in one direction and then you put an object like a chair or a person in the path of the sphere, it will collide with the real world object, bouncing as if both objects were real.

Safe guarding your holographics with Plastic SCM

Plastic SCM is integrated with Unity so it is very easy to set it up to safe guard your Unity project containing your scenes and scripts.

To do this the first thing you need is to have a Unity Pro license and a Plastic SCM license which you can obtain from the Unity Assets Store (Window menu > Assets Store):

After you have all the prerequisites in place the configuration is really straightforward. Go to Edit menu > Preferences, in External tools where you will find the revision control diff/merge option. If all is correctly installed, you will see Plastic SCM merge. If is not selected, select it from the dropdown.

The next step is to go to Edit menu > Project settings > Editor where the editor settings pane will be shown on the right side of the Unity environment. In the Version Control mode option, select Plastic SCM from the dropdown. If you receive an error this means that "The Unity3D plugin is unlicensed in your Plastic SCM server" and your Plastic SCM server version will not support the Unity3D plugin. In order to use the Unity 3D plugin your Plastic SCM installation needs to be Team or Enterprise. If you only have Personal or Community, the Unity 3D plugin is not included in the license agreement.

Once you have successfully selected Plastic SCM as the version control mode in Editor settings, you will need to specify a repository and workspace name.

If you have not previously created a workspace then you will need to go to Plastic SCM GUI > Repositories & Workspaces and create a new Workspace in the root folder of your Unity Project:

The final step is to press the connect button in the Unity editor settings and, if all is set up successfully, you will be ready to start working with Plastic SCM from within the Unity Editor. Note that all of the files in project tab are marked with a red dot:

This means that the files have not been added to source control.

You can select the root folder Assets by right clicking over it and selecting Source control > Mark as Add and all assets will be correctly added to the workspace, even the .meta files.

You can see what Plastic SCM is tracking and what it is not tracking by going to Window menu > Version control:

You can now commit the pending adds to your repository by clicking again in the Assets root folder > Version control > Submit. This will bring up a preview window where you can enter a commit description and see the changes being sent to the repository:

After completing this final step, you have now finished the basic integration between Plastic SCM and Unity 3D. But Plastic offers you more than simply preserving your files. Now that you have your files correctly checked into the repository you can go to any script and open it in Visual Studio or MonoDevelop, whatever you prefer. You can then edit the script to change the method, add code or do anything else you need then close the code editor and return to Unity.

The script you have modified is marked with a red check to let you know that it has pending changes. But instead of directly submitting the changes... you can use the first option to do a Diff to see what was changed compared to the repository version.

By right clicking on the updated script Version control > Diff > Against Head / Against Head with .meta, you can use this second option to create a diff between both the associated cs file and the meta file while the first option only compares the cs files:

Another common scenario while working with Unity is the modification of textures. Sometimes you need to compare changes in a picture file but normally image changes are compared at the binary level, and to be honest, this is basically useless unless you are a Matrix operator and can see the image in the code.

So the next best thing is to go to your project and Modify one texture. When you return to Unity3D right click on the texture and go to Version control > Diff > Against Head and this will be the result:

As I have said before, this is not very useful because it is just a bunch of binary changes that no one can understand. But there is a way around this. By simply opening the Plastic SCM GUI and IN the Items in the Workspace tab, you can see the texture checked out and with changes. By right clicking over it, select Diff > Diff with previous version and the result will be a side by side comparison of the image:

Now this is useful, because you can see both versions side by side. In this case the changes are quite obvious, but this is not always the case. In the top border of the Diff window, you can see four comparison options:

• Side by side: by default, one image side by side with the other.
• Onion skin: the repository version is displayed and you have a slider to show the edited version over it with transparency so you can see the changes appearing in the original image.
• Differences: quite useful for complex changes, display all image in white, except for the changed zones.
• Swipe: put one image over the other and you can display a greater percentage of one of them with a slider.

In this way, Plastic SCM can help improve the way you manage and diff your textures.

Conclusion

With Microsoft HoloLens, the future is here and now. Holograms are not just for entertainment but also for professional applications in a lot of fields such as: Mechanical, Industry, Medical, Engineering and Architecture. By using Unity3D, you are able to create your best work easily and quickly. And by also using Plastic SCM, you can be assured that your work is safe guarded and your work processes improved by using tools such as the Image Visual Diff. For more information about Plastic SCM for Unity visit Plastic SCM site.