MRTK 101: How to use Mixed Reality Toolkit Unity for Crucial Spatial Interactions (HoloLens 2, HoloLens, Windows Mixed Reality, OpenVR, OpenXR)

Learn how to use MRTK to achieve some of the most widely used common interaction patterns in Mixed Reality This article is based on MRTK 2.7 release (May 2021)

  • How to grab and move an object?
  • How to resize an object?
  • How to move or rotate an object with precision?
  • How to make an object respond to input events?
  • How to add visual feedback?
  • How to add audio feedback?
  • How to use HoloLens 2 style button prefabs?
  • How to make an object follow you?
  • How to make an object face you?

How to simulate input interactions in Unity editor?

MRTK supports in-editor input simulation. Simply run your scene by clicking Unity’s play button. Use these keys to simulate input.

  • Hold the Right mouse button and move the mouse to look around.
  • To bring up the simulated hands, press Space bar(Right hand) or Left Shift key(Left hand)
  • To keep simulated hands in view, press T or Y key
  • To rotate simulated hands, press and hold the Ctrl key and move the mouse
MRTK’s Input simulation in Unity editor

How to grab and move an object?

To make an object grabbable, assign these two scripts: ObjectManipulater.cs and NearInteractionGrabbable.cs(for direct grab with articulated hand tracking input) ObjectManipulator supports both near and far interactions. You can grab and move an object with HoloLens 2’s articulated hand tracking input(near), hand ray(far), motion controller’s beam(far), HoloLens gaze cursor & air-tap(far).

Grab and move with input simulation — far interaction with hand ray / near interaction with articulated hand

How to resize an object?

ObjectManipulator.cs supports two-handed scale/rotation. This works with various input types such as HoloLens 2’s articulated hand input, HoloLens 1’s gaze + gesture input, and Windows Mixed Reality immersive headset’s motion controller input.

Resizing object with two-handed manipulation

How to move, rotate, and scale an object with precision?

Assign BoundsControl.cs to an object to use Bounding Box which is the interface for scaling and rotating an object. By default, it shows HoloLens 1 style blue handles and wires. To use HoloLens 2 style proximity-based animated handles, you need to assign prefabs and materials. Please refer to Bounds Control documentation and the BoundingControlExamples.unity scene for the configuration details.

Default Bounding Box and HoloLens 2 style in the BoundingBoxExamples.unity scene

How to make an object respond to input events?

Assign PointerHandler.cs to an object. In the inspector, you will be able to use events OnPointerDown(), OnPointerUp(), OnPointerClicked(), OnPointerDragged()

How to add visual feedback?

Assign Interactable.cs to an object. In the inspector, create a new theme. Using Interactable’s theme profiles, you can easily add visual feedback to all available input interaction states.

Assigned material with MRTK Standard Shader’s proximity light and border light options

How to add audio feedback?

Add AudioSource to an object. Then, in the scripts that expose input events(e.g. Interactable.cs or PointerHandler.cs), assign the object to the event and select AudioSource.PlayOneShot(). You can use your audio clips or choose one from MRTK’s audio assets.

How to use HoloLens 2 style button prefabs?

MRTK provides various types of HoloLens 2’s shell(OS) style buttons. It provides sophisticated visual feedbacks such as proximity light, compressing box, and a ripple effect on the button surface to improve usability with hand-tracking input.

PressableButtonExamples scene

How to make an object follow you?

Assign RadialView.cs or Follow.cs script to an object. It is part of the Solver script series that allows you to achieve various types of object positioning in 3D space. SolverHandler.cs will be automatically added.

Example of using Radial View Solver on the Toggle Features Panel in the HandInteractionExamples scene

How to make an object face you?

Assign Billboard.cs script to an object. It will always face you, regardless of your position. You can specify the pivot axis option.

The left object always faces the camera with Billboard script, with Y-axis pivot option
Example of using Billboard script in the Tooltip prefab to make the labels always face the camera


Microsoft MR Dev Days 2020 Session Video: MRTK’s UX Building Blocks

Read my other stories

Designing Type In Space for HoloLens 2

Designer & Creative Technologist. UX Designer @Microsoft HoloLens. Creator of Opinions are my own.