Unity VR: Controller Tracking

Intro

In the XR Interaction Toolkit, interactions are broken down into Interactors and Interactables, forming a structured way for objects to interact in a VR/AR environment.

Interactors

Interactors are objects that initiate interactions with other objects. These are typically hand controllers, ray pointers, or even hands in hand-tracking systems.

Common Interactors

• XR Ray Interactor — Used for laser pointer-style interactions (UI, distant grabbing).
• XR Direct Interactor — Used for hand-based or controller-based direct grabbing.
• XR Socket Interactor — Used for snap-to-position object placements.
• XR Grab Interactable — Allows objects to be picked up and manipulated.
• XR Poke Interactor — Used for finger-poking interactions with UI or buttons.

Interactables

Interactables are objects that respond to interactions from interactors. These can be grabbable objects, UI buttons, teleportation areas, or physics-based objects.

Common Interactables

• XR Grab Interactable — Makes objects grabbable using direct or ray-based interaction.
• XR Poke Interactable — Allows finger-poke interactions with buttons and touch screens.
• XR Teleportation Area/Anchor — Allows users to teleport to a designated location.
• XR UI Canvas — Enables ray-based interactions with VR UI elements.
• XR Climbable — Allows users to grab and climb surfaces.

This article will explain how to build trackable controllers with position and rotation tracking, and demonstrate a Ray Interactor with a visual component to see it in action in VR.

Creating Controllers

When you create an XR origin, it creates an empty game object, Camera Offset, which can calculate the height from the physical floor to the headset on the user. It also contains the Main Camera that acts as the user’s eyes in the scene. A new component that is attached to the Main Camera, which comes with the Unity XR Interactive Toolkit is the Tracked Pose Driver.

Tracked Pose Driver

The Unity component Tracked Pose Driver (Input System) updates an object’s position and rotation using XR device tracking data. This component is typically applied to the camera or a controller to ensure it mirrors the movement of an XR device, like a VR headset, controllers, or AR devices.

Create Controllers

The XR Origin comes pre-configured, so the VR headset being used is already tracked and matches the position and rotation of that VR headset. This allows the user to look around in the virtual environment. However, to enable interaction within the virtual space, we need to introduce controllers as the headset alone does not provide hand tracking.

An empty game object should be created and parented to the Camera Offset, ensuring it moves with the user. Name this game object “Left Controller” and add a Tracked Pose Drive (Input System) to it to enable tracking of the XR device’s controller. A reference to the default XRI Input Action Asset can be used to assign the Left Position, Rotation, and Tracking State actions to the corresponding inputs, or assign the Action using the same Input Action Asset.

The position and rotation of the Left Controller are being tracked. However, there is no visual representation of the hands being tracked due to the absence of a model.

Ray Interactor

The XR Ray Interactor in Unity’s XR Interaction Toolkit (XRI) uses a laser pointer-style visual and allows users to interact with distant objects, commonly used for UI interactions, grabbing distant objects, and teleportation.

This will not yet give us the laser pointer to track our hand. This only handles the interactions with game objects that are interactables.

Laser Pointer

To add the laser-pointer, we need to add a Line-Renderer component to draw the line and the XR Interactor Line Visual component. The XR Interactor Line Visual provides a visible representation of an XR Ray Interactor.

You can set the line type width and color visual, but those Valid, Invalid, Blocked colors depend on the raycast of the XR Ray Interactor, which is another article, but the default color is based on the Material color.

Rinse & Repeat

For the right controller, you can just duplicate the left controller, rename it and make sure to switch the position, rotation and tracking actions from left to right.

For the XR Ray Interactor make sure the Handedness property is set to Right. This property defines which hand the interactor is associated with which helps distinguish between left-hand and right-hand interactions.

Conclusion

To create a controller that tracks hand movement using the XR Interaction Toolkit (XRI) and displays a laser-like visual, you will need the following components

• Tracked Pose Driver to sync position & rotation.
• XR Ray Interactor for raycasting and object selection.
• XR Interactor Line Visual & Line-Renderer to draw the XR Ray Interactor.

Now, your VR controller will track hand movement and display a laser pointer for interaction!