Use webcam to detect and visualize hands (3D World space)

An example on how to get camera feed, detect and visualize hands in real time in 3D world space.

This example demonstrates how to load and display camera feed in a Unity scene with a WebcamSource and an ImageView, implement hand tracking with the HandTracker, and use the HandManager to render detected fingers in 3D world space.

This is a code walkthrough of the LightBuzz_Hand_Tracking_3D Hand Tracking Unity plugin sample. The plugin includes the no-code demo that has the same functionality.

Step 1: Create a Unity scene

Open the Unity Project you created in the Installation section.

Right-click on the Assets folder and select Create > Scene.

Create new scene

Type the scene's name. In this example, we'll use the name WebcamDemo3D.

After the scene is created, right-click on the scene and select GameObject > UI > Canvas.

Add a Canvas

Navigate to the LightBuzz Prefabs folder at Assets\LightBuzz Hand Tracking\Runtime\Prefabs.

LightBuzz Prefabs

For this example, you'll need the ImageView, WebcamSource and HandManager prefabs.

Drag and drop the ImageView and WebcamSource prefabs into the Canvas.

Add prefabs into Canvas

Drag and drop the HandManager prefab on the Hierarchy pane (but not into the Canvas).

Add HandManager prefab

To see the 3D detections, the HandManager needs to be outside of the Canvas.

Then, right-click on the Hierarchy pane and select Create Empty.

Create empty component

Give a name to the new component. In this example, we'll use the name Demo.

Then, go to the Inspector pane and select Add Component. In the search bar, type new and select the New script option.

Add script

Type the script's name and select Create and Add. For this example, we'll use the name WebcamDemo3D.

Create script

Step 2: Initialize the visual components

Double-click on the newly created MonoBehaviour script and import the necessary namespaces.

using LightBuzz.HandTracking;
using System.Collections.Generic;
using UnityEngine;

For this example, we'll need a WebcamSource to get the frames, an ImageView to draw camera texture and a HandManager to visualize the detected hands.

[SerializeField] private ImageView _image;
[SerializeField] private WebcamSource _webcam;
[SerializeField] private HandManager _handManager;

After adding the serialized fields, go to the Unity Editor to connect these fields with the Demo component.

At the Inspector pane, select the round button next to each SerializeField.

Seriaze fileds

Then, at the Scene tab, select the corresponding prefab. For example, for the Image field, select the ImageView prefab.

Connect prefab

When all fields are connected, the result should resemble the following image.

Connected serialize fields

Then, select the HandManager prefab. Connect the Image field to the ImageView prefab and uncheck the Is 2D option.

Connect field to HandManager

Make sure the Is 2D option is NOT selected to see the hand tracking detections in the 3D world space. If the option is checked, detections are displayed in the 2D space.

After connecting all the fields, navigate to the Canvas to set the render options.

Change the Render Mode to Screen Space - Camera.

Change Render Mode

Then, set the Main Camera, from the Scene tab, as the Render Camera.

Select Render Camera

When all the render options are set, the result should look like the following image.

Render settings

Step 3: Adjust Scene components

To display the hand detections in the 3D world space we need to adjust the ImageView.

Navigate to the ImageView prefab, under the Canvas, go to the Aspect Ratio Fitter section and change Aspect Mode to None.

Aspect Mode settings

Then, go to the Rect Transform section of the ImageView prefab and click on the blue cross to open the Anchor Presets.

Anchor Presets

Select the middle and center option (the one that looks like a target).

Then, change the Width to 400, the Height to 200 and the Pos X to -600.

Rect Transform

The suggested Rect Transform settings are designed to reduce the size of the ImageView and reposition it to the side. Feel free to experiment and set the settings to different values.

For a proper view, navigate to the Main Camera component and set the Y Position to 0, since the 3D coordinates of the hands are estimated relative to the cartesian origin point. Also, set the Z Position to -1 to move the camera farther from the hands.

Main Camera Transform

Finally, return to the MonoBehaviour script and instantiate a HandTracker to detect hands.

HandTracker _handTracker = new HandTracker();

Step 4: Open the webcam

Open the webcam to get the live feed.

_webcam.Open();

In this example, the camera is opened in the Start() method. Alternatively, you could open the camera with the click of a button.

Step 5: Get the live feed

Check that the camera is open and available for capturing video.

if (!_webcam.IsOpen) return;

Load the new frame from the _webcam object onto the ImageView to show the live feed to your users.

_image.Load(_webcam);

Step 6: Detect hands

Pass the Texture2D object from the camera frame to the HandTracker for processing.

List<Hand> hands = _handTracker.Load(_image.Texture);

The HandTracker will analyze the texture and detect any hands present in the image.

Step 7: Visualize the hands

To display the detected hands in 3D world space, first sort the hands by position X.

hands.Sort((h1, h2) => 
    h1[FingerJointType.Root].Position2D.x.CompareTo(
        h2[FingerJointType.Root].Position2D.x));

By default, the HandManager positions the root joint of any detected hand at the origin point (0, 0, 0) within the 3D world space. To track multiple hands and visualize them clearly within the 3D environment, you need to provide an offset value in meters for each hand. Provided with an offset list, the HandManager will reposition each hand according to the offset value.

In this example, each new hand will be positioned 25 cm away from the previous one.

List<Vector3> offsets = new List<Vector3>();
float offset_x = 0f;
float step = 0.25f;

foreach (Hand hand in hands)
{
    offsets.Add(new Vector3(offset_x, 0f, 0f));
    offset_x += step;
}

Then, simply pass the hand detections and the offsets to the HandManager. It will manage the rendering and updates required to accurately depict the hands in 3D world space based on the detection data provided.

_handManager.Load(hands, offsets);

Steps 5 through 7 are incorporated into the Update() method.

Step 8: Release the resources

Close the webcam to stop the live feed, preventing further video capture.

_webcam.Close();

Dispose of the HandTracker object to ensure that all associated resources are released.

_handTracker.Dispose();

In this example, the resources are released in the OnDestroy() method. Alternatively, you could do that with the click of a button or in the OnApplicationQuit() method.

Full example code

Here is the full example code that has the same functionality as the Hand Tracking Unity plugin LightBuzz_Hand_Tracking_3D sample.

using LightBuzz.HandTracking;
using System.Collections.Generic;
using UnityEngine;

public class WebcamDemo3D : MonoBehaviour
{
    [SerializeField] private ImageView _image;
    [SerializeField] private WebcamSource _webcam;
    [SerializeField] private HandManager _handManager;
    
    private readonly HandTracker _handTracker = new HandTracker();

    private void Start()
    {
        _webcam.Open();
    }

    private void OnDestroy()
    {
        _webcam.Close();
        _handTracker.Dispose();
    }

    private void Update()
    {
        if (!_webcam.IsOpen) return;

        // 1. Draw the camera texture.
        _image.Load(_webcam);

        // 2. Detect hands in the camera texture.
        List<Hand> hands = _handTracker.Load(_image.Texture);
        
        // 3. Sort hands by position X.
        hands.Sort((h1, h2) => 
            h1[FingerJointType.Root].Position2D.x.CompareTo(
                h2[FingerJointType.Root].Position2D.x));

        // 4. Add offset to show all detected hands.
        List<Vector3> offsets = new List<Vector3>();
        float offset_x = 0f;
        float step = 0.25f;

        foreach (Hand hand in hands)
        {
            offsets.Add(new Vector3(offset_x, 0f, 0f));
            offset_x += step;
        }

        // 5. Visualize the hands on a Canvas.
        _handManager.Load(hands, offsets);
    }
}

By following these steps, you will be able to load the camera feed into your application, detect hands in real time, and finally, render these detections in 3D world space.