Detect hand and retrieve joints

An example on how to detect hands and retrieve finger joint information.

This example demonstrates how to load an image file into a Texture2D, implement hand tracking with the HandTracker, and retrieve joint information of the detected index finger.

Step 1: Load the Image into Texture2D

1. Ensure you have your image file ready, preferably in a supported format such as PNG or JPEG.

2. Use the appropriate method (e.g., Texture2D.LoadImage) to load your image into a Texture2D object.

// Create an empty Texture2D.
Texture2D texture = new Texture2D(0, 0);

// Load a file to a byte array.
string filename = "path/to/image";
byte[] rawData = System.IO.File.ReadAllBytes(filename);

// Load an byte array to the Texture2D.
texture.LoadImage(rawData);

Step 2: Create and Use a HandTracker

1. Instantiate a HandTracker object in your application. The HandTracker is responsible for detecting hands within the texture.

2. Pass the Texture2D object created in Step 1 to the HandTracker for processing.

3. The HandTracker will analyze the texture and detect any hands present in the image.

// Create a handTracker.
HandTracker tracker = new HandTracker();

// Detect hands.
List<Hand> hands = tracker.Load(texture);

Step 3: Retrieve Index Finger Joint Information

1. Once a hand is detected in Step 2, you can retrieve information on the detected hand, its fingers and their finger joints.

2. Specifically, get the hand side (left or right) and the index finger's information. The index includes, among other joints, the 2D and 3D positions of the index metacarpophalangeal joint as shown in this example.

// If at least one hand is detected
if (hands.Count > 0)
{
    // Get hand side. 
    HandSide side = hands[0].Side;
    
    // Get index finger.
    Index index = hands[0].Index;
    
    // Get index MCP finger joint.
    FingerJoint indexMCP = index.MCP;
    
    // Get joint 2D position.
    Vector2 screenPosition = indexMCP.Position2D;
    
    // Get joint 3D position.
    Vector3 worldPosition = indexMCP.Position3D;
}

Full example code

string filename = "path/to/image";
byte[] rawData = System.IO.File.ReadAllBytes(filename);
Texture2D texture = new Texture2D(0, 0);
texture.LoadImage(rawData);

// Create a handTracker.
HandTracker tracker = new HandTracker();

// Find hands in an image.
List<Hand> hands = tracker.Load(texture);

// If at least one hand is detected
if (hands.Count > 0)
{
    // Get hand side. 
    HandSide side = hands[0].Side;
    // Get index finger.
    Index index = hands[0].Index;
    // Get index MCP finger joint.
    FingerJoint indexMCP = index.MCP;
    // Get joint 2D position.
    Vector2 screenPosition = indexMCP.Position2D;
    // Get joint 3D position.
    Vector3 worldPosition = indexMCP.Position3D;
}

By following these steps, you will be able to load an image, detect hands within it, and obtain detailed joint information for the index finger using aTexture2D and the HandTracker.