The forearm as a touchscreen. A directly touchable interface, reconstructed in real time from the Meta Quest 3's environment depth. No controllers, no markers, no external hardware.

This is an on-body interaction system for extended reality: the forearm acts as the display surface and the opposite hand as the input, bringing phone-style direct touch to bare skin.

The forearm surface is reconstructed each depth frame from the Quest 3's environment depth, which the headset computes by stereo-matching its passthrough cameras. The pipeline spans the GPU and Burst-compiled worker threads and is frame-pipelined: reconstruction jobs are scheduled asynchronously, and the main thread uploads each finished mesh only once its jobs complete.

In brief: for each new depth frame, take the headset's depth image, discard everything that is not forearm, turn what remains into a mesh, and project a touchable UI onto it.

_{The per-depth-frame pipeline. On the GPU, the interacting hand is grown into a two-zone mask, carved out of the depth image, and stabilized with a 3-frame temporal median. An async readback hands the stabilized depth to CPU Burst jobs that unproject it into world space, isolate the forearm patch (seed, BFS flood, boundary smooth), and build the mesh. The mesh is uploaded back to the GPU to composite the touchable UI in passthrough.}

Each frame, the fingertip joints reported by hand tracking (currently the index fingertip) are tested against the reconstructed surface: the nearest surface cell within range is found, the signed distance above the surface is computed, and a UV coordinate is derived at sub-cell precision from the finger position. Touch comes from the tracked skeleton rather than from sensing the fingertip in the noisy depth map, which keeps it stereo-stable. The surface keeps updating throughout interaction (pronation included), with no freeze step.

UV is a linear projection (not cylindrical). The camera-fixed lateral axis keeps the viewport upright regardless of wrist rotation, and pronation adds a U scroll offset so rotating the wrist reveals new content rather than spinning the image. Two panels: U=[0,0.5] dorsal, U=[0.5,1] palmar.

Everything runs on the headset and the pipeline is built to disappear into the frame budget.

• Holds the Quest 3's 72 fps with the full pipeline running (~4 ms of app GPU time per frame against the ~13.9 ms budget).
• Reconstruction is gated to the depth sensor's cadence (~25 Hz). A bit-exact check on the depth reprojection matrix skips redundant frames, so the same depth data is never reconstructed twice; rendering and touch stay decoupled at the headset's refresh rate.
• No main-thread stalls. The CPU stages are Burst-compiled jobs spread across worker threads and frame-pipelined: the readback, segmentation, and meshing for one depth frame are harvested a frame later, so the main thread never blocks on them.
• Only the arm crosses the bus. The async GPU readback is cropped to the forearm at the depth texture's native resolution (one float per texel) so the GPU->CPU transfer scales with the arm, not the screen.

Figures are measured on-device from a clean boot and vary with build, scene, and device state.

Bracer XR is a research prototype with a deliberately narrow scope:

• Quest 3 only. It depends on Meta's environment depth API and Movement SDK body tracking; it does not run on other headsets.
• The depth source is low-resolution. Quest's environment depth is a single ~320×320 image spread across the entire field of view, so the forearm occupies only a fraction of it. The surface captures the arm and a finger fine, but thin gaps and small, fine-grained features are below what it can resolve.
• A hovering finger lifts the depth around itself. Quest's stereo depth pulls arm texels toward a nearby finger (a mixed-pixel "bleed"), which would raise the reconstructed surface at the contact point. The pipeline reconstructs that ring from the clean arm just outside it, but a slight residual raise remains at the resolution limit.
• The arm patch is found from depth, not trusted from the skeleton. Wrist and elbow bone tracking is only a rough starting point and can be inaccurate, so the patch is seeded from a tight, confident region near the bone axis and grown by a BFS flood through depth-connected cells. The flood is walled to forearm and palm cylinders because, left unbounded, BFS can leak across depth-continuous neighbors into surfaces that are not the arm.
• One forearm, one interacting hand. The system reconstructs a single arm and tracks a single touching hand at a time.
• The touching hand is masked out, not reconstructed under. To keep it from corrupting the mesh, the hand is cut from the depth, leaving a hole exactly where its silhouette sits; the surface around it is rebuilt from neighboring arm, so the canvas reaches the hand. Touch is detected from hand tracking rather than by sensing the fingertip in the depth map, so this does not break interaction.

• Hand tracking can occasionally fail to initialize on launch (OVRHand never reports valid data), so the hands don't lock on and touch never registers. A mitigation is in place but is not fully reliable; the issue is being investigated. Restarting the app usually recovers.

git lfs install
git clone https://github.com/treytuscai/bracer-xr.git

1. Open the project in Unity Hub with editor 2022.3.62f3.
2. Open Assets/_Project/Scenes/MainScene.unity.
3. Connect the Quest 3 over USB (adb devices to confirm).
4. File -> Build Settings -> confirm Android is the active platform and the scene is listed -> Build and Run.

All project, SDK, and OVRManager settings are committed. No manual configuration is required.

Once the app is running on the headset:

1. Hold out your left arm in view of the headset. The UI appears on your forearm once body and hand tracking lock on.
2. Touch with your other hand. Bring a fingertip to the surface. Contact registers from a slight hover down through a shallow press (touchHoverHeight / touchDepth).
3. Rotate your wrist to switch panels. The display has two: dorsal (U=[0,0.5]) and palmar (U=[0.5,1]).
4. Turn your arm upright or sideways to swap between the portrait and landscape images (portraitTexture / landscapeTexture). Set orientationMode to lock to one. The image is swapped, not rotated, so author each one in its own orientation.

The surface updates continuously. Keep your hand on it while moving or rotating the arm. There is no need to pull away or recalibrate.

Just want to see the reconstruction? Leave surfaceMaterial unset on the ForearmDepthSurface component and the mesh renders as a translucent cyan debug surface, no material or textures to author. To project a UI onto it, assign a material whose shader exposes _MainTex and set portraitTexture / landscapeTexture.

The on-arm UI is static art. portraitTexture and landscapeTexture are fixed images stretched onto the display window; they do not adapt to arm size or shape, so a layout authored for one forearm may not line up on another. Treat displayWidth / displayHeight / displayOffset (and homeTwistDeg) as physical fits to the wearer, then author the textures to the window you set.

master is the surface system itself. The experiments branch holds example experiment apps built on top of the surface together with Tyler (@tydevlieger). That branch is experimental: quick study code, unoptimized and unpolished. Read it as a demo of what the surface supports, not as part of the system itself.

Application code and assets live in Assets/_Project/; everything else under Assets/ is SDK- or Unity-managed.

Assets/_Project/
├── Scripts/
│   └── Surface/
│       ├── Core/                  depth pipeline stages
│       │   ├── ArmFrame.cs            wrist/elbow -> arm coordinate frame
│       │   ├── DepthReadback.cs       dispatch, async GPU readback + CPU unprojection
│       │   ├── DepthStabilizer.cs     GPU hand mask + 3-frame temporal median
│       │   ├── HandMask.cs            baked hand mesh + fingertip positions
│       │   ├── SurfaceExtractor.cs    seed + BFS flood isolation
│       │   ├── BoundarySmoother.cs    parallel Burst boundary smoothing
│       │   ├── MeshGenerator.cs       parallel mesh + UV + normal emission
│       │   └── SurfaceUV.cs           shared display UV mapping (mesh + touch)
│       ├── Buffer/                shared NativeArray data buses
│       │   ├── SurfaceBuffer.cs
│       │   └── MeshBuffer.cs
│       ├── ForearmDepthSurface.cs     root MonoBehaviour; orchestrates the pipeline
│       └── ForearmInteraction.cs      touch detection against the surface
├── Shaders/
│   ├── ForearmProjection.shader     URP transparent shader for the surface mesh
│   ├── DepthTemporalMedian.shader   3-frame reprojected median that stabilizes the depth
│   └── HandMaskRender.shader        hand silhouette grown into the two-zone mask
├── Materials/   Scenes/   Textures/

Primary tuning surface, on the ForearmDepthSurface component:

Touch tuning, on the ForearmInteraction component:

The original work in this repository (primarily the code under Assets/_Project/) is licensed under the Apache License 2.0, © 2026 Trey Tuscai. You are free to use, modify, and build upon it, provided you retain the copyright and attribution notices (see NOTICE).

Third-party SDKs vendored elsewhere under Assets/ and Packages/ (Meta XR SDK, Meta Movement SDK, etc.) are governed by their own license terms, not the Apache License above.