Supervisor: Taha Khan
HAND-CONTROLLED POINT CLOUD
EXPERIENCE
An interactive experience where hand gestures transform digital creatures in real time.
TouchDesigner · MediaPipe · Point Clouds · Computer Vision
OVERVIEW
This project explores how natural movement can replace traditional controls. Using MediaPipe hand tracking and TouchDesigner, users interact with 3D point-cloud creatures entirely through hand gestures. An open palm dissolves the creature into particles, while a closed fist switches between different creatures.
The aim was to make interaction feel immediate and physical, removing the keyboard and turning the user’s body into the interface.
THE EXPERIENCE
Each creature is constructed from thousands of individual points. When an open palm is detected, procedural noise displaces these points, transforming the recognizable creature into an abstract particle cloud. Closing the hand into a fist triggers the next creature, allowing the experience to continuously shift between different digital forms.
OPEN PALM → Dissolve CLOSED FIST → Transform
HOW IT WORKS
A webcam captures the user’s hand while MediaPipe tracks its landmarks in real time. TouchDesigner interprets this data, detects the gesture and connects it to the point-cloud system.
Camera → Hand Tracking → Gesture Detection → Point Cloud → Real-Time Response
This creates a direct relationship between physical movement and digital behaviour without requiring a controller.
THE CHALLENGE
The main challenge was stabilizing gesture recognition. Small fluctuations in the tracking data initially caused false fist detections and made the creatures flicker between states.
Through multiple iterations, I introduced thresholds, lag, and filtering to distinguish deliberate gestures from momentary tracking errors. This made the final interaction significantly smoother while maintaining responsiveness.
OUTCOME
The final experience combines computer vision, procedural animation and interaction design into a simple gesture-driven system. Complex technology operates behind the scenes, while the participant experiences something much more intuitive:
You move. The creature responds.
The project demonstrates how computer vision can transform the body itself into an interface for interactive digital environments.
