Clock it – Interactive Audiovisual System

Supervisor: Taha Khan

Clock it –  Interactive Audiovisual System

An interactive audiovisual system transforming physical hand gestures into real-time digital controls.

Project

Clock is an interactive audiovisual control system developed in TouchDesigner that converts webcam-captured hand gestures into precise digital control signals. By executing real-time spatial tracking via MediaPipe and applying signal conditioning through custom Math CHOP networks, the platform translates mid-air gestures into dynamic visual distortion, parameter modulation, and continuous spatial audio control without physical interface hardware.

The Brief

Design and deploy a low-latency, touchless interaction architecture that bridges physical gestural input with multi-sensory generative output, providing intuitive spatial control over complex visual compositions and audio environments.

What We Did

Interaction Architecture, Computer Vision Engineering, Signal Conditioning & Data Normalization, Generative Shader Programming, Real-Time Audio Synthesis & Spatial Mapping.

Technology

TouchDesigner, MediaPipe Hand Tracking Engine, Custom Math CHOP Operators, Advanced TOP Processing Pipelines (Displace, Switch, RGB Key, Transform, Ramp), Multi-Channel Audio CHOP Framework.

Goal

Establish an intuitive, hardware free gesture interface that translates spatial movement, finger proximity, and wrist rotation into continuous high-precision control streams for unified audiovisual modulation.

Process

1. Vision Capture & Tracking Infrastructure:

Configured a live USB video stream within TouchDesigner, establishing the baseline capture pipeline. Integrated the MediaPipe Hand Tracking operator to extract spatial landmark data including 3D coordinate matrices for fingertips, knuckles, and wrist joints from the incoming video stream in real time.

2. Signal Isolation & Data Conditioning:

Isolated specific positional channels from raw tracking matrices. Processed these coordinate

streams through dedicated Math CHOP pipelines to adjust gain, eliminate tracking anomalies, and normalize raw spatial values into predictable, bounded $0.0 – 1.0$ control arrays.

3. Gestural Logic & Threshold Engineering:

Developed mathematical constraints to differentiate intentional performance movements from ambient micro-jitter. Calculated real-time Euclidean distances (e.g., pinch gaps, finger extensions) and wrist rotation vectors to establish discrete trigger states and smooth continuous controllers.

4. Generative Visual Pipeline Architecture:

Engineered multi-layered visual compositions utilizing TouchDesigner TOP networks (Displace, Multiply, RGB Key, Transform, and Ramp). Synthesized dynamic wave patterns, ripple effects, and video compositing layers structured for live parameter binding.

5. Dynamic Visual Parameter Modulation:

Bound normalized gesture channels directly to visual operator parameters. Hand spatial coordinates dynamically modulated displacement scales, visual feedback distortion, keying thresholds, and scene transitions via Switch TOP networks.

6. Parallel Audio Processing Architecture:

Constructed a multi-channel audio processing pipeline handling organic sound layers (environmental wind, rain, and wave samples). Embedded CHOP-based audio filters, volume envelopes, and dynamic crossfaders to build a highly responsive sonic environment.

7. Synchronized Audiovisual Mapping:

Mapped processed hand-tracking signals simultaneously to visual and audio parameters. Physical gestures drove synchronized modulation across visual effects and audio filter cutoffs, achieving unified crossfading and real-time multi-sensory feedback.

8. System Integration & Performance Optimization:

Consolidated input tracking, signal conditioning, gesture detection logic, generative rendering, and audio synthesis into a unified TouchDesigner network architecture. Iteratively calibrated Math CHOP gain ranges and filter response curves to guarantee zero-latency execution.

Benefits

• Hardware-Free Spatial Control: Eliminates physical keyboards, mice, and MIDI controllers in favor of intuitive, mid-air gestural performance.
• Tactile Responsiveness: Direct parameter mapping links movement velocity and rotational orientation to immediate visual and acoustic output.
• Controlled Interaction Precision: Advanced Math CHOP filtering eliminates tracking jitter, preventing unintentional gesture triggers during performance.
• Unified Multi-Sensory Synthesis: Simultaneously controls generative visual networks and multi-channel audio environments through a single gestural pipeline.

Features

• Spatial Landmark Extraction: Tracks 3D coordinate data across fingertips, knuckles, and wrist joints via standard webcam input.
• Math CHOP Signal Conditioning: Dynamically scales, clamps, and normalizes spatial coordinates into precise control parameters.
• Gesture-Driven Scene Switching: Executes state transitions across complex visual compositions based on hand orientation and proximity vectors.
• Synchronized Audiovisual Modulation: Simultaneously drives displacement TOP parameters and audio filter CHOP cutoffs in real time.