Supervisor: Taha Khan
An interactive media study exploring sound as a primary user interface, real-time audio feature analysis, and machine learning-driven visual environments.
Project
This case study details the concept and technical workflow of Sound → Visual, an interactive media installation that transforms vocal and physical acoustics into dynamic, evolving digital visuals. By bypassing traditional input hardware, such as touchscreens, mice, or gamepads, the system positions sound as an expressive controller. Using real-time audio extraction and machine learning classification, the installation maps distinct acoustic events directly to reactive generative art, exploring alternative paradigms in human-computer interaction (HCI).
The Brief
Design and build an installation that uses audio inputs to control digital visual environments, replacing physical hardware with sound-based interaction powered by machine learning and Open Sound Control (OSC) communication protocols.
What We Did
-
Interaction Design
-
Acoustic Feature Extraction
-
Real-Time Audio Processing
-
Machine Learning Model Training
-
Generative Visual Programming
-
Cross-Software Integration (Processing & Wekinator)
Technology & Architecture
-
Visual & Audio Processing: Processing (Java-based core for audio input analysis and generative visuals).
-
Machine Learning Engine: Wekinator (real-time classification using machine learning models).
-
Communication Protocol: OSC (Open Sound Control for bidirectional data flow between Processing and Wekinator).
-
Acoustic Inputs: Microphone capture analyzing Volume, Low-Frequency Energy, and High-Frequency Energy.
Goal
This project uses machine learning to turn sounds into real-time visual triggers, proving audio can replace traditional physical controllers.
Process
Stage 1 – Sound Capture: A microphone records raw physical inputs (claps, snaps, whistles, loud voice).
Stage 2 – Feature Analysis: Processing extracts numerical audio metrics: volume, low-frequency (bass) energy, and high-frequency (treble) energy.
Stage 3 – ML Classification: Processing sends feature data via OSC to Wekinator, which classifies the audio signature into one of four trained sound categories.
Stage 4 – Visual Trigger: Wekinator routes the classified output back to Processing via OSC to instantly render the corresponding generative visual.
Benefits
1. Hardware-Free Interaction
Removes the friction of physical controllers, allowing users to engage naturally using their body and voice.
2. Intelligent Audio Recognition
Leverages machine learning to differentiate between nuanced acoustic profiles rather than relying purely on simple volume thresholds.
3. Cross-Modal Feedback Loop
Offers immediate, real-time visual reinforcement for acoustic actions, creating a tight perceptual link between sight and sound.
4. Scalable Framework
Provides a flexible, extensible pipeline for future audio-visual experiments, performance installations, or accessible control interfaces.
Key Features
| Sound Input | Acoustic Signature | Visual Response |
|---|---|---|
| Clap | High transient impact, sharp, broad frequency bursts | Particle Burst: Particles suddenly explode outward from a central point. |
| Snap | Brief, crisp high-frequency spike | Geometric Flash: Sharp, angular shapes instantly flash across the canvas. |
| Whistle | Sustained high frequency, narrow tonal spectrum | Flowing Wave: Smooth, continuous wave lines flow horizontally across the screen. |
| Loud Voice | High volume, heavy low-to-mid frequency concentration | Circle Expand: Large circular shapes expand outward and pulse dynamically. |
