Smooth Operator

This immersive installation transforms a 6 × 3 m space into a stage where participants create music through body movement. Each person embodies a unique instrument—piano, violin, cello, flute, vibraphone, or bassoon—and their gestures dynamically influence pitch, note duration, and accompanying visuals.

The project is built entirely with web-based technologies, weaving together real-time computer vision, sound synthesis, and generative visuals. HTML and CSS provide the structure and presentation, while JavaScript serves as the foundation connecting all components. Using p5.js, the installation renders the interactive canvas, animations, and visual effects. Pose detection and multi-person tracking are achieved with ml5.js (PoseNet), allowing the system to map human movement directly to sound and graphics. Musical expression is handled by Soundfont.js, which loads realistic instrument libraries and plays notes based on participants’ gestures, while the Web Audio API ensures precise audio timing and responsiveness.

When no participants are present, the installation enters an autonomous “DVD screensaver” mode, where bouncing instrument-colored forms generate both music and visuals on their own. By merging computer vision, sound synthesis, and design, the installation creates a playful, collaborative environment in which collective movement becomes a shared musical performance.