What does a Neural Net sound like?
In this project, I'm exploring how even a very minimal neural network, trained simply to recognize handwritten digits, can interact meaningfully with sound. Using a basic React frontend, you can draw digits or shapes directly onto the canvas, and these visuals immediately influence the neural network's activations in real-time. The activations are then sent over to SuperCollider using Open Sound Control (OSC) to create simple, ambient sonic responses.
Each neural network layer contributes something different to the sound. The hidden layers produce ambient pads, turning the neural activity into tonal textures, where each neuron's position in the array sets the pitch and its activation strength determines how loud it plays. Meanwhile, the output layer, which shows the digit the network thinks you're drawing, triggers sparse bell tones. It also sends monophonic control voltages to a Serge synthesizer, changing the pitch of the oscillator.
The project's design emphasizes exploration and intuition. Users interact with the neural network by drawing, influencing both visual and auditory outcomes simultaneously. Intriguingly, as drawings become ambiguous, the resulting audio-visual output decreases in intensity, highlighting moments of uncertainty and error within the network. It was fun to observe the network hallucinate numbers from odd shapes and mistake one number for another.
For me, this project is a memory of a good old-fashioned AI, designed to do one simple task: recognize digits, albeit poorly.
Sonification