Sonify, a plugin for the JACK Audio Connection Kit, iterates pixel-by-pixel through an image, encoding each pixel’s hue and luminance values as a waveform of a particular frequency and amplitude. It does this in real time, while simultaneously decoding audio it receives–according to the same rule–into an image file. Sonify makes use of Aubio for pitch-detection and GD Library for image-handling.
The leftmost image above shows the original image that produced the three derivative images. The first is a straight translation of the image using Sonify. The right two images were made by applying an LFO-controlled filter (think dubstep) and heavy reverb, respectively. Color frequency was mapped over 100-1100 Hz, with a time-per-pixel of 10 ms.
Time-per-pixel of 10 ms is relevant since the lower frequency bound is 100 Hz, and it takes 10 ms to produce a complete 100 Hz cycle. Aubio will have a better chance at detecting the pitch if we can provide it an entire cycle. Below shows the terminal interface to Sonify:
$ ./sonify
usage: sonify <name>
$ ./sonify client1
the sample rate is now 96000/sec
Source Image: cat.png
0-Png, 1-Jpg: 0
Destination Image: cat2.png
0-Png, 1-Jpg: 0
Pitch Scale: 5000
Lower Bounds: 1000
Time-per-pixel (ms): 1
Waveform Type: 0
If you are familiar with the HSL color model, you may have noticed that I don’t encode the saturation values of each pixel; I haven’t found a meaningful quality of audio to map this to. On a side note, some of the images I have generated suggest a coding error on my part due to the red color casts. Also, I fully acknowledge that the interface is still rough. I haven’t been able to install GTK+ on OS X, so I have since stripped the GUI I originally wrote under Fedora.
Also, Sonify used to run a more attractive terminal interface during the processing portion of the code on a pthread. Something about the code disagreed with OS X, so I have also stripped that.
The nine images below show the results of providing Sonify an image of a circle and running it through a number of effects in my favorite DAW, Ableton Live.
I invite you to download, build, and expand on Sonify as you choose. It should be as simple as running make in the source folder. You may download the most recently confirmed source that I used for the above, or you may try to compile the GTK+ version that I haven’t touched in ages.
Next up, I think it would be fun to write code (perhaps Haskell) that generates sounds based on an intended visual effect (perhaps generating geometry) when passed to Sonify.