Mark's Blog

I made a couple of revisions to Sonify over the last week. Most significantly I replaced GD Library with SDL. Now Sonify can draw decoded audio into a window and resize its output using nearest-neighbor scaling — preserve those pixels!

Note: The screenshot above also shows JackPilot — part of the Jack OS X project.

Sonify uses Pascal Getreuer's colorspace for converting between HSL and RGB color models. Puzzlingly, tests I've run show that the conversions I am doing are a bit more than imprecise. Compare the image above to the two below: the bottom left graphic is the source image; the screenshot above shows the image reconstructed using sine waves in the frequency range of 100-1100 hz; and the bottom right screenshot shows the image reconstructed using square waves over the same frequency range. The two instances of Sonify were launched using sonify sfy color.png 1000 100 10 sin 2 and sonify sfy color.png 1000 100 10 sq 2.

I am guessing the apparent phase-shift error either comes from imprecision in the data types of my variables or propogates through the colorspace conversions; however, I have yet to examine this.

Instead I spent time over the past couple of days scrubbing and commenting Sonify's source files. The code I posted below (including the manner of writing to disk, unused variables, and crap I wrote at 4:00 AM) is just horrible, really. But now anyone can peak into the project, get a clear sense of what's going on, and start hacking — which I hope people do — because...

Sonify is on GitHub! https://github.com/markandrus/Sonify

Killer! I was working on a Haskell program for a lab two weeks ago when I accidentally rm-ed my source code — some form of version control would have been helpful! Now I plan to use git with my future projects and push as many as are worthwhile to GitHub.

I'll leave you with some glitchy images I made while feeding Sonify a wavering synthesizer signal from Ableton Live:

It's really fun to see what kind of patterns emerge out of this system!

Sonify, a plugin for the JACK Audio Connection Kit, iterates pixel-by-pixel through an image, encoding each pixel's hue and luminance values as a waveform of a particular frequency and amplitude. It does this in real time, while simultaneously decoding audio it receives - according to the same rule - into an image file. Sonify makes use of Aubio for pitch-detection and GD Library for image-handling.

The leftmost image above shows the original image that produced the three derivative images. The first is a straight translation of the image using Sonify. The right two images were made by applying an LFO-controlled filter (think dubstep) and heavy reverb, respectively. Color frequency was mapped over 100-1100 hertz, with a time-per-pixel of 10 ms.

Time-per-pixel of 10 ms is relevant since the lower frequency bound is 100 hz, and it takes 10 ms to produce a complete 100 hz cycle. Aubio will have a better chance at detecting the pitch if we can provide it an entire cycle. Below shows the terminal interface to Sonify:

$ ./sonify
usage: sonify <name>
$ ./sonify client1
the sample rate is now 96000/sec
Source Image: cat.png
0-Png, 1-Jpg: 0
Destination Image: cat2.png
0-Png, 1-Jpg: 0
Pitch Scale: 5000
Lower Bounds: 1000
Time-per-pixel (ms): 1
Waveform Type: 0


If you are familiar with the HSL color model, you may have noticed that I don't encode the saturation values of each pixel; I haven't found a meaningful quality of audio to map this to. On a side note, some of the images I have generated suggest a coding error on my part due to the red color casts. Also, I fully acknowledge that the interface is still rough. I haven't been able to install GTK+ on OS X, so I have since stripped the GUI I originally wrote under Fedora.

Also, Sonify used to run a more attractive terminal interface during the processing portion of the code on a pthread. Something about the code disagreed with OS X, so I have also stripped that.

The nine images below show the results of providing Sonify an image of a circle and running it through a number of effects in my favorite DAW, Ableton Live.

I invite you to download, build, and expand on Sonify as you choose. It should be as simple as running make in the source folder. You may download the most recently confirmed source that I used for the above, or you may try to compile the GTK+ version that I haven't touched in ages.

• sonify-tested-works.zip
• sonify-gtk+-untested.zip

If you wish to contact me for questions or help, mark.andrus.roberts at gmail.com. Next up, I think it would be fun to write code (perhaps Haskell) that generates sounds based on an intended visual effect (perhaps generating geometry) when passed to Sonify.

I started the code that would ultimately generate the video below on Christmas, 2009, at my cousin's home, and I have made subsequent revisions since. For example, in September I needed to spawn particles based on an initial mask, such as text. Most recently I rewrote the particle class to record particles' x and y position in terms of t time.

My animation code, written in Java, spits out a sequence of PNG images, i.e. frame000.png frame001.png ... frame899n.png For a while I used Ffmpeg to convert these sequences into AVI files:

ffmpeg -r 30 -i frame%03.png out.avi

So 900 frames of 1080p video would generate a fat 30 second AVI.

When I chose to combine these sequences by hand, I started compositing them in After Effects.

I doubt H264 can ever really encode these in an attractive way that loads quickly. Case in point, the above video consistently loads painfully slow for me (at least at 720p).

Also, I've been debating whether or not to release this code, since I consider it more of a tool for my own video art production. But I decided why not release the code I have been hacking at for a while now. It's not clean or elegant because I don't care for it to be right now, but if you want to play with it have at it: aag.zip. You will also need:

• Filters from JH Labs.
• Parallel Java Library from RIT.

Good luck.

What is iPodLinux? Before the iPod Touch, iPhone, Android, and others, iPL offered iPod owners a chance to run homebrew code on their portable devices. Since then iPodLinux.org has expired, and the project is practically dead. Nevertheless it was a fun time when you could load the entire Wikipedia on your iPod or play Doom.

Below is a screenshot of Podzilla2, a graphical shell for iPodLinux, running my module Drum Machine. Drum Machine is coded in C, and provides 4 samples, 4 patterns, and 32 steps. Like other Podzilla modules, it is generally compiled into the main executable along with other modules.

The screenshots show Podzilla2, running under OS X. I originally wrote this under Fedora around October 2008, but I rediscovered the source code and built it for the desktop. You may download the source code here: xpod-drum_machine-src.tar.gz.

Note: Building this may be tricky. Building Podzilla2 itself may be tricky since the documentation on it is scattered across the net, but the links below should suffice.

• Build Podzilla2 following the instructions at http://ipl.derpapst.eu/wiki/Building_Podzilla.
• tar xzvf xpod-drum_machine-src.tar.gz in the modules directory of the Podzilla2 source.
• Attempt to rebuild Podzilla2 with Drum Machine
• Notice all the errors and attempt solutions
• Successfully build Podzilla2 with Drum Machine

So prepare to hack. I haven't really bothered with this since high school, but it will still work with some effort.

This is what happens when you don't ground the video signal coming from your Propeller Proto Board:

I will get this resolved soon so that I can move forward on my A/V project. The dorkbotSF list directed me to Parallax's Propeller chip, which I found attractive both for its price and the amount of code already written for it (specifically for NTSC video signal generation). I am not running Windows, so I don't use Parallax's Propeller Tool; instead, there is a very capable alternative (with a few of its own bonuses) called Brad's Spin Tool, or BST. Oh, and I'll need to learn some assembly over the next few weeks...

I was asked to respond to a question in my Haskell class today. We defined a type alias for a 2-tuple of Doubles named Complex, and a function cmul that accepts Complex arguments:

type Complex = (Double, Double) -- (real-part, imaginary-part)
cmul (ra, ia) (rb, ib) = (ra * rb - ia * ib, ra * ib + ia * rb)

one, two, i :: Complex
one = (1, 0)
two = (2, 0)
i = (0, 1)

> cmul i i
(1.0, 1.0)
> cmul (1, 1) (1,)
(0, 2)

My instructor asked a relatively simple question: "There is something a little odd about the last output. Why isn't it (0.0, 2.0)?". As a newcomer to Haskell this took me a little investigation before I had an answer. My instructor offered the hint, "What exactly is the type of cmul?".

> :t cmul
cmul :: (Num t) => (t, t) -> (t, t) -> (t, t)

The portion (Num t) => in the type definition above restricts t to the typeclass Num. We may pass cmul Complex types because they match the form expected, (Num t) => (t, t). Doubles are within the typeclass Num.

> :t (1, 1)
(1, 1) :: (Num t, Num t1) => (t, t1)

So (1, 1) is neither Integer nor Double, rather it is expressly Num; It is a polymorphic constant which will morph into any number type as required of the function operating on it. See Type Basics II in the Haskell WikiBook.

cmul (ra, ia) (rb, ib) = (ra * rb - ia * ib, ra * ib + ia * rb)

So when (1, 1) is passed to the function above, Haskell uses Integer multiplication, addition, and subtraction, whereas when passed a (Double, Double) or Complex type it uses Fractional operations. We can verify this with the following in GHCi:

> let x = cmul (1, 1) (1, 1)
> :t x
x :: (Integer, Integer)
> let y = cmul one one
> :t y
y :: (Double, Double)

Perhaps this was helpful to someone! Get started programming with The Haskell Platform.

This is my first blog post. Surely I will have to switch to a "real" system soon, but right now I just edit all of this from vim and upload to my host.