Sonifying UDK Skeletal Mesh Components with UDKOSC and Supercollider

5. Sonification with SuperCollider

OSC output from UDKOSC can be parsed and processed by any software or language that can understand custom OSC namespaces, including SuperCollider, ChucK, Pure Data or Max/MSP. That being said, the sound servers we’re building for ECHO::Canyon and other recent UDKOSC projects tend to use SuperCollider. The main reason SC has been so successful for us to date is that it scales well as new sound objects are spawned and tracked individually across a multi-channel/ambisonic soundfield. I’ll talk through a basic example what we’re doing for ECHO::Canyon’s wing tracking here but similar sonification could be carried out in other languages with vastly different approaches just as well.

Our first wing sonification in the video clip above is a simple mapping of wing-bone Local z-coordinate to the frequency of a sine wave. One sine wave is attached to each wing and for the demo’s sake, they’re each hard panned to their own respective stereo output channel.

Here are the two simple SuperCollider SynthDefs used in this clip. Each one has a central frequency (here 440 and 442 Hz) set slightly apart to promote a nice beating effect while the Valkordia is flying around.

SynthDef("wingRight", {
	arg freq = 442, amp=0.01, gate= 1, atk= 0.01, rel= 0.25, gain=1.0;
	var env, source;
	env=, 1, rel), gate, doneAction:2);
	source =, 0, env*amp);[1],gain * source);

SynthDef("wingLeft", {
	arg freq = 440, amp=0.01, gate= 1, atk= 0.01, rel= 0.25, gain=1.0;
	var env, source;
	env=, 1, rel), gate, doneAction:2);
	source =, 0, env*amp);[0],gain * source);

In SuperCollider, we built a custom class (~udkosc) to represent the current game-state (i.e. an uber-object containing references to everything) as well as a custom player class (~osc_player) containing data structures and methods that act on each player. As new ECHO::Canyon performers enter the UDK game-space and OSC messages from their Pawns are received in SuperCollider for the first time, an instance of our custom Player object for that UDK Pawn is instantiated and populated with position, rotation and other data from the OSC Stream. Player IDs are generated in UDK and sent in each OSC message to aid in correctly routing all that data when there are multiple Pawns moving around. In this example, synths for our wings are instantiated and attached to the currentPlayer:

currentPlayer = ~osc_player.copy;

currentPlayer.wingRightSynth ="wingRight");
currentPlayer.wingLeftSynth ="wingLeft");

An OSC message from UDKOSC for an individual Pawn’s output generally looks something like this:

/player 256, 918.78692626953, -416.78961181641, 9946.888671875, F, 61376, 5073, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 

In this message, the “/player” namespace tells us that the message is taken from a human-controller Pawn, its unique ID is 256, X = 918.78, Y = -416.78, etc. A more detailed breakdown of OSC data from this message:

int32 OSC Namespace
float32 Player unique id
float32 Pawn World Space X Coordinate
float32 Pawn World Space Y Coordinate
float32 Pawn World Space Z Coordinate
bool “Crouch” boolean
float32 Pawn Rotation: Pitch
float32 Pawn Rotation: Yaw
float32 Pawn Rotation: Roll
float32 View Rotation: Camera’s Pitch
float32 View Rotation: Camera’s Yaw
float32 View Rotation: Camera’s Roll
float32 Bone 1: Local Xcoordinate
float32 Bone 1: Local Y coordinate
float32 Bone 1: Local Z coordinate
float32 Bone 2: Local Xcoordinate
float32 Bone 2: Local Y coordinate
float32 Bone 2: Local Z coordinate

To process incoming OSC messages within SuperCollider we create an OSC Responder object which listens for OSC messages on a given Port. All messages that arrive on that port will be dumped into an array (here called “msg”):

~osc_player_responder = OSCFunc({ arg msg, time, addr, recvPort;

OSC messages are currently spewed out of UDK as fast as possible, essentially one per tick (or as fast as the UDK -> DLLBind pipeline will allow). As OSC “/player” messages arrive at the SuperCollider client, each parameter is passed immediately to individual parameters of the current player’s (identified by the unique player ID from UDK) object. In SuperCollider we can either use a function to write all these values to our player instance or set them one by one:

// Set position, rotation, and bone data all in one fell swoop
~udkosc.players[currentIndex].setPosition(msg[2], msg[3], msg[4], msg[9], msg[10], msg[11], msg[13], msg[14], msg[15], msg[16], msg[17], msg[18]);
// Set bone parameters explicitly on the currentPlayer
currentPlayer.bone_1_x = msg[13];
currentPlayer.bone_1_y = msg[14];
currentPlayer.bone_1_z = msg[15];
currentPlayer.bone_2_x = msg[16];
currentPlayer.bone_2_y = msg[17];
currentPlayer.bone_2_z = msg[18];

Once all that data is written into the client, we have the current state of that player in UDK available in SuperCollider. As this is happening many times per second (when blasting packets over a local gigabit ethernet connection), we have a near-continuous data stream to tap as our control data. It’s then simple to take one of those parameters, here the Local Z coordinate of each wing bone, and map it to the Left and Right wingSynths we declared earlier. Below we’re adding the variation of wing height (only +- 20 or so UU) to our starting frequencies (set with a slight offset as mentioned before) and voila! we have sonified wings!

~udkosc.players[currentIndex].wingLeftSynth.set(\freq, (440+~udkosc.players[currentIndex].bone_1_z));
~udkosc.players[currentIndex].wingRightSynth.set(\freq, (442+~udkosc.players[currentIndex].bone_2_z));

This is a super simple mapping schema with one of the simplest synthesis-based sounds around, but when coupled with our manual wing control rig, all of a sudden we have a Theremin-like instrument. Out of bird… wings… Awesome.

Tagged with: ,
Posted in UDKOSC

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: