As part of our ongoing work on ECHO::Canyon, I started exploring methods to use a UDK actor’s Skeletal Mesh, or more specifically the bones that make up that Skeletal Mesh to drive sound and music. Think of it along the lines of digital puppetry, but with each movement or gesture mapped to a sound-generating process using UDKOSC. Using an older-school wired XBox controller, UDKOSC and our custom ‘Valkordia’ Skeletal Mesh from ECHO::Canyon, it wasn’t too difficult to rig up a pretty straightforward example of sonified musical bone data.
Here’s a demo capture of two simple sonifications of the right and left wing-tips bones of the Valkordia wings. The first shows simple sine waves mapped to each wing’s Z coordinate with a slight offset between their base frequencies to make the sines beat just a bit. The second show the Valkordia’s “call” sound modulated slightly by the distance measured between the two wing tips.
Fun to play with and portends many things to come.
To explain what’s going on, this tutorial will step through the process of configuring the Skeletal Mesh controllers, mapping 2-dimensional Joystick data to drive our Inverse Kinematic controller, identifying individual bones in the skeletal mesh in UnrealScript and finally the sonification steps to use UDKOSC and Supercollider to make some sweet, sweet music.
While most of the concepts and examples I’ll cover here aren’t anything too crazy for experienced Unrealscript jockeys, they’ll require a pretty comprehensive understanding of how Unrealscript classes, custom animations and SkeletalMeshes and control systems all fit together. For those reasons, I’d say this tutorial is targeted at medium to high level UDK users. In that light I’ll refer to a number of processes and objects without actually describing each and everyone in detail as that would expand the scope of this post well beyond its already large size. I will however try to hyperlink UDK forum posts and UDN example pages into the text where appropriate.
Also, this may get long so to aid in digestion, I’ll break it down into the following chunks:
- Understanding the Skeletal Mesh
- Our Friend: the SkelControl_CCD_IK
- Conditioning our Input data
- Bone tracking in UDKOSC
- Sonification with Supercollider
After the Break: Understanding the Skeletal Mesh…