MUSIC IN VIRTUAL WORLDS (MvW)
The Music in Virtual Worlds (MvW) research group focuses on the intersections between musical performance/composition and the use of virtual environments as controlling and performative spaces.
Recent research and performance projects include:
ECHO::Canyon – UDKOSC work premiered at the 2013 Music and Games concert @ccrma, 4/25/23013
Play Your Phone! – Evening-length concert at the 2010 MiTo Festival. Two evenings of performances featured new Mixed Reality works built in UDKOSC for piano and virtual performer: Tele-harmonium by Robert Hamilton and Perkussionista by Juan-Pablo Caceres. Art direction for both works by Chris Platz and piano by Chryssie Nanou.
UDKOSC is an implementation of the Unreal3 engine built using the Unreal Development Kit(UDK) with an integrated OscPack Windows dll linked to custom UnrealScript classes.
By linking UDKOSC’s UnrealScript classes to a customized oscpack Open Sound Control c++ implementation, actions and motions taken by virtual actors in the Unreal3 game engine are encapsulated as OSC messages over UDP and used to control dynamic music and sound synthesis systems in a manner similar to q3osc.
UDKOSC takes OSC input as well, allowing external OSC control of Pawns, the player Camera and projectile targeting as well.
UDKOSC is the fourth project in a lineage of integrating virtual worlds and environments with interactive audio and music servers with a common goal of creating shared and distributed musical performance spaces. Like our custom Sirikata implementation for the 2009 MiTo festival, q3osc and Maps & Legends before, UDKOSC combines a game-based virtual world implementation (in this case the Unreal Development Kit or UDK) with customized classes to stream game-data to interactive sound-servers using Open Sound Control messages over UDP. In this manner, gestures, motions and actions from actors in virtual space can be transformed in real-time into control messages for complex audio and musical software systems.
While the UDK offers only free access to the engine’s UnrealScript programming layer (leaving the core C++ engine accessible only through significant financial investment) UnrealScript has a provision for binding Windows Win32 .dll’s to UnrealScript classes, enabling code such as a C++ implementation of Open Sound Control (in this case, Ross Bencina’s OSCPack) to provide bi-directional communication between game engine and sound-server. By then creating mirrored data structures both in UnrealScript classes and in the Windows .dll, data can be passed back and forth between the UDK and any OSC-enabled control system or software, including ChucK, Supercollider, Pure Data (PD) and Max/MSP.
For UDKOSC, a custom Windows oscpack_1_0_2.dll was compiled with specific extern methods and mirrored data structures to communicate with the game engine, both to stream specific game data out as OSC over UDP (including Pawn XYZ coordinate positioning, XYZ coordinate tracking for in-game projectiles, state and XYZ coordinate info for Static Mesh and Interp actor classes) and to send control data from iPad controllers and from the audio-engine itself back into the game to control Global GameInfo parameters such as world Gravity and GameRate, as well as specific positioning information for projecties and static actors.