I decided to archive this blog under a more descriptive and specific name, rather than “Audio Programming Blog”, for people to know what they are reading, in the future.
And by archive, I mean no more updates on this blog, since the project is already completed.
It will be archived under a new link, which is gameaudiomiddleware.tumblr.com.
If anyone currently reading this, is unfamiliar with what this blog is, please start from the very first post!
Thank you and looking forward for future projects!
I had the honor of giving an interview to Percussa (the AudioCubes team from Belgium) about my game audio middleware prototype, video game music, tech talk, life etc. Feel free to check it out!
The interview can be found here: http://t.co/azsoGujV
Project on Cycling ‘74!
It sure has been a while. I hope you have enjoyed reading this blog as well as I have enjoyed writing it. Just wanted to let everybody know that the project is now on Cycling ‘74’s Project page, and you can find it here: cycling74.com/project/talktome-adaptivedynamic-audio-prototyping-for-video-games/
Again, thank you for joining me in this journey and looking forward to update this blog with future long-term projects!
Thank you and have a good day!
And as promised, this is the demo presentation I showcased at Berklee College of Music. It was amazing and people enjoyed it quite a bit.
Also, as promised, the entire project has been submitted tonight in Cycling 74’s Project section and everything will be available to you in a week.
Thank you all for joining me in this amazing journey and I hope you enjoyed it as much as I did!
cdesk said: I have a concept to take the output of two separate Unity games on two separate computers and feed that info to Max/MSP to create a generative sound design. I am in the process of testing Max/MSP for the first time to see if it is within my skill set, and since you have been directly working with a similar implementation I was wondering if based on your experience, you think it is technically feasible.
If I understand correctly, you want to have 2 separate computers that feed info to a third computer that runs Max/MSP…
In this case, absolutely, it is feasible. The only differences between my project and yours is that:
- you will not be using localhost IP anymore (127.0.0.1) which is the default IP for your computer.
- you will have to be connected to the internet or probably to a wireless network and find the net IPs of your 3 computers.
- From there on, you have to change the send IP of the UDPOut script, designate 2 new (different from each other) ports, like 32000 for the first game and 32001 for the second game.
- and of course, to create 2 different UPD receivers in Max/MSP which they have their own IP address in the [mxj net.udp.recv] object, as well as their designate port.
As soon as you do that, just take your time and enjoy making your generative sound design system, as you’ll have nothing to worry about, regarding the UPD network and data transfer.
If you’re looking into heavy data flow transfer from the Unity games to Max, you might want to look into TCP/IP instead of UDP. Almost similar process to UDP, but more reliable data transfer. But the downside of it is that it’s a bit more network-heavyweight.
Inside Unity3D #3: Creating The Energy Levels
So I am pretty sure that this is the last blog update of this project, as I will be publicly showcasing this tool in 2 days from now and I will release all materials on Cycling ‘74’s project page, as well as my presentation video here in Tumblr, so that’s exciting!
But… we are still not done, so let’s not get too comfy! In this last post, we will talk about how to create the energy levels through Unity3D and have Max/MSP respond to these messages with music that corresponds to the gameplay.
Now, as I mentioned in my previous update, since this is a personal prototyping attempt, some examples might not apply to your project or need further modification, it’s just to have a general insight on how to go about creating your own tool and have it create music instances based on your own personal preference.
Alright, so having said that, the goal here is to get messages in “val1" route in Max, which is responsible for the energy level.
Before we start talking about examples, let’s create our master energy level, called EnergyLevel, that’ll take all incoming messages that are pointed as EnergyLevel.sendLevelToEnergy:
Again, just like in the MaxLaunch script explained in the previous post, we’re connected through UDP and the variable sendValueToEnergy is the one responsible for getting all info in “val1” in Max/MSP.
From the moment we established that, let’s talk about some examples you may consider for your project:
Example #1: Environmental music:
A good way to create dynamic and non-linear musical instances within Unity3D is to make environmental or room music: the music will change when the player walks in a different room of importance. Here’s how you do it:
Create an empty gameObject on your project and place it in front of a door, like so:
Awesome. Now give it a meaningful name, like _Location1 or _LocationScript1. Now, let’s attach it this script, called specificlocation:
Very simple script. What that does is that it opens 2 values in the object inspector’s GUI, which should be preferably named as “Minimum Distance" and "Value Sent To Max”.
From there on, you set a minimum distance of proximity between the player and the object and the value it’ll trigger upon proximity.
Now, this object will not do anything to Max/MSP, as is. Why? Because it has not been linked with the EnergyLevel script in any way, and also, what’ll happen if something* happens momentarily and as soon as it stops, you want to go back to the previously called environmental music? You need some sort of a buffer or… a clipboard.
*Remember that something for later.
That’s why I implemented this script right here, called motherSpecificLocation:
This script is responsible for gathering all gameObjects (assuming that you’re using multiple ones) that have the specificlocation script attached to it and save the last called “specificlocation” value, whatever that is.
Now, let’s attach it to a new gameObject, preferably named _MasterLocation or something.
Now, motherSpecificLocation has 2 very important static variables open for you in the object inspector’s GUI: the amount of the gameObjects used and the Player slot. First, drag the “Player” category (that is located in the Hierarchy section) and put it in the Player slot. We did this because we want all changes to be based on the player’s coordinates and movement towards the location objects.
After you do that, open up the Specific Game Objects static variable and set the amount of gameObjects you’ll be using as location objects in the Size sub-variable and then drag, for eg., _Location1, _Location2, _Location3 etc. to the appropriate element slot in the list.
Essentially, all location objects that have the specificlocation script attached will be automatically reporting to motherSpecificLocation script for any actions taken.
Example #2: Enemy Proximity/Destruction:
Now that we have our location triggers throughout the game, it’s time to talk about what happens if something happens after you triggered a location script and after that something is over, you want to restore the previously called location value.
In our case, that something is the enemies.
Let’s make a hypothetical scenario. Let’s say you start of with a “0" value for energy, that represents the utmost state of peace in the game: no action whatsoever.
After a while, you encounter a group of enemies. Logically, the music’s intensity will increase from the previously called “0”.
Here’s how you do that. Let’s create a new script, called scriptforenemy:
Now, attach this script to all enemies in the game. (assuming you want to enable enemy proximity music for all enemies.) Now, it’s very important to attach the “Player” (again, from the Hierarchy) to the Player static variable that we opened through this script, so enemy proximity will be depended on the player’s current position and how close his position is to the enemy.
From there on, set the minimum distance of proximity activation and the value you want to trigger through proximity. For example, if you set “75”, Max/MSP will receive “75” in “val1” which is responsible for the energy level, so in other words, you will have energy level 4 in the middleware.
Now, proximity was the first part of this example. What happens after you destroy the enemies? You’d normally want to restore the previously called environmental music.
Well, in my case, enemy destruction was not so simple. After digging hard in the project, I had to change a few parameters in the existing scripts, so here’s how you officially declare an enemy destruction:
If you’ll be using my material for cross-reference (they’ll be uploaded on Thursday, 3rd of May.) then we have to open up the Health script, that is attached to all enemies. (if you change one, they all change unilaterally.)
Once you open it, go to the Awake function and change it to this:
For some reason, it was enabled = false by default, which does not make any sense at all, but moving on.
Then, on the onDamage function, scroll down a little till you find the “if (health <= 0)" which signifies the death of the enemy.
Change the lines below to this:
What that little modification in the script does is that it recalls the saved value of environmental music in motherSpecificLocation and sends that to EnergyLevel, as well as resetting the enemy proximity to 0. (since the enemy… does not exist anymore.)
These are two fundamental examples that a lot of games of non-linear gameplay, use for interactive and dynamic music. In the end of the day, these are the ones I had to use for this specific game, but let’s say, for an art game, other examples might apply better, as well as omitting a lot of steps from here, but hopefully, this blog might have helped you get acquainted with a few techniques and approaches to interactive music.
This is the hardest part where I have to announce that this was the last technical update for this project. It is hard because I enjoyed every moment of writing these documentations, as I’ve also enjoyed the attention it has received from fellow composers and sound artists, as well as game developers and designers.
After this post, I will post final presentation in video form this Thursday (3rd of May), to give you a quick demo of what that tool we’ve created does and maybe next week, I might also upload a live presentation I will do of my thesis work. (if I get on hold of the video recording they’ll do.)
I would like to thank everybody that has taken the time to read some of this blog or the entire thing and also, everybody who has supported me in the process of the project’s development. Again, if you have any questions on the project, please feel free to ask me by click “Ask me anything” on the top of the page.
Thank you everybody for reading and I’ll see you soon!
Inside Unity3D #2: Launching Max/MSP
Alright, so having established our UDP network, it’s time to talk about our first parameter that is responsible for launching Max/MSP upon Unity3D’s startup.
If you remember from the “UDP in Max” post, in the [route] object, there were two values: val0 and val1.
Today, we will talk about val0 that connects to Max’s master toggle, in two parts.
First, create a script a descriptive name, like MaxLaunch in this case and write the following:
Then, create an empty gameObject, preferably called _MaxLaunch and attach the MaxLaunch script to this gameObject.
Now, the public string routeVal = “MaxRouteValue” will allow you to make a user-input in object inspector’s GUI, like so:
As seen in the picture, we assign that RouteVal to val0.
This code works upon Unity’s startup, as the routing process happens in the script’s main function.
Simple as that.
Now, we have a designated script that sends the 0s and the 1s in val0, let’s call it gameStart:
Again, attach this script to a new empty gameObject, preferably called _gameStart, so it exists somewhere in the project.
When the game’s “awake”, the scripts sends a value of 1 to the MaxLaunch script. Now, if playing the game through the Unity’s built-in Preview, then if the player presses the letter Q, (as in Quit) it sends a value of 0 to the MacLaunch script. If playing a standalone build of the game, then if the application closes, then it also sends a 0 automatically.
That’s all, simple as that. There is nothing else to be mentioned here really, since it’s just a simple binary mechanism of sending 0s and 1s conditionally, and well, that’s how you set it.
Next up, I will write out and post some examples on how to play with Max/MSP’s second parameter, val1 which is responsible for the energy level of the music, but just to mention it in advance, these examples won’t work with all projects. They are personalized examples that fit this game’s music and to an extend, changes some of the game’s functions itself, in order to create dynamic/adaptive music. So in other words, I’ll post them to give you a general insight of things you can do with it.
Thanks everybody for reading and see you next time!
Inside Unity3D #1: UDP Network
Important notice: If you are new to reading this blog and have no idea where to start, feel free to head back to the first blog post.
Hi folks and welcome back to another (and probably short) update of this R&D blog! Last time, we officially finished with the Max/MSP portion of this project and from now on, I’ll be posting things regarding only Unity3D and its preparation for proper communication and above all, to create a mutual and musical understanding between the two softwares.
Before we get started, a few important points that ought to be mentioned:
- I worked on version 3.5.0f5.
- It’s very important to go to File ➔ Build Settings… and make sure to have “PC and Mac Standalone” in Platform selected. This is a very important part, as most Unity versions have Web Player selected as default, and some of the libraries used in this project seem to conflict with the Web Player’s build settings libraries. At least, that what I got from trial and error.
- It’s a good idea to create a custom scripts folder in your Scripts asset folder, like I have: “Yiannis’s Custom Scripts”. Being organized will take you a long way!
Alright, so with no further due, let’s dig in!
So this is the script, written in C#, that allows the UDP network to happen. I should probably mention at this point that I am still a learning programmer and I had to modify a script that I found on the Unity forums.
For the most part, I had to focus on two spots: Making sure that the UDP network uses the same local IP and the same port number (again, this only applies in one-way communication) and then, making sure that values go in Max.
As for modifying this script, I wouldn’t really suggest that, as it’s a static script: it just needs to somewhere in the entire Unity project and it will automatically work itself on launch.
After we finished creating the script, create an empty gameObject and attach the script to this object. Doesn’t matter where the gameObject lays, as long as it’s in the project, that’s fine.
It’s also a good idea to create meaningful names for all of our (current and forthcoming) gameObjects that we’ll be using, like: _UDPOut, _ABCfunction etc. Underscores in the name make the object appear in the top of the assets list.
Now, this script alone won’t do anything; In the next post, we’ll talk about user-input scripts and how to route them to Max/MSP.
Since this update was a short one, I’ll make sure to post a new one sooner than usual.
Thanks for reading and again, if you have any questions, regarding the project and if other people can benefit from these questions, I will be posting them here, along with the answer. Just click the “Ask me anything” link on the top of the blog.
Thanks everybody for reading and happy afternoon!
Questions so far?
In the meantime, I just enabled the “Ask me” option so if you have any good questions relating the project that would benefit you and everybody else, I will publish them on my blog, along with the answer.
So ask away!
Creative Preparations and Considerations in Max/MSP #3: UDP Network
And finally, the final and most important update for the Max portion, before we move to Unity3D: establishing a UDP network.
This is the abstraction that allows everything to start and stop accordingly, depending on what’s going on in the game.
Now, I have a lot of people asking me how come I chose the UDP network, instead of the TCP/IP network. (following Virginia Tech’s example with [muy]). The answer is simple: It didn’t really matter, to begin with. The project, as is, does not send an intense amount of data to Max in a short time, so with either ways, the data will reach the Max end, safe and sound. The only difference is that TCP/IP needs an extra protocol establishment footwork, in order to work.
To put things plainly:
UDP makes sure to have data barging in and out. In a small fragment of time, there’s a chance that data can get lost and never delivered to the receiver: that’s the cost of using UDP network.
TCP/IP, on the other hand, has a startup routine: It looks left and right, to see if both sender and recipient are ready for data transferring. If so, the data transferring happens periodically and systematically and TCP/IP gives permission for things to come and go accordingly.
Both options sound truly appealing, but in the end, for my project, I was looking for the most network-lightweight solution, which is UDP. UDP does not need any maintenance or any special preparations: As soon as it “sees” a common open port, it automatically opens.
So given that you understood the difference, let’s see how it works:
- I am using the Java version of the [udpreceive] object, called [mxj net.udp.recv].
- The attribute “@port 7470” means that the object reserves the port 7470. Now, if it’s a one-way communication (Unity ➔ Max/MSP or vice versa) then both softwares should share the same port number. (like in this project - more on this later.) If there’s to be a two-way communication, then if the sender’s port is 7470, the receiver’s should be 7471. Not really relevant to this project though, but worth mentioning.
- You can easily assign a different port number, just make sure that this port is not already taken by default! (best advised to search for a Windows / Mac port chart.)
- Max/MSP’s default IP is your computer’s, 127.0.0.1 or as knows as “localhost”.
- So [mxj net.udp.recv] imports all messages from Unity.
- [iter] literally “breaks” the joined messages to single-message strings. For instance, if a message comes like “0 27 bob 52”, [iter] will break them into “0” “27” “bob” “52”. For this project, this is an optional step to follow, as there can’t really be any data overflow and I’ve made sure in Unity to send messages in time, but it’s still a great way to filter your messages.
- [sel :], [t b] and [zl group X] is also part of the iteration process.
- Now, focusing on the object called [route val0 val1].
- A small disclaimer: This will make much more sense after we’ve been through the Unity stage.
- With this object, Max makes sure, through our UDP object, that it receives int values named “val0” and “val1” from the port assigned to both softwares, so in other words, from Unity. (much more on this later)
- For my project’s purpose, I needed just two values: val0 is responsible for launching / stopping the patch, and val1 to control the energy level.
- You can have more than two, it’s limitless really. Just make sure to also change the [zl group]’s argument.
IMPORTANT EDIT - 04/23/2012 : Very important to also have [change] object between your [route] object and your [int] objects! If your Unity project updates, for example, “1” every 1 second, Max will filter out any repetitions of the same number.
And with explaining how the UDP network works, we have officially concluded all updates regarding the Max portion of this project.
From now on, we will move to the second part of this project, which is Unity3D and how to implement our Max/MSP middleware onto Unity3D.
Next up, we’ll start with the fundamental script that’s responsible for the UDP establishment on Unity.
I hope you have enjoyed the journey so far and I hope for everybody to be still interested in this R&D blog, if not, more!
Thanks everybody and happy afternoon!