Scripting Example: Llbacks.Html

The FMOD & Unity Essentials Course Module 6 - Lesson 3 FMOD Callback Events

Resources: Examples: Witcher 3: Blood and Wine Implementation Talk: https://www.youtube.com/watch?v=aLq0NKs3H-k Event Callbacks FMOD Scripting Example: https://www.fmod.com/resources/documentation-unity?version=2.1&page=examples-timeline-ca llbacks.html

System.Runtime.InteropServices: https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices?view=net-5.0 GCHandle: https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.gchandle?view=net- 5.0 StructLayout(LayoutKind.Sequential): https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.structlayoutattribute ?view=net-5.0 Marshal: https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshal?view=net-5 .0

Concepts: Managed vs Unmanaged Memory: - https://docs.microsoft.com/en-us/dotnet/standard/automatic-memory-management - https://engineering.hexacta.com/memory-a-forgotten-topic-355088e2f2c1 - https://www.red-gate.com/products/dotnet-development/ants-memory-profiler/solving-me mory-problems/understanding-and-troubleshooting-unmanaged-memory Garbage Collection and Automatic Garbage Collection: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals What is a Runtime Framework (.NET): https://dotnet.microsoft.com/learn/dotnet/what-is-dotnet-framework FMOD threads and asynchronicity: https://fmod.com/resources/documentation-api?version=2.1&page=white-papers-threads.html https://fmod.com/resources/documentation-api?version=2.1&page=white-papers-studio-threads. html Singleton Pattern: http://wiki.unity3d.com/index.php/Singleton

Unity Physics: https://docs.unity3d.com/Manual/PhysicsSection.html Triggers and Colliders: https://learn.unity.com/tutorial/physics-interactions-colliders-and-triggers-2019-3# Layers: https://docs.unity3d.com/Manual/Layers.html LayerMasks: https://docs.unity3d.com/ScriptReference/LayerMask.html

C# Comparison Operations: Logical Operators: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/boolean-logical-o perators Bitwise Operators: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/bitwise-and-shift- operators

Terms: "this" keyword: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/this Event delegates: https://learn.unity.com/tutorial/events-uh#5c894782edbc2a1410355442 Blittable types: https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types Pointers (in C): https://www.educative.io/edpresso/what-are-pointers-in-c IntPtr: https://docs.microsoft.com/en-us/dotnet/api/system.intptr?view=net-5.0 OnGUI: https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnGUI.html System.String.Format(): https://docs.microsoft.com/en-us/dotnet/api/system.string.format?view=net-5.0

Unity Rendering: https://unity3d.com/real-time-rendering-3d Post Processing: https://docs.unity3d.com/Manual/PostProcessingOverview.html Cinemachine: https://docs.unity3d.com/Packages/[email protected]/manual/index.html

______Intro - 0:00 Henry:

Okay now that we’ve looked at both Delegates and Return Functions, it’s time to look at using those for something more FMOD specific.

We’re going to start by looking at FMOD Callbacks, however I’m not going to be the one teaching you this lesson. A good friend of mine, Colin, is going to be the instructor for you and as you’re about to see, he really knows his stuff as you’re about to see!

He really understands the sort of “backend” and these “deeper systems” that FMOD uses for both Callbacks and for Programmer Instruments which is the topic of the lesson after this.

I’ve just watched his lesson that he has created for Callbacks, it’s very impressive.

The one thing I wanted to say is that I don’t want you guys to worry too much about learning every single specific system, tool and thing that Colin points out in this lesson.

The main thing is we just want you to get confident with using these tools, so Colin is going to run you through them all, and show you how you can create a really cool system where your game responds to your music, and we can change the color of our camera, just by reading our FMOD event.

So I’m going to leave you in the capable hands of Colin.

As always, if you have any issues our Discord is always available to you, or you can email me of course.

And I hope you enjoy!

Colin:

Hi, I’m Colin Vandervort, also known as ColinVAudio, and I’ll be helping Henry out for these next two lessons.

If you’re interested, I have a youtube channel, twitter and instagram page, where I post about technical game audio implementation concepts and strategies.

In this lesson, we’ll be unpacking FMOD Event Callbacks.

I’ll warn you ahead of time, the implementation for this topic gets pretty complicated and ‘heady,’ but the example I’ll be providing, as well as the one provided in the FMOD documentation is pretty easy to copy and paste, and get working at a basic level.

I highly recommend doing the previous lessons before this one if you haven’t, as we’ll be building on a lot of the previously covered material.

-- In most game design scenarios, audio is expected to respond and react to the events that are happening in the game.

If you click a button, a UI sound is played.

If a creature takes a step, a footstep sound is played.

If you enter combat in an action game, battle music starts playing.

These are very common situations, and are pretty easy implementation problems to reason about.

However, sometimes, we might want parts of the game to respond to something that’s happening in our audio instead.

It’s the reverse of what we’re used to.

Examples of this can be found in any rhythm game, where certain events are synchronized with the tempo or instrumentation in the music, but you’ll also find cases such as in The Witcher 3: Blood and Wine, where enemy combat animations are being predictively assigned to line up with with events in the music.

Event Callbacks are the means FMOD uses to allow us to build these types of audio-reactive systems.

More specifically, Event Callbacks give us access to any tempo, bar and beat information on a track, in real time, as well as our current position along that track, and any custom markers or time signature markers we have assigned to the track.

Each time a change in any of these is detected, FMOD will call a function to send that new information to us in the game engine.

We can then distribute this information to any systems that will be relying upon it.

Today, I’ll be walking through the method that’s used in Unity, for setting up Event Callbacks on a music track, and then a common method for distributing that information to other systems.

We’ll be using one of these systems in our 3D Adventure Game to trigger some screen shake on each beat of the music in our final Level 2 boss fight, and to set some post-processing color changes on our camera to match the current mood of our music.

For reference, I’ll be using FMOD Version 2.01 and Unity Version 2020.3.

To start out, let’s hop over into our FMOD project, and create or select a music track.

FMOD Music Setup - 3:54 This will be easier to visualize if you’re using a music track with a pronounced beat, and for that reason, I’m using the battle music from the end of our second level in the 3D game kit.

Make sure to add a Timeline tab to our event if you don’t have one already, and place your music on it.

Now, right click the Logic Tracks section in our timeline and add a tempo marker for each tempo or time signature section in your track.

In our case, this is 112 BPM at 4/4 time.

We’ll be using this tempo and time signature data to control our beat event callbacks, and trigger our screen shake event.

Additionally, make sure the track loops.

From here, create a few destination makers, and scatter them around your track.

We won’t be using these for any logical purpose within FMOD, but we will be using their “string” names to apply specific behaviors when they’re reached.

Place them in any section of your track where you’d like to highlight a tonal shift, and then give them a name.

In my case, I’ll be naming them directly after the post processing colors I’ll be applying to our camera when they’re reached, but they can be realistically named anything.

Chorus, Verse and Bridge are some example names that could work. Intro, low-intensity, and high-intensity might also be good names.

The only caveat is there cannot be two markers that share the same name, back to back, or the second one won’t be detected as a new marker by our system. --

Now, because I’m mimicking the functionality that the Unity Battle music already had, I’ll be adding a global parameter entitled “MusicSection.”

This parameter will be used to fade our battle music in and out when we enter and exit our boss battle arena.

The parameter will be continuous, and mapped from 0 to 1. We’ll assign our volume curve such that 0 is silent and 1 is full volume.

If you already have general level music set up in your scene, you’ll want to create the opposing automation curve on that music so that it fades out as the battle music fades in.

In case you don’t have this set up, I’ll quickly walk through the creation of this event.

Just like our previous event, we’ll import our music, name our event, and assign volume automation to the inverse of how it’s assigned for our battle music.

Additionally, I’ll be applying AHDSR curves to both events, to make sure they start and stop smoothly.

Finally, we’ll select the parameter and set the seek speed to give us a gradual fade, and simplify our implementation.

Now, assign our music events to their respective bank for Level 2, and set up placement in our mixer.

I’ll just be assigning them to the master bank for now, but feel free to assign them to their respective (Level 2 Music) bank if you have it set up.

In my case, I’ll just be dropping them into a music group, and routing some of that signal to a reverb.

Let’s build our banks and hop over to our level 2 scene in Unity.

Unity Scene Setup - 7:38 For our initial scene setup, let’s make sure we have an FMOD Studio listener component in our scene, and an object to manage the loading of our FMOD banks.

I’ll be placing my listener on my primary camera, and creating a stand-alone object for bank management.

Additionally, I’ll be muting the default implementation of our level music, because I haven’t disabled all of the stock audio in my scene. Feel free to ignore this step.

Let’s also make sure our banks are assigned in the FMOD Studio Settings window.

Now from here, we’ll create a folder for our Event Callbacks scripts.

We can name this folder anything that makes sense for the context and organization of our project. I’ll just be calling mine “CustomScripts” for now.

In this folder, I’ll quickly create and add the MusicPlayer script that you all put together in the previous lessons.

I’ll now add an object into my scene, which I’ll call “GameplayMusicPlayer” and I’ll attach this script to it. If you don’t already have your basic gameplay music playing in your scene, follow along with this as well.

And finally, I’ll assign my basic gameplay music event to this object.

MusicManager Script - 9:18 Now, back to our scripts folder, let’s create a script and call it “MusicManager.”

This script will handle our Event Callbacks functionality as well as the music we’ll be tying our callbacks to. In our case, this will be the battle music.

Note that this could be combined with our MusicPlayer script, but for the sake of clarity, we’ll be splitting them up in this lesson.

Now, let’s open up our Music Manager script and get started.

The very first thing we’ll need to do is include a couple namespaces to use.

We’ll add the FMODUnity namespace, as well as a namespace that’s probably new to most of you: the System.Runtime.InteropServices namespace.

The C# scripting language has us largely working within the confines of “managed code,” or in other words, code that’s sandboxed for us, by a runtime framework -- in this case, the .NET framework.

There are a lot of benefits to using managed code, including automatic garbage collection, which I’ll talk about in a bit, but the main drawback to it, is it prevents us from talking directly to the operating system, and accessing low level functionality such as direct memory allocation.

For our music manager, we need to be able to explicitly reserve a chunk of memory so that it won’t be collected or moved by the garbage collector.

This is needed because the FMOD event callbacks we’re listening for exist outside of Unity’s managed memory, and are processed asynchronously from the rest of our game.

Our only reliable access to them is through unmanaged code on our operating system.

That all sounds pretty complex, but to simplify it, we need to use some unmanaged memory to access data directly from the FMOD audio engine, and we need to tell the Unity automatic garbage collector to ignore that memory.

But let’s first get our music set up.

We’ll first need a variable in the editor, for assigning the event.

Because I’m still using FMOD verion 2.01, I’m using the EventRef tag here.

If you’re using FMOD 2.02 or later, this will have been replaced by an EventRef “type,” which will go in place of the “string” type.

We’ll also need a field for our global parameter.

This looks similar to an event, but uses the ParamRef type.

We’ll want to cache our parameter ID at some point, because we’ll be calling updates on our parameter frequently, so we’ll need a variable for that as well.

(Zooming in on the code)

And finally, we’ll need a variable to store the instance of our music in. This is the primary access point for most of what we’ll be doing.

From here, we’ll create a Start function, and in that start function, we’ll simply create an instance of our battle music, and assign it to our instance variable.

Additionally, we’ll use the getParameterDescriptionByName function to capture our parameter description information from our music parameter event, and then cache our music section parameter ID from the description, into our respective parameter variable.

Next, because we’re assigning this instance manually, we’ll also need to release it manually, in order to avoid memory leaks.

In this case, we can simply do this in an OnDestroy function, which is called whenever the instance of this script is destroyed, or in other words, when an object in our scene containing this script is destroyed.

In our case, this is at the end of our level.

We’ll also do a null check against our music event variable -- not the instance, but the variable that’s exposed in the inspector, to make sure this is only ever run if something has been assigned as our music.

This step is somewhat optional, but it’s a nice safety check for preventing editor errors if your music hasn’t been assigned yet.

Next, let’s hook up our music in the editor.

Jumping back over into our scene, let’s create a new object for our music manager, as well as an object to act as a trigger zone for our battle music.

We’ll place the Battle Music Trigger at (103.23, 9.738728, 147.2741), and give it a Sphere Collider with a radius of 27.04.

If you’re wondering where I got these numbers, I’ve copied them from the preexisting collider from the old music system.

We’ll also add the trigger object to the Environment layer, and enable it as a Trigger, rather than a traditional collider. Both of those steps are necessary for getting our logic to function.

For more information, you can look into the Unity Physics System.

BattleMusicStateManager Script - 14:32 Now, let’s create a new script and entitle it “BattleMusicStateManager,” and attach this script to our trigger.

We’ll be using this script to trigger our shift between game and battle music, when we enter and exit our trigger collider.

All we need to add to this script are the stock OnTriggerEnter and OnTriggerExit functions, as well as a private LayerMask variable that’s visible in the editor.

In both functions, we’ll next check to see if our layermask contains anything at all, and also, if there’s any overlap between what it contains and our other layer.

This formula looks a little strange, but essentially, the value of the right hand side of the equation will return as 0 only if there are no layer matches between our trigger collider and our layer mask, and if that is the case, we can ignore the collision from here.

In the remaining cases -- when the formula doesn’t return 0, it means we’ve collided with a layer that is part of our LayerMask, and we can execute the code within the if statement.

If you want to know more about these sorts of logical comparisons, look up C# logical operators and C# bitwise operators.

It’s also worth noting that in this operation, gameObject.layer can be interpreted as an integer value by our compiler.

If you want to see which integer values correspond to a given layer, select any game object and open the ‘Layer’ dropdown. Their respective integer values are displayed next to the layer names.

For example, our environment is on layer 16.

-- Singletons - 16:18 For this next section, we need access to our Music Manager from our BattleMusicStateManager.

Typically, you might create a field -- public or serialized -- and drag in the script component for our MusicManager, or do a GetComponent call on start or awake. Instead, however -- because we will also be needing access to a few specific functions in our Music Manager later on, and because we only ever want a single Music Manager in our scene, we’re going to follow what is known as the Singleton pattern for our Music Manager.

In Section 3 of the third Module, we briefly covered singletons and statics.

What we will be doing here is saving a static instance of our Music Manager class into our scene.

All we need to do for this is create a public static variable with the type of our script’s class. We’ll name this variable ‘instance.’

Now, on Awake in our script, we’ll set ‘instance’ equal to the key word “this”.

“this” is a reference to the specific instance of our class.

But because the singleton pattern starts to break if we ever have more than one copy of a static instance in our scene, we must first check to make sure that our instance variable hasn’t yet been assigned.

If it has been assigned, that means we’re attempting to overwrite an existing static instance, so instead, we’ll destroy this current instance (the newer instance).

Otherwise, we’ll assign this instance as our static instance.

In short, this guarantees that we’ll always only have one instance of this class in our scene, and it will always be the oldest copy of that instance possible.

Newer copies will always be destroyed on Awake.

What we can now do, is access public fields and functions from our MusicManager from anywhere in our scene, even if those fields and functions aren’t static themselves.

Let’s start by creating a public function in our Music Manager, that will react to our trigger volume, and set our Music Selection parameter based on whether we’re entering or leaving our volume.

We’ll name this function “SetGlobalMusicParameter,” and we’ll give it an input type of float called “value.” Now, moving back into our Battle Music State Manager, all we have to do to transition our music between gameplay and battle music when we enter or leave this trigger volume, is pass our parameter setting function a “0” or a “1.”

The seek speed on the parameter that we set up in FMOD will handle the gradual fades for this transition.

To access this function, we simply query the MusicManager type, and from there, we can directly access any statics it holds.

This means, we can directly access whichever static instance of this class currently exists in our scene.

Now, through that instance, we can call our SetGlobalMusicParameter function, and pass in a 1 for entering the trigger, and a 0 for leaving it.

In case an instance hasn’t been assigned yet, we also may wish to do a quick null check as well.

Finally, moving back to Music Manager, let’s fill in our SetGlobalMusicParameter function.

All we need here, in order to set the parameter, is to call setParameterByID, through StudioSystem in the RuntimeManager namespace, and pass in our parameter ID, which we cached earlier, and the value we’ve received from our trigger functions.

You may notice, however, that we’re still not starting the battle music anywhere.

We’d like to start our battle music, only on the first entry of our trigger collider.

To set this up, we can create a simple private function called “StartBattleMusic(),” do a quick null check against our battle Music Event Reference, and then check the state of our music instance using “getPlaybackState.”

“getPlaybackState” uses the ‘out’ return prefix, which indicates that it’s returning a value, rather than requesting one.

You’ll see this used frequently, when the actual return type of a function is being reserved for something else, such as a function execution result, or a boolean, true/false.

This is common in functions that contain expensive calculations, but also have failure conditions or cases where the calculation can be skipped, and notifications that the function has skipped a calculation matters.

We can inline a variable declaration here, meaning the value of our output variable is being immediately assigned to a new variable within the current scope.

To do that, we simply put the type in front of the variable.

The output type is of a custom FMOD type PLAYBACK_STATE.

This is simply an enum that tells us what state the music is currently in.

In our case, we want to see if the music is either stopped or stopping.

If so, we haven’t yet started the music, so we should do so.

Now, all we need to do is confirm that our parameter value was a 1 -- or in other words, that we’re entering our music trigger, and if so, we call this new private function to start our music.

Hopping back over into the editor, let’s double-check that our battle music event and our global parameter are properly assigned on our MusicManager object.

And with that, our music should be hooked up and working.

To test, I’m going to go ahead and move Ellen into the hallways before the final battle encounter.

One final thing to check is our Layers field on our Battle Music State Manager. Make sure this is assigned to the “Player” Layer.

Now let’s test the game. If everything is working properly, the music should start as we enter the final room, fade out if we leave, and fade back in if we enter again.

...

Planning An Event Callback System - 22:47 Perfect, everything seems to be working. We can now move onto the complicated part of this lesson -- building a system for receiving event callbacks from our music event instance.

The way this system will function is every time a new beat event or FMOD marker is detected on our event instance timeline, the system will call a delegate function, which our script will be tied to, and will pass us whatever available data it has with respect to markers or our timeline. In other words, we’re simply setting up a system that listens to changes on our timeline.

Now, the first thing we’ll need for this system is somewhere to store our data.

We’ll start by creating a class within our script, which we’ll call TimelineInfo.

To start, we’ll add a public int for our current beat, and a public FMOD String Wrapper -- a custom FMOD string type -- for our last marker reached.

Next, we’ll create a variable for the instance of this TimelineInfo, as well as a variable from our InteropServices namespace -- a GCHandle (short for garbage collector handle).

We’ll name this GCHandle TimelineHandle.

GCHandles are entry points into specific chunks of unmanaged memory.

Normally, memory is accessed directly, or through pointers.

GCHandles are kind of like pointers, but they add restrictions that enable them to bridge managed and unmanaged memory spaces.

In other words, bridge our code here, with FMOD’s internal systems.

In our case, we’ll be using our GCHandle to allocate space for our TimelineInfo and pin it in memory, allowing us to easily write to it from our operating system level systems, without needing to worry about garbage collection cleaning up or moving the memory associated with our TimelineInfo.

Garbage Collection - 24:38 Now, garbage collection is the process of scanning through memory and finding all the chunks of memory that are no longer needed or assigned to any value that’s in use, and marking those chunks of memory as memory that’s free to be used again.

Unity uses an automatic garbage collection system by default, that will also periodically reorganize chunks of memory to create larger consecutive chunks of free memory for use.

If the memory we were trying to write to for your TimelineInfo was to move due to this process, it’s very likely we would miss out on some callback information, or run into some strange behaviors, as the data our TimelineInfo is receiving writes from, is our streamed audio, which runs independently from our games’ runtime, and thus, our garbage collector. In other words, we could run into problems if FMOD tried to write to our TimelineInfo as the automatic garbage collector was attempting to reorganize where that TimelineInfo was being stored in memory.

Because we’re pinning TimelineInfo to memory, we also need to specify how TimelineInfo will be organized in memory.

In our case, we can simply accomplish this by adding the StructLayout(LayoutKind.Sequential) tag to it, indicating that it will have a predictable, unchanging sequence in memory.

We also need to make sure that each data type in our class in memory is Blittable.

A blittable type is simply a type that is known by both the system and the application -- in our case, by both our operating system and Unity.

Fortunately all the types we’ll be using are blittable, so this won’t be relevant to know in depth for this lesson, and probably for future applications of this system, but it’s worth knowing if you’d like to dig deeper into how these types of interoperational systems work.

Creating An EventCallback System - 26:23 Finally, we’ll also need a variable for our Event Callbacks.

This is where we’ll store our delegate object, ensuring it isn’t caught up in garbage collection.

Now that we have all of our Event Callback system variables prepared, we can move on to setting up the initialization for our Event Callbacks system itself.

First, we’ll simply instantiate a new default TimelineInfo class, and save it in our timelineInfo private variable.

We’ll also do the same for our beatCallback variable, but this is where things start to get pretty interesting.

The type we save into beatCallback is an Event Callback.

Event callbacks are delegates.

This means that the field we need to pass into our new Event Callback is a function itself. Specifically, the function that will be receiving and processing our raw callback data.

This function doesn’t exist within the FMOD API so we’ll need to create it ourselves.

If you ever need a quick reference of this function, it can be found in the FMOD Unity Integration scripting examples under Timeline Callbacks.

We’ll be naming this function BeatEventCallback.

To start, we’ll make our function private and static, because we’ll only ever need this function in this script, and we’ll only ever want one instance of it to exist.

The return type of our function is an FMOD.RESULT, which is simply a custom FMOD enum that returns specific values depending on the result of the execution.

For example, if the function succeeds in executing as expected, it will generally return a RESULT.OK.

Next, we’ll name our function. In this case, I’ll be naming it “BeatEventCallback.”

It will require three inputs.

An Event Callback Type, a pointer to our music event instance, and a pointer to our music event parameters, which holds the event data of our event instance.

This can be a little confusing because these “parameters” are not the same as the parameters we usually talk about within FMOD.

As you can see, we’re getting an error on IntPtr, and that’s because we have not yet included the System namespace.

Now, the first thing we’ll need to do is copy in a new event instance from our instance pointer.

Next, we’ll capture the pointer to the place in memory where our current event instance data is being held with the getUserData method.

We’ll also store the result here.

We’ll do a safety check to see if anything went wrong, and log an error if so. If not, we’ll continue, and assuming our pointer is looking at anything relevant (IntPtr.Zero is effectively a pointer-specific null check) we’ll save a new GCHandle to access our event data through.

Using this GCHandle, we’ll then, finally, directly grab the Timeline Info, using the .Target property. This property simply grabs the object our GCHandle is connected to.

Next, we’ll create a switch statement to check the type of event callback we’ve received.

We’ll create two separate cases. One for if our type is a beat event, and one for if it registered a marker.

Inside both of these cases, we’ll create a ‘parameter’ variable which will hold our beat properties and marker properties respectively.

Again, this type of “parameter” we’re talking about is a programmatic parameter; it’s just a stream of data.

The properties data types generally encompass all of the different types of data that can be returned with their respective type of callback.

Now, the data we care about is currently located in our parameterPtr -- the final function input that I mentioned held our event instance data.

Because this data is still in unmanaged memory, and we’re now working in the land of managed memory, we’ll need to use the Marshal class from the InteropServices namespace, to copy over that data.

We’ll use Marshal.PtrToStructure, which takes an input of our pointer into memory -- our parameterPtr -- as well as the type of output we are looking for.

Now, because PtrToStructure returns a generic object, even if we specify a type to return, we need to also cast this object as the respective type we’re looking for.

And with that, we finally have access to our event callback data in managed memory.

From here, we can simply save values from parameter (our data), straight into our local timelineInfo variable. In the Beats section, we’ll be saving parameter.beat into timelineInfo.currentBeat, and in the Markers section, we’ll save parameter.name -- the name of the most recent marker -- into timelineInfo.lastMarker.

Now you may have noticed there are a few other types of data we have access to, on beat and on marker event.

I’m going to go ahead and fill the rest of these out, just so you’re able to see them.

I’ll also add them to my timelineInfo as well.

Feel free to follow along, but know that none of these additional values are required for what we’ll be doing today.

Finally, assuming all of our code in our BeatEventCallback function has executed properly ‘till this point, we can safely return an FMOD.RESULT of OK at the end of the function.

And with that, the difficult and confusing part of this is over with.

It’s okay if you don’t completely understand everything that’s happening here.

Most likely, you’ll never need to in order to be able to use this system, but if you’re curious, there will be a number of links below that you can dig into, that might help you develop a deeper understanding of what’s happening here, should you need to extend this system or modify it in the future.

Now, to wrap this system up, let’s move back into our Start function.

You may recall that I mentioned before that we need to pin our timelineHandle into unmanaged memory, but we still haven’t done that.

Let’s go ahead and do that now, using the “GCHandle.Alloc” method.

Additionally, we also need to tell our music event instance where this timelineHandle is in unmanaged memory.

And finally, we’ll set our beatCallback variable, as the callback variable our event instance should pay attention to, and specify that it should send callbacks on both ‘beat’ and ‘marker’ events. I’ll also quickly grab our track length from our ‘event description’ on start, though we won’t be using this today.

And I’ll store it in another private variable.

Now, before we forget, let’s also make sure we zero out the rest of our manually assigned memory values OnDestroy.

First, let’s set the pointer for our UserData to IntPtr.Zero.

Also, if we’re not already, it might be good to make sure our music instance stops playing here. We can give it an immediate fade or allow a custom fade depending on what’s needed.

Finally, we need to free our “timelineHandle” memory allocation.

We can simply use the “Free()” function here.

OnGUI Debugging - 34:48 And with that, our beat and marker detecting event callback system is fully set up and ready to test!

Now, we could test this using debug statements on update, but that’s flimsy and can be difficult to look at when monitoring multiple types of information.

So instead, let’s create an OnGUI function.

OnGUI is a function that’s great for quickly rendering UI information in Unity.

It’s great for testing and prototyping specific types of features quickly.

In this case, we’ll simply create a GUILayout Box, that will, by default, display in the upper left hand corner of the screen during runtime.

Inside this box, we’ll write our debug info as a string, but to make this a little easier to work with, we’ll use the C# System namespaces’ custom String.Format function.

This function allows us to specify object values to print, directly as strings in a field.

We’ll start by typing in quotes, “Current Beat = “, and then in curly brackets, a 0 representing the first object we want to print, followed by a comma, and “Last Marker = “ followed by a 1 surrounded by curly brackets, representing the second object.

Now, we’ll end our quotation, put a comma, then our first object -- timelineInfo.currentBeat, another comma, and then our second object, timelineInfo.lastMarker -- explicitly declared as a string.

These are the only chunks of data we’ll be using for our current system.

I’m going to go ahead and reorganize this for all of the additional ‘optional’ fields I added earlier.

Feel free to do the same.

Either way, with that, we have our OnGUI function complete.

One final thing we can add is a surrounding UNITY_EDITOR editor #if denotation, indicating that this code will only run if we’re in the editor itself.

The syntax for this is a little different, but what this means is this code will only run if we’re in the editor.

It will not run if we’re in an actual build of the game.

Now, let’s jump back into Unity, and quickly open up our FMOD settings panel.

Under “Play In Editor Settings,” make sure to disable our Debug Overlay, as it overlaps with our OnGUI display.

Now, we can test our game. Once we run into an area where our music is triggered, you should now see the beats and markers being updated in the upper left corner of your game view.

…

Event Delegates - 37:36 Great! We now have working beat and marker detection based on event callbacks.

Now… How do we use these?

We’ve talked about delegates a few times now, including in previous lessons, however we may not have touched on event delegates specifically. Events are essentially a special type of delegate that allow for safe, external subscription to a behavior.

In other words, we can create our own event callback by creating an event delegate that fires in our Music Manager, each time a marker or beat change is found, and then subscribe external functions to this event in order to trigger them reactively.

In our case, we’ll build two functions to trigger screen shake on our beat events, and color changes to our camera overlay on custom marker detection.

Back in our Music Manager script, let’s create two public delegates, with void return types.

Have one define a BeatEventDeleage type, and have this require that an integer “beat” value be passed through it.

Let’s do the same for our MarkerEventDelegate, and require a string for the “markerName” be passed through it.

Now, let’s create a public static event of type “BeatEventDelegate,” and one of type “MarkerEventDelegate.”

We’ll name these BeatUpdated and MarkerUpdated respectively.

Finally, because we only want these to fire off calls when a beat or marker changes, let’s add private variables to store the last beat and the previous marker detected. Set these to 0 and an empty string, respectively.

Now, let’s create an Update function to check for these changes.

Let’s create two if checks, comparing our Last Beat and Last Marker to see if they’re different from what’s currently registered in our “timelineInfo.”

If these pass, we’ll update our Last Beat and Last Marker with the new values we just found, and if our BeatUpdated and MarkerUpdated events aren’t null, we can call both of those, and pass in the respective beat and marker data values from our TimelineInfo class.

-- With that, we should have universal access across our project, to events that fire every time a change in our beat or marker data is detected.

Now, let’s go about subscribing to these events, in order to test that functionality.

Let’s create one more new script. We’ll call this script MusicEventListener.

In order to test our events, we only really need four functions.

We’ll add a Start and OnDestroy function, and we’ll create two custom private functions called OnBeat and OnMarker, that take an integer and string value respectively, and return void.

In order to subscribe a function to an event, you simply add it to the event.

To unsubscribe, you subtract it.

Because our events are static, we can do this directly.

In Start, through our MusicManager class, grab our Beat and Marker Updated events, and subscribe our OnBeat and OnMarker functions to them.

OnDestroy, we’ll do the same thing, but unsubscribing this time.

Now, OnBeat, we’ll create a debug log statement, and we’ll simply tell it to print our int value for our beat.

Let’s do the same for our OnMarker, but this time with our marker string name.

Now we’re ready to test!

Simply add this new “MusicEventListener” script to any object in your scene, press play, and it should work.

I’ll be adding mine to my CameraBrain under my CameraRig, as I’ll be showing off an applied example there in a minute.

Now press play, and navigate to your music trigger, and you should start to see a print-out of the beat and marker values that match those from your OnGUI display (in your Console). Camera Color Changes - 42:06 Congratulations, you now have a working beat and marker detection system!

You’re most of the way to having the foundation of most rhythm games set up!

To highlight this, I’m going to quickly extend this script we just made, to push color changes to our post process layer, as well as add a bit of screen shake on the beat of our music.

Feel free to follow along, though I won’t necessarily be explaining all the details of the subsystems we’ll be using here.

First, I’ll be adding the UnityEngine.Rendering.PostProcessing namespace to the script, to give myself access to our color filter.

Next, I’ll create a field to add the camera post process volume to.

Additionally, I’ll add a float for an internal timer I’ll be using for interpolation.

I’ll cache a record of our default color value, as well as a custom orange value, because Unity doesn’t store one by default, and I would like to use orange here.

I’ll create a field to store a “ColorGrading” variable into, which we’ll use to set our color filter value through.

And finally, I’ll create a variable for our target color and old color, which we’ll be using for color interpolation.

I’ll also assign our default color as their default values.

Now, on Start, I'll pull our colorGrading from our post process volume, assuming it’s not equal to null.

And on Update, I’ll set the value of our color filter to the current interpolated value between our old color and new color, with respect to our timer T divided by 1.867.

1.867 is a rough subdivision of the battle music BPM.

After this, I’ll increment our time variable T by time.deltaTime -- the duration of our frame update. This simply makes our interpolation time frame-rate independent.

Now, whenever a marker is called, I’ll set our old color = to our target color, and reset our timer T to 0;

From here, I’ll create a switch statement, and pass in our markerName.

If you recall from the start of the lesson, I set up my markers as color values.

Depending on which color we read in, I’ll set our targetColor to a cached color value to represent that color.

Just in case, I’ll also set up a default for our switch, to return to the default color.

Finally, On Beat, I’ll simply use a built-in 3D Gamekit function under the CameraShake namespace called Shake, and pass it in a small impulse (.02f) and a small duration (.1f).

And that’s all we really need!

Hopping back into the editor, let’s make sure our post process volume is assigned, and let’s test.

I’ll drag my camera into the post process volume field.

...

As you can see, we now have a small camera shake on beat, and each time the music reaches a new marker, we’re shifting the post process colorFilter to a new color.

Outro - 45:50 As I’ve mentioned a few times now, this was a lot of material, and likely a lot of new material as well.

A lot of what we covered requires a pretty deep understanding of operating system level code, compilers, audio engines and computer hardware to fully understand.

That said, with what we covered in this lesson, you should have at least a general idea of what’s happening with this system, and though it may take a bit to settle in, I hope you feel comfortable using Event Callbacks when you need them! To summarize, FMOD Event Callbacks are an incredibly powerful way to build unique experiences and create gameplay that reacts to audio, rather than just having audio react to gameplay.

The next lesson will build on much of the material we covered here and I promise, it’ll be at least the slightest bit less complicated.

If you do have any concerns or questions about the material we covered here, please feel free to ask about them in the Discord.