MicArrayEchoCancellation Walkthrough: C++

Capturing Audio Streams with Acoustic Echo Cancellation and Beamforming

About This Walkthrough In the Kinectct™™ for Windows®® (SDK), the MicArrayEchoCancellation sample shows how to capture an audio stream from the microphone aae rray of the Kinect for Xbox 360®® se sensor by using the MSRKineinectAudio DirectX ®® media object (DMO) in a Microsoft DirectShow®® graph. This document provides a walkthrough review of the MicArrayEchoCancellation sample. Resources For a complete list of documentation for the Kinect for Windows SDK Beta, plus related reference and links to the online forums, see the bbe eta SDK website at: http://kinectforwindows.org

Contents Introduction ...... 2 Program Description ...... 3 Create and Configure the MSRKinectAudio DMO ...... 4 Select the Kinect Sensor‘s Microphone Array ...... 55 Enumerate the Device Index...... 6 Determine the Device Index ...... 7 Record the Captured Stream and DeDetertermine the Source Direction ...... 8 Set Up the Data Buffer ...... 8 Set the Output Format ...... 8 Allocate Resources and the Output Buffer ...... 9 Capture the Audio Stretream and Determine Source Direction ...... 10

License: The Kinect for Windows SDK Beta is licensed for non-commercial use only. By in stalling, copying, or otherwise usin g the beta SDK, you agree to b e bound by the terms of its li cecensense.. Read the license.. Disclaimer: TThis document is provided ―as--is‖. Information and views expressed in this document, including URL and otherher InInternet Web site references, may change without notice. You bear the risk of using it. This document does not provide you with any lelegal rights to any in teltellectual property in aain ny Microsoft proproduct. You may copy an d uud se this document for youou r ininternal, reference purposes.es. © 2011 Microsofsoft Corporation. All rights reserved.. Microsoft, DirectShow, DirectX, Kininect, MSDN, Windows, and are trademarks of th e Microsoft group of compan ies. All other trademarks are property of their respective owners. MicArrayEchoCancellation Walkthrough: C++ – 2

Introduction The audio component of the Kinect™ for Xbox 360® sensor is a four-element linear microphone array. An array provides some significant advantages over a single microphone, including more sophisticated acoustic echo cancellation and noise suppression, and the ability to determine the direction of a sound source. The primary way for C++ applications to access the Kinect sensor‘ s microphone array is through the MSRKinectAudio Microsoft® DirectX® media object (DMO). A DMO is a standard COM object that can be incorporated into a Microsoft DirectShow® graph or a Microsoft topology. The Kinect for Windows® Software Development Kit (SDK) Beta includes an extended version of the Windows microphone array DMO—referred to here as the MSRKinectAudio DMO—to support the Kinect microphone array. The MSRKinectAudio DMO supports all the standard microphone array functionality, which includes:

 Acoustic echo cancellation (AEC)

 Microphone array processing (MicArray)

 Noise suppression (NS)

 Automatic gain control (AGC)

 Voice activity detection (VAD)

 Sound source localization, which identifies the direction of the source in the horizo ntal plane

 Beamforming, which allows the array to function as a steerable directional microphone. The DMO supports 11 beams, with fixed directions that range from -50 to+50 degrees in 10-degree increments. For more information on the standard microphone array, see ―Microphone Array Support in ‖ and ―How to Build and Use Microphone Arrays for Windows Vista‖ on the Microsoft Developer Network (MSDN®) website. Although the internal details for MSRKinectAudio DMO are different, you use it in much the same way as the standard microphone array DMO, with the following exceptions. The MSRKinectAudio DMO:

 Has its own class identifier (CLSID)—CLSID_CMSRMSRKinectAudio.

 Exposes sound source localization functionality through a new interface—ISoundSourceLocalizer.

 Supports an additional microphone array mode—adaptive beamforming)—which uses an internal source localizer to automatically determine the beam direction. The MicArrayEchoCancellation sample shows how to capture an audio stream from the Kinect sensor‘s microphone array by polling the MSRKinectAudio DMO in source mode. The application uses AEC to record a high-quality audio stream and beam-forming to determine the direction to the sound source. The DMO can also be used with a Microsoft Media Foundation topology. For an example, see ―MFAudioFilter Walkthrough: C++ Sample‖ on the beta SDK website. Note DirectShow is COM-based, and this document assumes that you are familiar with how to use COM objects and interfaces. You do not need to know how to implement COM objects. For the basics of how to use COM objects, see ―Programming DirectX with COM‖ on the MSDN website. That MSDN topic is written for DirectX programmers, but the basic principles apply to all COM -based applications. MicArrayEchoCancellation Walkthrough: C++ – 3

Program Description MicArrayEchoCancellation is installed with the Kinect for Windows Software Development Kit (SDK) Beta samples in %KINECTSDK_DIR%\Samples\KinectSDKSamples.zip.. MicArrayEchoCancellation is a C++ console application that is implemented in MicArrayEchoCancellation.cpp. The basic program flow is as follows:

1. Create and configure the MSRKinectAudio DMO. 2. Enumerate the available capture devices and select the Kinect sensor‘s microphone array. 3. Record 10 seconds of audio stream and determine the source direction as the capture process progresses. To run MicArrayEchoCancellation, start MicArrayEchoCancellation.exe and follow the instructions in the console window.

Tip Before attempting to capture audio from the microphone array, you must be actively streaming to the audio render device that is specified for the DMO —typically the system‘s speakers. Otherwise, the MSRKinectAudio DMO fails. AEC is designed to cancel interfering sounds, so there must be something to cancel. The simplest solution is to start playing a tune on Windows Media ® Player before you run the application. The Libraries\Music\Sample Music folder on your Windows PC contains some sample music files.

The following is a lightly edited version of the output from a MicArrayEchoCancellation session, where the sound source moved from side to side as capture progressed: Start a song in and then press any key to start recording (echo cancellation processing expects speakers to be producing sound).

Recording using DMO

AEC-MicArray is running ... Press "s" to stop

Position: -0.051290 Confidence: 1.000000 Beam Angle = 0.0000000 Sound output was written to file: C:\KDK\Samples\Audio\MicArrayEchoCancellation\CPP\AECout. The recording process uses beamforming, which creates a single directional channel from the four microphones in 16-kHz, 16-bit mono pulse code modulation (PCM) format. The channel is oriented to one of the 11 beam directions. MicArrayEchoCancellation uses adaptive beamforming, which automatically selects the beam that is closest to the source direction.

You can use the captured stream for many purposes. MicArrayEchoCancellation simply writes the captured audio stream to AECout.wav—which is a .wav file that can be played with Windows Media Player.

The rest of this document is a walkthrough of the MicArrayEchoCancellation sample. It describes all the sample‘s functionality except for writing the capture stream to a .wav file. For details on that process, see the sample code. MicArrayEchoCancellation Walkthrough: C++ – 4

Note This document includes code excerpts, most of which have been edited for brevity and readability. In particular, most routine error-correction code has been removed. For the complete code, see the MicArrayEchoCancellation sample. Hyperlinks in this walkthrough refer to content on the MSDN website.

Create and Configure the MSRKinectAudio DMO The application‘s entry point— _tmain—manages the overall program execution, with private methods handling most details. The first step is to create and configure an instance of the MSRKinectAudio DMO, as follows: #include "MSRKinectAudio.h" ... int __cdecl _tmain(int argc, const TCHAR ** argv) { HRESULT hr = S_OK; CoInitialize(NULL); int iMicDevIdx = -1; IMediaObject* pDMO = NULL; IPropertyStore* pPS = NULL; ... SetPriorityClass (GetCurrentProcess(), HIGH_PRIORITY_CLASS);

CoCreateInstance(CLSID_CMSRKinectAudio, NULL, CLSCTX_INPROC_SERVER, IID_IMediaObject, (void**)&pDMO); pDMO->QueryInterface(IID_IPropertyStore, (void**)&pPS);

PROPVARIANT pvSysMode; PropVariantInit(&pvSysMode); pvSysMode.vt = VT_I4; pvSysMode.lVal = (LONG)(4); pPS->SetValue(MFPKEY_WMAAECMA_SYSTEM_MODE, pvSysMode); PropVariantClear(&pvSysMode); ... } In addition to standard header files, the file includes the beta SDK header file, MSRKinectAudio.h. This file contains the DMO‘s globally unique identifiers (GUIDs) and interface declarations for the MSRKinectAudio DMO. Before creating the DMO, MicArrayEchoCancellation calls the SetPriorityClass function to set the process‘s priority to HIGH_PRIORITY_CLASS. This helps ensure that the microphone is not preempted during the capture process. MicArrayEchoCancellation calls the CoCreateInstance function to create an instance of the DMO and obtain its IMediaObject interface, which supports a set of methods that let you control the DMO. A DMO supports a property store—which you access through an IPropertyStore interface—that contains a collection of key-value pairs. You configure a DMO by setting the appropriate keys in the object‘s property store. MicArrayEchoCancellation Walkthrough: C++ – 5

DMO property store keys are GUIDs that are defined by Media Foundation. The associated values are packaged as a PROPVARIANT structure, which must be initialized by calling the PropVariantInit function. You then assign values to the following two members:

 PROPVARIANT.vt, a VARENUM value that specifies the data type. For example, a 4-byte unsigned integer corresponds to a VT_I4 data type.

 A value member, whose name depends on vt. For VT_I4, the value member is PROPVARIANT.lVal. MicArrayEchoCancellation calls the DMO‘s QueryInterface method to obtain an IPropertyStore pointer, and then calls the IPropertyStore::SetValue method to specify the system mode. The key is MFPKEY_WMAAECMA_SYSTEM_MODE , and four modes are available, each of which has a corresponding VT_I4 value as shown in the following table.

Mode Value Single-channel with AEC 0 Microphone array 2 Microphone array with AEC 4 Single-channel with automatic gain control 5 MicArrayEchoCancellation uses the array with AEC, so the value is set to 4. Note MicArrayEchoCancellation uses the MSRKinectAudio DMO‘s default microphone array mode— which enables adaptive beamforming. So MicArrayEchoCancellation does not explicitly set MFPKEY_WMAAECMA_FEATR_MICARR_MODE .

Select the Kinect Sensor’s Microphone Array Because a system can have more than one active microphone, MicArrayEchoCancellation must specify which microphone stream is to be captured by setting the MFPKEY_WMAAECMA_DEVICE_INDEXES key to the appropriate device index. To determine the index, MicArrayEchoCancellation enumerates the available microphones, determines which one is the Kinect sensor‘s microphone array, and uses that index to set the value, as follows: int __cdecl _tmain(int argc, const TCHAR ** argv) { ... hr = GetMicArrayDeviceIndex(&iMicDevIdx);

PROPVARIANT pvDeviceId; PropVariantInit(&pvDeviceId); pvDeviceId.vt = VT_I4; pvDeviceId.lVal = (unsigned long)(iSpkDevIdx<<16) | (unsigned long)(0x0000ffff & iMicDevIdx);

pPS->SetValue(MFPKEY_WMAAECMA_DEVICE_INDEXES, pvDeviceId); PropVariantClear(&pvDeviceId); ... } MicArrayEchoCancellation W alkthrough: C++ – 12

 If ProcessOutput returns S_OK, the buffer is filled and the DShowRecord calls the buffer object‘s GetBufferAndLength method to get the buffer‘s length. The DMO_OUTPUT_DATA_BUFFERF_INCOMPLETE flag is cleared, so the do-while loop terminates and the outer loop can proceed to the next buffer. For details on GetBufferAndLength, see the sample.

The loop next writes the buffer to a file, which does nothing if the buffer is not ready. For details, see the sample. Finally, the loop calls ISourceLocalization::GetBeam and ISourceLocalization::GetPosition:

 GetBeam retrieves the direction of the currently selected beam as 0.175-radian (10-degree) increments relative to camera coordinates.

 GetPosition return the estimated source direction. Because the beam is limited to one of the 11 supported directions, it typically does not point directly to the source and is accurate to only ±5 degrees. GetPosition uses a source localization algorithm that provides a more accurate estimate of the source direction, including a confidence value that is a measure of the estimate‘s accuracy. You must call ProcessOutput before you call GetBeam or GetPosition. The ProcessOutput method drives processing audio samples through the audio pipeline, which is required to determine the source direction. GetBeam is successful only after ProcessOutput returns S_OK:

 If ProcessOutput returns S_OK, the loop checks the GetBeam return value to determine whether the source direction has changed and, if it has, prints the new direction.

 If ProcessOutput returns S_FALSE, GetBeam fails silently and the loop repeats.

For More Information For more information about implementing audio and related samples, see the Programming Guide page on the Kinect for Windows SDK Beta website at: http://kinectforwindows.org