J-DSP Editor

iJDSP – Interactive Illustrations of Speech/Audio Processing Concepts NSF Phase 3 J-DSP Workshop, UCy Presentation of an Independent Study By Girish Kalyanasundaram, MS by Thesis in EE Advisor: Dr. Andreas Spanias, Professor, Electrical Engineering, ASU

http://jdsp.asu.edu 1 J-DSP Block Diagram Based Learning in iJDSP Editor

• Effective for constructing basic systems for visualizing speech/audio DSP concepts.

• Requirements: – Provision of speech/audio signals – Microphone Recording and Playback facility – Frame-by-Frame Processing Capability – Effective visualization tools

http://jdsp.asu.edu 2 J-DSP Extension of iJDSP Architecture Editor

• Added framework to create blocks capable of processing long signals. • These blocks can also interface with conventional blocks processing short signals. • Frame by frame processing functionality. • Paradigm designed for creating planned functions in iJDSP – instant setup of complex diagrams.

http://jdsp.asu.edu 3 J-DSP The Long Signal Generator Editor

• Hosts pre- defined speech, music and noise signals. • Frame-wise traversal and visualization facilitated. • Configurable Parameters: – Frame Size – Inter-frame overlap – Choice of signal – Gain applied to signal

http://jdsp.asu.edu 4 J-DSP Sound Recording & Rendering Editor • Records sound at specified sampling rate (8, 16 or 44.1 kHz). • Frame-wise traversal of signal through playback buttons. • Sound player aggregates parsed data for rendering.

http://jdsp.asu.edu 5 J-DSP Frame-By-Frame Visualizations Editor

Plot

Frequency Response

• Playback buttons used to traverse through frames of incoming signal.

http://jdsp.asu.edu 6 J-DSP Speech/Audio DSP Functions Editor

• Spectrogram • (LPC) • Quantization • Line Spectral Pairs • MPEG I Layer 3 Psychoacoustic Model • Loudness Estimation

http://jdsp.asu.edu 7 J-DSP Spectrogram Editor

http://jdsp.asu.edu 8 J-DSP Spectrogram (Contd.) Editor

• Enables visualization of spectrograms of long signals. • Can study the properties of spectrograms. • Effect of window length and overlap in time and frequency resolution. • Window types and effect of taper on resolution.

http://jdsp.asu.edu 9 J-DSP Linear Predictive Coding Editor

• LPC dashboard to view LPC coefficients, formants, residuals, and PZ Plot.

http://jdsp.asu.edu 10 J-DSP LPC & Quantization Editor • Observation of quantization effects on LPC coefficients and residual. • Observation of overall SNR of signal.

http://jdsp.asu.edu 11 J-DSP Quantizer Block Editor • Quantizer block can be use in quantizing LPC coefficients.

http://jdsp.asu.edu 12 J-DSP LPC & Quantization – SNR Editor • The SNR block has been equipped with capability to aggregate SNR for entire signals.

http://jdsp.asu.edu 13 J-DSP Line Spectral Pairs (LSPs) Editor

• Block to display LSPs and Line spectral frequencies of input LPCs. • LPC – LSP Placement Demo. • Block demonstrating quantization effects of LPC.

http://jdsp.asu.edu 14 J-DSP LSP Block Editor

• Displays the LSP Filters Pole Zeros Plots and the line spectral frequency values.

http://jdsp.asu.edu 15 J-DSP LPC-LSP Demo Editor • LPC poles can be placed and the LSPs will be shown dynamically. • Relation between the LPC pole locations and the LSP poles properties can be studied.

http://jdsp.asu.edu 16 J-DSP LPC-LSP Quantizer Editor • This block can demonstrate that LSF quantization prevents unstable poles better than LPC quantization.

http://jdsp.asu.edu 17 J-DSP LPC-LSP Quantizer (Contd.) Editor • The block can be used in combination with the LPC-LSP placement demo block to achieve this.

http://jdsp.asu.edu 18 J-DSP LPC-LSP Quantizer (Contd.) Editor

• Displays to view LPC poles and LSF Zeros

http://jdsp.asu.edu 19 J-DSP Psychoacoustic Model Editor

• The psychoacoustic model block implements the MPEG I Layer 3 psychoacoustic model . • A modified form of the ISO standard C source code, with improved efficiency for the psychoacoustic model is adopted. • Visualizations shown: – Global Masking Threshold (JND curve) – Psychoacoustically peak-picked spectrum – Comparison between original and psychoacoustically peak-picked signal

http://jdsp.asu.edu 20 J-DSP Psychoacoustic Model Interface Editor

• JND curve.

http://jdsp.asu.edu 21 J-DSP Psychoacoustic Model Interface Editor

• Psychoacoustically masked spectrum.

http://jdsp.asu.edu 22 J-DSP Psychoacoustic Model Interface Editor

• Original vs. peak-picked signal.

http://jdsp.asu.edu 23 J-DSP Loudness Editor

• A measure of perceived intensity. • A ‘psychophysical’ phenomenon. • A non-linear quantity. • Reflects properties of the auditory system. Human Auditory System: • The ear acts as a bank of band-pass filters detecting frequencies in the sound. • Perceptual models capture properties of the human auditory system.

http://jdsp.asu.edu 24 J-DSP The Units of Loudness Editor

The ‘Phon’ scale The ‘Sone’ Scale

• Derived relative to a 1 kHz • For a signal with loudness sine wave reference. level X phons, loudness in

• A signal has loudness level sones is: X 40 of X phons when a 1 kHz L  2 10 sinusoid of X dB SPL is • A signal with L sones is perceived to be equally twice as loud as a signal of loud. loudness L/2 sones.

http://jdsp.asu.edu25 J-DSP Loudness Estimation Methods Editor • Neural Excitations (Auditory representations) – Compute neural activity for stimuli Excitation Specific Loudness Outer & Pattern Auditory Computation Sound Middle Ear Filters (Non-linear Filter Transformation)

Loudness Integrate

Specific MOORE-GLASBERG MODEL Loudness • Approved by ANSI as a new loudness standard. • Satisfactory performance for several kinds of spectra.

http://jdsp.asu.edu 26 J-DSP Loudness in Psychoacoustic Model Block Editor • Spectrum, Specific Loudness Pattern and Total Loudness Displayed in Psychoacoustic Model Block.

http://jdsp.asu.edu 27 J-DSP Loudness Illustration Editor • Two Signals with same energy in dB level can have different loudness.

Signal with 3 sines

Signal with 1 sine

http://jdsp.asu.edu 28 J-DSP Editor

THANK YOU

http://jdsp.asu.edu29 J-DSP Loudness Estimation Methods Editor

• A, B, C Weighting – Filters spectrum to compute loudness (derived from ELC) • Steven’s Law – A power law for loudness of signal with intensity I: L  kI0.3

Equal Loudness Contours (ELC) * ReplayGain: A loudness leveler using ELC. SpotifyTM uses ReplayGain.

* http://www.music.princeton.edu/~newton/teaching/music3/2007/fletchermunson.html

http://jdsp.asu.edu 30