J-DSP Editor iJDSP – Interactive Illustrations of Speech/Audio Processing Concepts NSF Phase 3 J-DSP Workshop, UCy Presentation of an Independent Study By Girish Kalyanasundaram, MS by Thesis in EE Advisor: Dr. Andreas Spanias, Professor, Electrical Engineering, ASU http://jdsp.asu.edu 1 J-DSP Block Diagram Based Learning in iJDSP Editor • Effective for constructing basic systems for visualizing speech/audio DSP concepts. • Requirements: – Provision of speech/audio signals – Microphone Recording and Playback facility – Frame-by-Frame Processing Capability – Effective visualization tools http://jdsp.asu.edu 2 J-DSP Extension of iJDSP Architecture Editor • Added framework to create blocks capable of processing long signals. • These blocks can also interface with conventional blocks processing short signals. • Frame by frame processing functionality. • Paradigm designed for creating planned functions in iJDSP – instant setup of complex diagrams. http://jdsp.asu.edu 3 J-DSP The Long Signal Generator Editor • Hosts pre- defined speech, music and noise signals. • Frame-wise traversal and visualization facilitated. • Configurable Parameters: – Frame Size – Inter-frame overlap – Choice of signal – Gain applied to signal http://jdsp.asu.edu 4 J-DSP Sound Recording & Rendering Editor • Records sound at specified sampling rate (8, 16 or 44.1 kHz). • Frame-wise traversal of signal through playback buttons. • Sound player aggregates parsed data for rendering. http://jdsp.asu.edu 5 J-DSP Frame-By-Frame Visualizations Editor Plot Frequency Response • Playback buttons used to traverse through frames of incoming signal. http://jdsp.asu.edu 6 J-DSP Speech/Audio DSP Functions Editor • Spectrogram • Linear Predictive Coding (LPC) • Quantization • Line Spectral Pairs • MPEG I Layer 3 Psychoacoustic Model • Loudness Estimation http://jdsp.asu.edu 7 J-DSP Spectrogram Editor http://jdsp.asu.edu 8 J-DSP Spectrogram (Contd.) Editor • Enables visualization of spectrograms of long signals. • Can study the properties of spectrograms. • Effect of window length and overlap in time and frequency resolution. • Window types and effect of taper on resolution. http://jdsp.asu.edu 9 J-DSP Linear Predictive Coding Editor • LPC dashboard to view LPC coefficients, formants, residuals, and PZ Plot. http://jdsp.asu.edu 10 J-DSP LPC & Quantization Editor • Observation of quantization effects on LPC coefficients and residual. • Observation of overall SNR of signal. http://jdsp.asu.edu 11 J-DSP Quantizer Block Editor • Quantizer block can be use in quantizing LPC coefficients. http://jdsp.asu.edu 12 J-DSP LPC & Quantization – SNR Editor • The SNR block has been equipped with capability to aggregate SNR for entire signals. http://jdsp.asu.edu 13 J-DSP Line Spectral Pairs (LSPs) Editor • Block to display LSPs and Line spectral frequencies of input LPCs. • LPC – LSP Placement Demo. • Block demonstrating quantization effects of LPC. http://jdsp.asu.edu 14 J-DSP LSP Block Editor • Displays the LSP Filters Pole Zeros Plots and the line spectral frequency values. http://jdsp.asu.edu 15 J-DSP LPC-LSP Demo Editor • LPC poles can be placed and the LSPs will be shown dynamically. • Relation between the LPC pole locations and the LSP poles properties can be studied. http://jdsp.asu.edu 16 J-DSP LPC-LSP Quantizer Editor • This block can demonstrate that LSF quantization prevents unstable poles better than LPC quantization. http://jdsp.asu.edu 17 J-DSP LPC-LSP Quantizer (Contd.) Editor • The block can be used in combination with the LPC-LSP placement demo block to achieve this. http://jdsp.asu.edu 18 J-DSP LPC-LSP Quantizer (Contd.) Editor • Displays to view LPC poles and LSF Zeros http://jdsp.asu.edu 19 J-DSP Psychoacoustic Model Editor • The psychoacoustic model block implements the MPEG I Layer 3 psychoacoustic model . • A modified form of the ISO standard C source code, with improved efficiency for the psychoacoustic model is adopted. • Visualizations shown: – Global Masking Threshold (JND curve) – Psychoacoustically peak-picked spectrum – Comparison between original and psychoacoustically peak-picked signal http://jdsp.asu.edu 20 J-DSP Psychoacoustic Model Interface Editor • JND curve. http://jdsp.asu.edu 21 J-DSP Psychoacoustic Model Interface Editor • Psychoacoustically masked spectrum. http://jdsp.asu.edu 22 J-DSP Psychoacoustic Model Interface Editor • Original vs. peak-picked signal. http://jdsp.asu.edu 23 J-DSP Loudness Editor • A measure of perceived intensity. • A ‘psychophysical’ phenomenon. • A non-linear quantity. • Reflects properties of the auditory system. Human Auditory System: • The ear acts as a bank of band-pass filters detecting frequencies in the sound. • Perceptual models capture properties of the human auditory system. http://jdsp.asu.edu 24 J-DSP The Units of Loudness Editor The ‘Phon’ scale The ‘Sone’ Scale • Derived relative to a 1 kHz • For a signal with loudness sine wave reference. level X phons, loudness in • A signal has loudness level sones is: X 40 of X phons when a 1 kHz L 2 10 sinusoid of X dB SPL is • A signal with L sones is perceived to be equally twice as loud as a signal of loud. loudness L/2 sones. http://jdsp.asu.edu25 J-DSP Loudness Estimation Methods Editor • Neural Excitations (Auditory representations) – Compute neural activity for stimuli Excitation Specific Loudness Outer & Pattern Auditory Computation Sound Middle Ear Filters (Non-linear Filter Transformation) Loudness Integrate Specific MOORE-GLASBERG MODEL Loudness • Approved by ANSI as a new loudness standard. • Satisfactory performance for several kinds of spectra. http://jdsp.asu.edu 26 J-DSP Loudness in Psychoacoustic Model Block Editor • Spectrum, Specific Loudness Pattern and Total Loudness Displayed in Psychoacoustic Model Block. http://jdsp.asu.edu 27 J-DSP Loudness Illustration Editor • Two Signals with same energy in dB level can have different loudness. Signal with 3 sines Signal with 1 sine http://jdsp.asu.edu 28 J-DSP Editor THANK YOU http://jdsp.asu.edu29 J-DSP Loudness Estimation Methods Editor • A, B, C Weighting – Filters spectrum to compute loudness (derived from ELC) • Steven’s Law – A power law for loudness of signal with intensity I: L kI0.3 Equal Loudness Contours (ELC) * ReplayGain: A loudness leveler using ELC. SpotifyTM uses ReplayGain. * http://www.music.princeton.edu/~newton/teaching/music3/2007/fletchermunson.html http://jdsp.asu.edu 30.