ESE250 – Week 5

Nyquist-Shannon Theorem

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Course Map

Today

Numbers correspond to course weeks

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem2 Over Where are we ? r(t) DFT LPF Sample q + n Q + N Q Week 2: Sampling C Store / Perceptual Received signal is sampled & quantized Produce Decode q = PCM[ r ] p(t) Transmit Coding Week 3 : Compression Quantized Signal is Coded c =code[ q ] Week 4 : Frequency Domain Sampled signal first transformed into Week 4 Week 5 Week 3 frequency domain Q = DFT[ q ] Week 5 signal oversampled & low pass filtered Q = LPF[ DFT(q+n) ] Week 6 Transformed signal analyzed Using human psychoaoustic models Week 7 [Painter & Spanias. Proc.IEEE, 88(4):451–512, 2000] Acoustically Interesting signal is “perceptually coded” C = MP3[ Q]

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem3 4 Reconstruction Generic Digital Signal Processor ● Acquisition side: we need to: Store/ ● Sample r(t) Sample Code Transmit Decode Produce p(t) q c ● Then process ● Then store discretely ● Then transmit. ● On the other end: ● How should the signal be reproduced? ● How fast do we send information to sound card? ● What functions do we use to interpolate samples? ● Last week: Harmonic Reconstruction ● Gets better with more terms ● But need more samples to compute them ● And error seems unpredictable ● This week: General reconstruction ● Make some assumptions about signal ● That allow us to exactly reconstruct from samples

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Bohemian Rhapsody

● Versions sampled at different sampling rates ● 400Hz, then 1kHz, 2kHz, 3kHz.... 44kHz. ● How can we describe the distortion?

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Shannon's Theorem

● Given a signal whose frequency does not exceed f max ● We can perfectly reconstruct the signal

● From samples taken once every T seconds s

● If 2 x f ≤ 1/(T ) max s the “”

● More simply: 2 x f ≤ f max N - the “Nyquist Frequency”

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 1

● Consider a pure sine wave, sin(2πft), where f = 5Hz

● Only one frequency component, so f = f max ● Let's sample at 25Hz.

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 1

● Let's sample at 25Hz. – 25Hz is greater than 2 x 5Hz, so should be able to perfectly reconstruct signal – Connect the dots – looks good! – Now let's sample at 10Hz.

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 1

● Let's sample at 10Hz. – 10Hz is equal to 2 x 5Hz, so should still be able to perfectly reconstruct signal – Connect the dots – looks good! – Now let's sample at 6.25Hz.

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 1

● Let's sample at 6.25Hz. – 6.25Hz is less than 2 x 5Hz, so we should expect some problems. – Connect the dots – what do we see? – We'll return to this later in the lecture

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem “Connect-The-Dots?”

● We're actually doing “pulse shaping” – Formally: use Δ – functions or sinc functions – Practically: use pulses of height (sample) and width T . s

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Shannon's Theorem

● Given a signal whose frequency does not exceed f max ● We can perfectly reconstruct the signal

● From samples taken once every T seconds s

● If 2 x f ≤ 1/(T ) max s the “Nyquist Rate”

● More simply: 2 x f ≤ f max N - the “Nyquist Frequency”

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Question

● Imagine we have a signal with many harmonics

● DFT will yield a large number of frequencies ● For perfect reconstruction, we need to store – the amplitude – of each frequency – at each sample point

● OR we could just sample at 2f and store max – ONE amplitude – per sample point ● Why do we need DFT at all?

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Noise

● In reality we receive a mixed signal r(t): r(t) = q(t) + n(t) q(t) = signal of auditory interest n(t) = noise (no information) – Typically high-frequency – From recording, transmission, background sounds ● We only get to sample r(t), not q(t)!

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 2

● Consider signal r(t): ● As in Example 1, r(t) = q(t) + n(t) – Higher-frequency noise – = 0 (silence) – Can alias down into the – = noise (33kHz and above) recorded signal – Sample r @ 22kHz

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem

● Widely known problem ● Take a signal s = sin(2π(f+δ)t) ● If we sample at f ● “Folding” - a manifestation of aliasing: ● The aliased noise will be sin(2πδt) ● “Folds over” as if was actually a ● “Wagon wheel” effect lower frequency.

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Example 1 Revisited

● 5Hz Signal sampled at 6.25Hz. ● Bottom line: Aliasing is – Folding predicts an aliased frequency undesirable. of (10-6.25) = 3.75Hz ● – Is this what we observe? How do we prevent it?

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Anti-Aliasing

● Given the “mixed” signal r(t) = q(t) + n(t) – q(t) = Auditory signal, f = 22kHz max – n(t) = High-frequency noise ● We require a process to filter out the noise: – Some sort of “anti-aliasing” technique – To “smooth away” the noise – (which would otherwise appear in our samples) – With some determined cut-off frequency between noise and signal ● How can we do this?

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Interlude: Visual Aliasing

● Strobe light: making water run uphill

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Shannon's Theorem

● Given a signal whose frequency does not exceed f max ● We can perfectly reconstruct the signal

● From samples taken once every T seconds s

● If 2 x f ≤ 1/(T ) max s the “Nyquist Rate”

● More simply: 2 x f ≤ f max N - the “Nyquist Frequency”

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Fourier's Theorem (review)

● Given a “nice” periodic signal, ● We can perfectly re-construct the signal ● From just its harmonics – Lecture 4 – Lab 4 (Synthesizer)

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Clipping a non-periodic signal

● Consider a non-periodic signal ● Look at just a finite time window t

● t t + T The signal is s “clipped.”

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Clipping a non-periodic signal

● Fourier Analysis gives us the frequency content of our clipped signal ● Can use this to – perfectly reconstruct – the periodic extension – Of the clipped signal

t

t t + T s

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Fourier + Shannon

● Fourier: Clipped – Over a finite time window frequency forces finite – harmonic We can construct an exact series harmonic representation ● Shannon: – Over a finite range of Clipped time frequencies forces finite samples – We can construct an exact representation in time

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Fourier + Shannon

● Summary:

– If we have a finite sampling window T 0

– And finite bandwidth f max – We are guaranteed a finite, exact harmonic reconstruction

● We can now solve the anti-aliasing problem

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Anti-Aliasing Revisited

● Given the “mixed” signal r(t) = q(t) + n(t) – q(t) = Auditory signal, f = 22kHz max – n(t) = High-frequency noise ● We need to remove n(t) to be left with just the audible signal q(t). ● If we remove n(t), then no high-frequency noise to alias into our sample!

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Q + N = DFT(q) + DFT(n) = DFT(q+n)

● We can filter out the noise thus: – Take the received time-sampled signal r(t) – Fourier: DFT(r):

● Finite sample period f max ● So can find frequency-domain representation – Assume q ranges from 0-f Hz, n from f -∞ Hz max max

● Can split DFT into terms < f , and terms ≥ f max max ● i.e. DFT(q+n) = DFT(q) + DFT(n) – Throw out DFT(n) terms – Perform inverse DFT on DFT(q) – Shannon: can now perfectly reconstruct q(t)!

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Oversampled Pre-Filter

● Anti-Aliasing requires Low-Pass Filter “” r Filter q Transform Q – f > 2 x maximum noise frequency s ● How do we know this maximum noise frequency ? – Microphone is band-limited; has r Pre-Sampling Low-Pass its own cut-off frequency – Might also add a “low-pass filter” ADXL001 MEMS accelerometer

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Open Questions

● How do we decide on f ? max – At what frequency do we distinguish between signal and noise? ● Why do CDs sample at 44.1kHz? – What is the significance of this number?

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem Big Ideas

● In order to perfectly sample a signal, 2 x f ≤ f max s ● Sampling at too low a frequency will cause aliasing – which is detrimental to signal quality ● In order to prevent high-frequency noise from aliasing down into our signal – we must remove it with a low-pass filter

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem ESE250 – Week 5

Nyquist-Shannon Theorem

ESE250 S'13: DeHon, Kadric, Kod, Wilson-Shah Week 5 – Nyquist-Shannon theorem