Sound of Music How It Works Session 3 Hearing Music and the Ear
OLLI at Illinois Spring 2020
Endlessly Downward Beatsystem D. H. Tracy emt 5595 (1995) Sound of Music How It Works Session 3 Hearing Music and the Ear
OLLI at Illinois Spring 2020
D. H. Tracy Course Outline
1. Building Blocks: Some basic concepts 2. Resonance: Building Musical Sounds 3. Hearing Music and the Ear 4. Musical Scales 5. Musical Instruments 6. Singing and Musical Notation 7. Harmony and Dissonance; Chords 8. Combining the Elements of Music
2/11/2020 Sound of Music 3 4 OLLI-Vote 2020 Wands
NO MAYBE BAD YES ??? GOOD
2/11/2020 Sound of Music 3 5 2/11/2020 Sound of Music 3 6 Human Ear
2/11/2020 Sound of Music 3 7 The Middle Ear
Blausen
2/11/2020 Sound of Music 3 8 The Inner Ear
2/11/2020 Sound of Music 3 9 Auditory Transduction: Sound to Electrical Impulses
Cochlea Ear Video
Brandon Pletsch (2002) Medical College of Georgia
2/11/2020 Sound of Music 3 10 Detailed Look at the Cochlea
Basilar Membrane Jennifer Kincaid
2/11/2020 Sound of Music 3 11 Another Cartoonish look at ear….
Ear “Journey of Sound to the Brain” Video NIH - 2017 [Wikimedia|Cochlea]
Stereocilia
2/11/2020 Sound of Music 3 12 Detailed Look at the Organ of Corti
Ear Video
Outer Hair Cells
Inner Hair Cells
2/11/2020 Sound of Music 3 Jennifer Kincaid 13 Detailed Look at the Organ of Corti
Waves Coming Towards Us 3500 Inner Hair Cells Supercilia
Outer Hair Cell (Amplifier)
Inner Hair Cell (Sensing)
2/11/2020 Sound of Music 3 Jennifer Kincaid 14 Detailed Look at the Organ of Corti
Outer Hair Cell (Amplifier)
Inner Hair Cell (Sensing)
Nerve Fibers to Brain
2/11/2020 Sound of Music 3 Jennifer Kincaid 15 Tectorial Membrane Peeled Back
Electron Micrographs of Guinea Pig Organ of Corti [Prof. Andrew Forge]
2/11/2020 Sound of Music 3 16 Severe Damage
2/11/2020 Sound of Music 3 17 Outer Hair Cells Shake the Tectorial Membrane
Inner Hair Cell Supercilliae almost (but not quite) touching the Tectorial Membrane. Outer Hair Cells (attached to Membrane)
Inner Hair Cells
2/11/2020 Sound of Music 3 18 Dancing Outer Hair Cell with Stereocilia
Isolated Guinea Pig Outer Hair Cell with Patch Clamp
J. Santos-Sacchi Yale University
2/11/2020 Sound of Music 3 19 Unrolling the Cochlea
Response at one
2/11/2020 Sound of Music 3 middle frequency 20 Unrolling the Cochlea Animated Video of Basilar Membrane 0.1 mm
35 mm High f 10,000 x Stiffer!
0.5 mm Basilar Membrane Low f
Howard Hughes Medical Institute Sound of Music 3 2/11/2020 21 Traveling Wave on Basilar Membrane
Traveling wave moves slowly – ~1% of speed of sound in air
Relative Amplitudes 1600 Hz 800 Hz 400 Hz 200 Hz 100 Hz 50 Hz 25 Hz
Distance from Stapes (mm) But these Basilar Membrane responses were measured on “dead” Cochleas…
2/11/2020 Sound of Music 3 22 Traveling Wave on Basilar Membrane
Relative Amplitudes 1600 Hz 800 Hz 400 Hz 200 Hz 100 Hz 50 Hz 25 Hz
Distance from Stapes (mm)
2/11/2020 Sound of Music 3 23 Dancing Outer Hair Cells to the Rescue
Basilar Membrane Response Relative Amplitudes 400 Hz 1600 Hz 800 Hz 400 Hz 200 Hz 100 Hz 50 Hz With Outer 25 Hz Hair Cell Distance from Amplification Stapes (mm)
Outer Hair Cell Activation gives Inert Cochlea stronger and sharper signal for Inner Hair Cell sensors: Positive Feedback 0 10 20 30 Distance from Stapes (mm)
2/11/2020 Sound of Music 3 24 Dancing Outer Hair Cells to the Rescue
High f Basilar Membrane Response Low f Relative Amplitudes 1600 Hz 800 Hz 400 Hz 200 Hz 100 Hz 50 Hz With Outer 25 Hz Hair Cell Distance from Amplification Stapes (mm)
Critical Bands ▪ 25 bands across audible spectrum ▪ 1.3 mm wide along Basilar Membrane ▪ 150 Inner Hair Cells within a band ▪ Each Inner Hair Cell belongs to a Critical Band Inert Cochlea ▪ Frequency range of bands varies: o 100 Hz at low frequencies o 3000 Hz at high end 0 10 20 30 Distance from Stapes (mm) ▪ Important for understanding Harmony 2/11/2020 Sound of Music 3 25 Spontaneous OtoAcoustic Emission (SOAE)
• 24/7 • Frequencies are individual, per ear • At least 70% of people have SOE! • Can be loud enough to hear (rarely!)
③
②
①
2000
de Kleine et al JASA (2000) 2/11/2020 Sound of Music 3 26 SOAE is Similar to PA System Squeal…
2/11/2020 Sound of Music 3 27 Transient Evoked OtoAcoustic Emission (TEOAE)
• “Click” Stimulus evokes delayed emission • Works on everyone • Routine baby screen for ear function • Very High signal – no need for quiet booth
2/11/2020 Sound of Music 3 28 Hermann Helmholtz had it mostly right
Place Theory: Frequency perception is determined by distance along the Basilar Membrane
2/11/2020 Sound of Music 3 29 Hair Cells Fire Near Sound Wave Peak For Low frequencies (50-300 Hz):
2/11/2020 30 Sound of Music 3 Hair Cells Fire Near Sound Wave Peak For Medium Frequencies (500-5000 Hz):
Volley Theory: (Ernest Wever 1939) Multiple nearby hair cells taken together can send a spike on every cycle
2/11/2020 Sound of Music 3 31 The Decibel Scale of Sound Pressure Level
D=2m Units are Decibels (dB) 200
180 D=5m Each 20 dB → 10x Pressure 160 Each 10 dB → 10x Power
SPL 140 Each 10 dB → ~2x “Loudness” (dB) 120 → D=50m Each 1 dB Min Change (JND) 100 Just Noticeable Difference 80 70 60 D=2m
40 D=2m 20 D=2m 0 Min -20 2/11/2020 Sound of Music 3 32 Equal Loudness Contours (ISO 226:2003)
Units of Subjective
Loudness
are Phons [Phons = dB
@ 1kHz]
Old Data For 20-year olds! (1937) We’re 15-30dB
worse off
Frequency (Hz) 2/11/2020 Sound of Music 3 33 Hearing Threshold Drops with Age
2/11/2020 Sound of Music 3 34 (Edited)
A Day in The Life…
1967
0 5000 10000 15000 20000 Hz 2/11/2020 Sound of Music 3 35 (Edited)
A Day in The Life…
1967
0 5000 10000 15000 20000 Hz 2/11/2020 Sound of Music 3 36 Two Approaches to Understanding Musical Sound Perception
1. Follow the neurons from the ears onward - Bottom up
2. Look at the final perceptions of sound
- Top down
2/11/2020 Sound of Music 3 37 Simplified Auditory Pathways
5
4
1. Cochlea Sinauer Associates 3 2016 2
2/11/2020 Sound of Music 3 38 Simplified Auditory Pathways
6. Auditory Cortex
5. Medial Geniculate Body
4. Inferior Colliculus
1. Cochlea Sinauer Associates 3. Superior 2016 Olivary 2. Cochlear Complex Nucleus 2/11/2020 Sound of Music 3 39 to Left Medial to Right Medial Most Auditory Geniculate Body & Geniculate Body Auditory Cortex Brain Regions Inferior Colliculus are Tonotopic (at least partially)
Lateral Lemniscus
Cochlear Nucleus
Cochlea
Superior Olivary Complex
2/11/2020 Sound of Music 3 40 Including Auditory Cortex In Humans
fMRI: High Resolution Functional MRI on Human Subjects: While hearing tones of various frequencies!
Kamil Ugurbil et al (2015-2016) 2/11/2020 Sound of Music 3 41 “Typical” Neuron
Dendrites (inputs) Nerve Impulse
Cell Body Axon
Axon Terminals (outputs) Myelin Protective Sheaths
2/11/2020 Sound of Music 3 42 3D Anatomy of
Mouse Brain Reconstruction of All 89 Neurons (with axons)
Tiny Cube from Mouse somatosensory cortex
5 Example Neurons with axons
Motta et. Al. Max Planck Inst. for Brain Research (Nov 2019)
34,221 Axons Total length 2.7m
2/11/2020 Sound of Music 3 43 Using 2 Ears: Sound Localization in Superior Olive
Lateral Superior Olive (LSO) Neurons Compute Left-Right Intensity Difference (High f)
Medial Superior Olive (MSO) Neurons Compute Left-Right Arrival Time Difference (Low - Medium f)
2/11/2020 Sound of Music 3 44 Vertical Sound Localization via Frequency Notches in White Noise
Pinna— Highly Asymetric
2/11/2020 Sound of Music 3 45 Example of 3D Auditory Neural Spatial Organization: Small region in Cat Inferior Colliculus
“Pitch” Neuron 100 Hz Periodicity ~20 Hz ~500 Hz 1.8 kHz f 1.95 kHz → Gerald Langner, The Neural Code of Pitch and Harmony 2.1 kHz (2015) 30 Planar Layers, each receiving input 2.1→2.4 from a narrow section of the 2.4→2.8 Basilar Membrane
2.8→3.3
3.3→3.9
1/27/20 Sound of Music 1 46 OLLI-Vote 2020 Wands
NO MAYBE BAD YES ??? GOOD
2/11/2020 Sound of Music 3 47 Can We Hear Phases?
WaveGen
Fixed Harmonic Phases Hear the Difference?
C4 [262 Hz] YES NO
Random Harmonic Phases
3/2/2017 Hear All About It 48 What If We Combine Lots of Pure Tones?
300 and 300.3 Hz 4 All at once: 3 2 ① Phases: Random 1
0 300 Hz 0 5 10 15 20 25 30 35 40 45 -1 ② Phases: In Phase -2
-3 at Center
-4
P1 P2 1001 Tones: 599.7 and 600 Hz 4 300 to 600 Hz
3 3 sec long
2
1
0 600 Hz 0 5 10 15 20 25 30 35 40 45 -1
-2
-3
-4
P4 P5
3/2/2017 Hear All About It 49 Spectrograms for 1001 Tones
① Phases: Random
② Phases: In Phase at Center
3/2/2017 Hear All About It 50 So Why Can We Detect Phase in One Case … and Not the Other?
3/2/2017 Hear All About It 51 It’s the Basilar Membrane, Stupid
Critical Bands Relative Amplitudes 1600 Hz High f Low f 800 Hz Basilar Membrane Response 400 Hz 200 Hz 100 Hz 50 Hz 25 Hz
Thus individual Partials are detected by Distance from different hair cells and do not interfere. Stapes (mm)
Lower Harmonics Fall on 0 10 20 30 Different Critical Bands Distance from Stapes (mm)
1. Musical Note with Harmonics P6 P5 P4 P3 P2 P1
2. “Thousand Partial” Case All 1001 Closely Spaced Partials Fall on 1 or 2 Critical Bands Thus constructive/destructive interference 2/11/2020 Sound of Music 3 of Partials applies at each hair cell 52 Missing Harmonic Hardly Noticed…
AuditoryNeuroscience, from concept in NOW can you hear it? Neuweiler’s Hear the Difference? “Vergleichende 11 Tierphysiologie” 10 9 NO 8 YES 7 6 1200 Hz 5 4 3 2 6th Harmonic not 1 200 Hz noticed until it turns on and off – then heard as separate isolated tone 2/11/2020 Sound of Music 3 53 Pitch vs. Frequency in Complex Tones
For Simple sine wave tones, Pitch is directly determined by Frequency
Question: For Complex Tones, is Perceived musical Pitch determined simply by August Seebeck the Fundamental Georg Simon Ohm (1805-1849) .. or Lowest Frequency Component? Technische Universität Dresden (1789-1854) Polytechnic School of Nuremburg Siren or, is Pitch something quite different?
2/11/2020 Sound of Music 3 54 Pitch vs. Frequency in Complex Tones
For Simple sine wave tones, Pitch is directly determined by Frequency
Question: For Complex Tones, is Perceived musical Pitch determined simply by August Seebeck the Fundamental Georg Simon Ohm (1805-1849) .. or Lowest Frequency Component? Technische Universität Dresden (1789-1854) Polytechnic School of Nuremburg Siren or, is Pitch something quite different?
2/11/2020 Sound of Music 3 55 Pitch vs. Frequency in Complex Tones
For Simple sine wave tones, Pitch is directly determined by Frequency
Georg Simon Ohm (1789-1854) Question: Polytechnic School of Nuremburg For Complex Tones, is perceived musical Pitch determined simply August Seebeck by the Fundamental or Lowest (1805-1849) Frequency component? Technische Universität Dresden or, is Pitch something quite different? Herman von Helmholtz (1821-1894)
2/11/2020 Sound of Music 3 56 The Strange Case of the Missing Fundamental
Fundamental Only: No Harmonics Fundamental + 9 Harmonics No Fundamental: Only Harmonics
Composite Result
10 9 8 7 6 5 4 3 200 Hz Fundamental 2 1 No Fundamental 2.5 5 7.5 0.0110 s 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5
2/11/2020 Sound of Music 3 57 The Strange Case of the Missing Fundamental
Fundamental Only: No Harmonics Fundamental + 9 Harmonics No Fundamental: Only Harmonics
Composite Result
10 9 8 7 Same perceived 6 pitches in all cases, 5 although different 4 Timbres 3 200 Hz Fundamental 2 1 No Fundamental 2.5 5 7.5 0.0110 s 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5
2/11/2020 Sound of Music 3 58 The Strange Case of the Missing Fundamental
Fundamental + 9 Harmonics No Fundamental: Only Harmonics 4->10
10 9 8 7 6 5 Same perceived 4 pitch, although 3 different Timbre 2 1
2.5 5 7.5 10 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5
2/11/2020 Sound of Music 3 59 Phase Scramble Hear the Difference?
Fundamental + 9 Harmonics Fundamental + 9 Harmonics: Random Phases YES NO
10 9 8 7 6 5 4 3 2 1
2.5 5 7.5 10 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5
2/11/2020 Sound of Music 3 61 Phase Scrambled + Missing Fundamental
Partials 3-12: Phases Aligned Partials 3-12: Random Phases
12 11 10 9 8 7 6 5 4 3 600 Hz No 200 Hz Fundamental or 2 No Fundamental or 2nd Harmonic 400 Hz 2nd Harmonic 1 This high tone is the lowest 2.5 5 7.5 10 12.5 15 17.5 2.5 5 7.5 10 12.5 15 17.5 frequency present! 2/11/2020 Sound of Music 3 62 Missing Fundamental Fundamental in a Only Complete Melody
Fundamental + 9 Harmonics
Higher Harmonics Only
2/11/2020 Sound of Music 3 63 Absolute Pitch
Ability to quickly and accurately name the Pitch of a complex tone
• Fairly rare – 1 in 10,000 estimate in general population • Not to be confused with Relative Pitch • Odds go up if you – are musically trained (up to 4%) – were exposed to intensive musical training as a young child – have a tonal first language (e.g. Chinese, Vietnamese) – are on the autism spectrum – are named Mozart or John Phillip Sousa – are Synesthetic • Many non-musicians have good pitch recall
1/27/20 Sound of Music 1 64 Pitch Perception Test
Tone Pairs Pair 1 Pair 2 Pair 3 Pair 4 Test A (pure tones)
Test B (pure tones)
Test C (complex tones)
Test D (complex tones)
2/11/2020 Sound of Music 3 65
Spectrogram of Test A Frequency (Hz) Frequency
1 2 3 4
2/11/2020 Sound of Music 3 66 Pitch Perception Test
Tone Pairs Pair 1 Pair 2 Pair 3 Pair 4 Test A (pure tones)
Test B (pure tones)
Test C (complex tones)
Test D (complex tones)
2/11/2020 Sound of Music 3 67 Spectrogram of Test B
-2% +2% Frequency (Hz) Frequency -1% +3%
1 2 3 4
2/11/2020 Sound of Music 3 68 Pitch Perception Test
Tone Pairs Pair 1 Pair 2 Pair 3 Pair 4 Test A (pure tones)
Test B (pure tones)
Test C (complex tones)
Test D (complex tones)
2/11/2020 Sound of Music 3 69 Spectrogram of Test C
+1%
-6% Frequency (Hz) Frequency
-2% +2%
1 2 3 4
2/11/2020 Sound of Music 3 70 Pitch Perception Test
Tone Pairs Pair 1 Pair 2 Pair 3 Pair 4 Test A (pure tones)
Test B (pure tones) Test C (complex tones) Test D (complex tones)
2/11/2020 Sound of Music 3 71
Diana Deutsch’s Tritone Paradox Frequency (Hz) Frequency
1 2 3 4
2/11/2020 Sound of Music 3 72
Diana Deutsch’s Tritone Paradox Frequency (Hz) Frequency
1 2 3 4
2/11/2020 Sound of Music 3 73 Continuity Illusion
Series of beeps…
Noise
Noise Noise
Now concentrate on the beeps… Ignore the noise
2/11/2020 Sound of Music 3 74 Shepard-Risset Glissando
Infinite Descent….
2/11/2020 Sound of Music 3 75 Shepard-Risset Glissando
2/11/2020 Sound of Music 3 76 Risset’s Accelerando
Jean-Clause Risset (1938-2016) Composer, Bell Labs
2/11/2020 Sound of Music 3 77 Risset’s Accelerando
Jean-Clause Risset (1938-2016) Composer, Bell Labs
2/11/2020 Sound of Music 3 78 Course Outline
1. Building Blocks: Some basic concepts 2. Resonance: Building Sounds 3. Hearing Music and the Ear 4. Musical Scales 5. Musical Instruments 6. Singing and Musical Notation 7. Harmony and Dissonance; Chords 8. Combining the Elements of Music
2/11/2020 Sound of Music 3 79