1/18/2019

18 January 2018

History and Future of Processing ELEC-E5620 - Audio , Lecture #1

Vesa Välimäki Lab, Dept. Signal Processing and Acoustics, Aalto University

Course Schedule for 2019 (Periods III-VI)

0. General issues (Vesa & Benoit) 11.1.2019 1. History and future of audio DSP (Vesa) 18.1.2019 2. Digital filters in audio (Vesa) 25.1.2019 3. design (Vesa) 1.2.2019 4. Analysis of audio signals (Vesa) 8.2.2019 5. Audio effects processing (Benoit) 15.2.2019 * No lecture (Evaluation week for Period III) 22.2.2019 6. Synthesis of audio signals (Fabian) 1.3.2019 7. and 3-D (Benoit) 8.3.2019 8. Physics-based sound synthesis (Vesa) 15.3.2019 9. Sampling rate conversion (Vesa) 22.3.2019 10. Audio coding (Vesa) 29.3.2019

Vesa Välimäki January 18, 2019 2

1 1/18/2019

Demo Grading

• Presentation (60%): Communication and accessibility, sound examples, visualization and animations, creativity, slides • Report (40%): Readability, code quality, results, complexity, figures, references

Benoit Alary & Vesa Välimäki January 18, 2019 3

Outline

• The most important events in the history of DSP and its applications to audio, acoustics, and music • Publications • Inventions • Products • Standards • Some predictions of the future

January 18, 2019 Vesa Välimäki 4

2 1/18/2019

Old Times

• Alexander Graham Bell, , 1876 • Elisha Gray, ‘musical telegraph’, 1876 • T. A. Edison, , 1877- • Thaddeus Cahill, telharmonium, 1897...1906 • US commissioner of patents (1899): “Everything that can be invented has been invented” • Lee De Forest, triode vacuum tube ( ), 1906 • Leon Theremin, theremin, 1919 • Rice & Kellogg, dynamic , 1925 • Talking movies, 1925 • Laurens Hammond, electric organ, 1929...1935

January 18, 2019 Vesa Välimäki 5

Phonograph

• Invented by Edison in 1877 • Birth of sound recording • Storage of sound as a mechanical waveform

Source: http://www.americaslibrary.gov/page s/aa_edison_phonograph_1.html

January 18, 2019 Vesa Välimäki 6

3 1/18/2019

Theremin

• Air violin based on two antennas • Amplitude and frequency control • Heard on soundtracks of old sci-fi films and in pop music

January 18, 2019 Vesa Välimäki 7

1948: Birth of Signal Processing

• , “A mathematical theory of communication,” Bell System Tech. Journal, 1948 • Oliver, Pierce & Shannon, “The philosophy of PCM,” 1948 (PCM = Pulse Code ) • Richard W. Hamming, error-correcting codes, 1948 • Bell Lab: transistor, 1948 • Pierre Schaeffer: musique concrète, 1948 • Two professional societies were founded:  IRE Audio Group was founded by Leo Beranek & others (since 1963: IEEE Signal Processing Society)  Audio Engineering Society was founded in 1948

January 18, 2019 Vesa Välimäki 8

4 1/18/2019

1950s

• Ragazzini & Zadeh: the Z transform, 1952 • Texas Instruments: development of a digital processor for seismic data, 1956 • First graphic equalizer: Langevin Model EQ-251A • Electric guitars: Fender Telecaster, 1950; Stratocaster 1954 • Stereo recording, 1956 • Sony: small transistor TR-63, 1957 • Max Mathews: music experiments, 1957- • Stereo LP record, 1958 • Koss Corporation: Headphones (“stereophones”), 1958 • Kilby, Noyce: integrated circuit, 1959

January 18, 2019 Vesa Välimäki 9

Sony TR-63 Transistor Radio

• An early “pocketable” transistor radio • Popular in the late 1950s • Birth of personal electronics! • Pocket calculator • Walkman • Mobile phone •MP3 player •PDA • Tablet computer •etc. http://www.geocities.com/SoHo/Atrium/1031/trans/cool/Sonys.html

January 18, 2019 Vesa Välimäki 10

5 1/18/2019

1960s: Early DSP Algorithms

• Manfred Schroeder: digital reverberation, 1961 • Atal & Schroeder: 3-D sound with 2 loudspeakers, 1962 • Moog: , 1964 • James Cooley & John Tukey: , 1965 • J. Flanagan & R. Golden: phase vocoder, 1965 • The first DSP text book by Gold & Rader, 1969

January 18, 2019 Vesa Välimäki 11

3D Sound Using a Loudspeaker Pair

Patent by Schroeder & Atal, 1963 • Hear sound from any direction with just two speakers • Within a restricted listening area • Invention: cross-talk cancellation!  Each ear receives only one loudspeaker signal, like in binaural listening  Implemented with digital filters

January 18, 2019 Vesa Välimäki 12

6 1/18/2019

Moog

• Analog electronics • Subtractive synthesis  Periodic waveforms and filters • Popular in pop and and in commercials  Wendy Carlos: Switched-on- Bach (1968)

January 18, 2019 Vesa Välimäki 13

1970s: Practical Applications

• Bishnu Atal: , 1970 • Blesser & Lee: real-time digital unit, 1971 • First digital reverb: EMT Model 250, 1976 • BBC: Digital sound recording started in 1971 • E. Kurenniemi: DIMI-600, 1972 • Chowning: , 1973 • Stockham: Restoration of old Caruso recordings, 1975 • recording for consumers  U-matic video cassette + a Sony PCM-1600 adapter, 1978 • First samplers: Synclavier, 1975; Fairlight CMI, 1978

January 18, 2019 Source of figure: Vesa Välimäki http://mixonline.com/TECnology-Hall-of- 14 Fame/1976-EMT-reverb/

7 1/18/2019

Fairlight Instrument

• Early digital synthesizer Fairlight CMI IIx (1982) and sampler (1980-1984)  Included a sequencer • Expensive (30000 USD) • Used in film industry and pop music in 1980s  E.g. Kate Bush, Thomas Dolby, Peter Gabriel, Jan Hammer, Stevie Wonder

(source: www.obsolete.com/120_years)

January 18, 2019 Vesa Välimäki 15

Fairlight CMI Preset

• Typical samples –Drums – A ”trained” dog – Human voices – Orchestra – Sounds from nature

(source: http://120years.net)

January 18, 2019 Vesa Välimäki 16

8 1/18/2019

Synclavier

• Manufactured by New England Digital (USA 1976-1992) • Digital synthesizer and sampler  Additive and FM synthesis  Spectrum analyzer • Extremely expensive • Used in the film and pop music industry  E.g., Michael Jackson, Sting, Frank Zappa

(source:http://120years.net)

January 18, 2019 Vesa Välimäki 17

1980s: Audio DSP Became Mainstream

• Texas Instruments, Intel, NEC, Motorola: first digital signal processors, 1980- • Sony: Walkman cassette player, 1980 • Philips & Sony: , 1982 • Digital Interface, MIDI, 1983 • Popular digital music synth: , 1983 • Sony: Digital Audio Tape (DAT), 1984 • First digital mixing consoles, 1986 • Ingrid Daubechies: wavelets, 1988

January 18, 2019 Vesa Välimäki 18

9 1/18/2019

CD

• Developed by Philips and Sony since late 1970s • Brought audio DSP into homes since mid-1980s • Contains much digital signal processing  Decoding, error correction, , digital filters, DA converters

Sony prototype (1981) Sony CDP-101 (1982)

January 18, 2019 Vesa Välimäki 19

Yamaha DX-7 Music Synthesizer

• The first affordable digital music synthesizer (2000 €) • FM synthesis only  Good for metallic, clear sounds, such as electric piano, tubular bells, celesta etc. • Huge commercial success in mid-1980s

(source: www.obsolete.com/120_years)

January 18, 2019 Vesa Välimäki 20

10 1/18/2019

1990s: Multi-Channel Sound and MPEG

• International standards by Moving Picture Experts Group (MPEG)  MPEG-1, MPEG-2, and MPEG-4 standards for video and audio coding  MPEG-1 layer 3 = MP3, 1990; popular since late 1990s • New music players: Philips DCC, 1992-1996; Sony MiniDisk, 1992- • Digital multichannel sound for movies and home theater (5.1), 1992 • 3-D audio reproduction using HRTFs, ca. 1992 • First applications of active noise control • Physical modeling synthesis: Yamaha VL-1, 1994 • 1995: The Internet changed everything! (Not immediately) • Steinberg, VST plug-ins (Virtual Studio Technology), 1996 • First MP3 players, 1998 • Auto-tune for the singing voice, 1998 • European (Digital Audio , DAB)

January 18, 2019 Vesa Välimäki 21

Korg Prophecy Synthesizer

• A physical and analog modeling synthesizer (1995)

January 18, 2019 Vesa Välimäki 22

11 1/18/2019

2000-2009: Mobile Audio

• More disk formats: Super Audio CD, DVD Audio, Blu-ray • New audio coding techniques  WMA, mp3PRO, AAC+, AMR-WB+, and about 100 others • Apple iPod music player, 2001 • MPEG-7 standard, “Multimedia Content Description Interface”, 2001  How to search and manage digitized audiovisual content  Requires high-level audio processing (sound recognition, classification,…) • MPEG-21 standard, “Networked Multimedia Framework”  Managing the whole multimedia delivery and consumption chain • Nokia: Polyphonic ring tones, 2002 • Sony-Ericsson: Music phones, 2005 • Music games: SingStar, 2004; GuitarHero 2005; RockBand 2007 • More music files sold than CDs, 2008-

January 18, 2019 Vesa Välimäki 23

2010-2019

• Group work • Please brainstorm with the person next to you. Try to list products/inventions related to audio signal processing that have appears during this decade. • How has the world changed? What’s possible now but wasn’t earlier?

January 18, 2019 Vesa Välimäki 24

12 1/18/2019

2010-2019

Vesa Välimäki January 18, 2019 25

Current Audio DSP Applications

• Phones, music & media players (DVD, MP3) • Loudspeakers & headphones • & gaming • Music synthesizers & effects boxes • Digital radio (DAB) & TV broadcasting • Cinema, home entertainment & car audio • Hearing aids • Professional audio • Active noise control in headsets and cars

January 18, 2019 Vesa Välimäki 26

13 1/18/2019

Turning Points in Audio History

• Birth of electroacoustics (ca. 1925)  , loudspeaker, amplifier, radio, sound recording, talking movies, PA systems • Solid state electronics (ca. 1955)  Transistors and diodes replace tubes, audio electronics becomes affordable and small • Digital audio (ca. 1980)  Sound becomes data  CD, DAT, MP3 • Virtual audio technology (ca. 1992)  Virtual audio reality, interaction, immersion, simulations, spatial sound reproduction using loudspeakers or headphones • Major changes every 15-30 years

January 18, 2019 Vesa Välimäki 27

Future of (Music) Listening

• Music like water  Wireless internet with solved billing routine • Integration of phone, music player & hearing aid  Audio entertainment and information, binaural telephony, aids for handicapped, or superman’s hearing • Intelligent music companion  Query device, automatic music identification, personalization, interactive disco, social networks • New reproduction methods  Headphones with haptics, bonephones, augmented-reality glasses, video-headphones, brain-computer interface

January 18, 2019 Vesa Välimäki 28

14 1/18/2019

Future of Audio Signal Processing

• Many devices will incorporate a processor, a microphone, and a loudspeaker • More intelligent processing of audio signals  Auditory scene analysis, source separation, etc. • Digitization of audio technology will continue  Digital (IP) loudspeakers and class-D amplifiers  Software versions of all effects boxes and instruments  More robotics in music: synthetic singers and guitarists? • New MPEG Interactive Music Applications Format (IMAF) standard, 2009  Enables personalized mixes of music

January 18, 2019 Vesa Välimäki 29

Acoustic Telepresence

• You need a stereo headset with mic, wireless full-duplex connection, and 3-D sound processing  Requires a signal processor  Pseudo-acoustic environment = normal + transmitted sounds

• Many new applications Bidirectional  Virtual tourist guide (requires GPS) binaural headset  Virtual and synthetic sound objects (e.g., sound calendar, audio post-it notes)  Hearing aid, hearing enhancer (hearwear)  Hearing protection  Voice control Ref: Härmä et al., 2004

January 18, 2019 Vesa Välimäki 30

15 1/18/2019

Grand Challenges for Audio DSP

• Compare to grand challenges in other fields  A chess program that beats humans – achieved  The translating telephone – in the making • Natural-sounding music synthesis  Parametric modeling of musical instruments • Computer imitation of famous singers  For example Elvis • Unmixing of audio files  For example, reconstruct multi-track versions of Beatles songs • A computer that understands music like humans  And how to go beyond that?

January 18, 2019 Vesa Välimäki 31

Bibliography

• D. A. Bohn, “Operator adjustable equalizers: An overview,” in Proceedings of the Audio Engineering Society 6th International Conference, pp. 369–381, 1988. • R. J. Burgess, The History of Music Production. Oxford University Press, 2014. • J- Chadabe, Electric Sound: The Past and Promise of Electronic Music,Prentice- Hall, 1997. • A. Härmä et al., “Augmented reality audio for mobile and wearable appliances,” Journal of the Audio Engineering Society, June 2004. • M. Kahrs (editor), “The past, present, and future of audio signal processing,” IEEE SP Magazine, Sept. 1997. • S. P. Lipshitz, “Dawn of the digital age,” Journal of the Audio Engineering Society, Jan./Feb. 1998. • F. Nebeker, Signal Processing: The Emergence of a Discipline, 1948 to 1998,IEEE History Center, 1998. • P. Noll, “MPEG digital audio coding,” IEEE Signal Processing Magazine,1997. • C. Roads, “Interview with Max Mathews,” Computer Music Journal,vol.4,no.4, winter 1980. • M. Vail, Vintage Synthesizers, Miller Freeman, 1993.

January 18, 2019 Vesa Välimäki 32

16 1/18/2019

Links

• History according to the Audio Engineering Society http://www.aes.org/aeshc/docs/audio.history.timeline.html • History of electronic musical instruments http://120years.net/ • A. Hutchinson, “The Top 50 Inventions of the Past 50 Years,” Popular Mechanics, Dec. 2005. http://www.popularmechanics.com/technology/gadgets/news/2078467 • C.E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal, 1948. http://onlinelibrary.wiley.com/doi/10.1002/j.1538-7305.1948.tb01338.x/abstract

January 18, 2019 Vesa Välimäki 33

Course Schedule for 2019 (Periods III-VI)

0. General issues (Vesa & Benoit) 11.1.2019 1. History and future of audio DSP (Vesa) 18.1.2019 2. Digital filters in audio (Vesa) 25.1.2019 3. Audio filter design (Vesa) 1.2.2019 4. Analysis of audio signals (Vesa) 8.2.2019 5. Audio effects processing (Benoit) 15.2.2019 * No lecture (Evaluation week for Period III) 22.2.2019 6. Synthesis of audio signals (Fabian) 1.3.2019 7. Reverberation and 3-D sound (Benoit) 8.3.2019 8. Physics-based sound synthesis (Vesa) 15.3.2019 9. Sampling rate conversion (Vesa) 22.3.2019 10. Audio coding (Vesa) 29.3.2019

Vesa Välimäki January 18, 2019 34

17