MUSIC IN REAL LIFE VS MUSIC IN DIGITAL WORLD
AIST2010 Lecture 2 SOUND AND DIGITIZING SOUND MUSICAL MUSIC HARDWARE MUSIC IN REAL LIFE AND MUSIC DATA INSTRUMENTS AND SOFTWARE
OUTLINE
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 2 SOUND AND MUSIC IN REAL LIFE
Vibration and sound Music The vibration of air particles from A clever combination of sounds and the source into your ears voids Or via other media (e.g. water) Vibrations detected by human auditory system Representing sounds è “audio” signals It’s always an abstraction, e.g. Different vibrations give rise to Waveform (amplitude vs. time) different kinds of sounds Spectrum (frequency vs. amp. vs. time) Some are harmonious, some are noise, some are high, some are low
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 3 Video from: https://youtu.be/XBP0jmpjzBw
VIBRATION
Image from: Fund. of Music Processing, p.20 For periodic signals, frequency measured in Hz à cycle per second (cps)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 4 frequency amplitude amplitude time AIST2010 L2 L2 AIST2010 WAVEFORM SPECTROGRAM – MUSICIN REAL LIFE VS. MUSIC IN DIGITAL WORLD spectrum too spectrum waysrepresenttheto areThereother Note: 5 BASIC PERIODIC WAVEFORMS
A pitch is heard for periodic waveforms (at certain frequencies) Sine wave Cosine wave Phase-shifted sine wave Triangular wave Sawtooth wave Image from: https://moogfoundation.org/learning-synthesis/synthesis-fundamentals/ Square wave Commonly used in vintage synthesizers (i.e. electronics)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 6 HUMAN EAR AND PSYCHOACOUSTICS
Human sound perception is subjective General range: ~20 Hz – ~20,000 Hz (20 kHz) Hearing range narrows with age, especially at high frequencies Image from: Fund. of Music Processing, p.26 How well do animals hear? https://lsu.edu/deafness/HearingRange.html Human is more sensitive to ~2 to 4 kHz A not-so-flat “frequency response” For the sake of human communication? Perception of loudness is also affected by age Equal loudness contours
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 7 MEASURING LOUDNESS
Image from: Fund. of Sound power = (amplitude)2 Music Processing, p.25 Sound intensity = sound power per unit area (watt per sq. metre) “Threshold of hearing” (TOH) vs “Threshold of pain” (TOP) ()* * !"#$ ≝ 10 W/m TOP is roughly 1013 times of the intensity of TOH
Note: Power depends on distance!
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 8 MEASURING LOUDNESS
Image from: Fund. of Decibel (dB) is a logarithmic unit to measure the ratio Music Processing, p.25 ! dB ! ≝ 10 ⋅ log)4 !"#$ i.e. TOP is 130 dB when TOH is 0 dB This value relative to TOH is also called the sound pressure level (SPL), i.e. dBSPL
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 9 MEASURING LOUDNESS
Besides the intensity, the amplitude (which we discuss more) can also be represented in dB, usually we assume amplitude is 1 at 0dB This term is 20 because 567 8 = 20 ⋅ log)4 8 intensity is amplitude squared Do not mess up with:
dBA or dB(A) represents the loudness perceived by human ear Compensation using the A-weighting basing on equal-loudness curves
Image from: https://en.wikipedia.org /wiki/A-weighting 10 MEASURING PITCH
Frequency range on a piano keyboard An octave Freq (Hz) Freq 29.135 34.648 38.891 46.249 51.913 58.27 69.296 77.782 92.499 103.83 116.54 138.59 155.56 185 207.65 233.08 277.18 311.13 369.99 415.3 466.16 554.37 622.25 739.99 830.61 932.33 1108.7 1244.5 1480 1661.2 1864.7 2217.5 2489 2960 3322.4 3729.3
C1 C2 C3 D3 E3 F3 G3 A3 B3 C4 D4 E4 F4 G4 A4 B4 C5 C6 C7 C8 Freq (Hz) Freq 27.5 30.868 32.703 36.708 41.203 43.654 48.999 55 61.735 65.406 73.416 82.407 87.307 97.999 110 123.47 130.81 146.83 164.81 174.61 196 220 246.94 261.63 293.66 329.63 349.23 392 440 493.88 523.25 587.33 659.26 698.46 783.99 880 987.77 1046.5 1174.7 1318.5 1396.9 1568 1760 1975.5 2093 2349.3 2637 2793.8 3136 3520 3951.1 4186 “Middle C” = 261 Hz “A440” = 440 Hz Double frequency = “octave” higher The next occurrence of the same letter-name in music
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 11 MEASURING PITCH
Pitch, like dynamics, is perceived in a logarithmic manner Feeling of doubling amplitude The change from 0.25 to 0.5 is similar to that from 0.1 to 0.2, a doubling of volume Feeling of doubling frequency The change from 100 to 200 Hz is similar to that from 250 to 500 Hz, an octave higher Perceptual scales of pitch Mel-scale Critical bands (Bark scale) Empirical scales! Equivalent rectangular bandwidth (ERB) rate log Frequency
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 12 ASPECTS OF MUSIC
When sounds and rests are cleverly Dynamics put together, we have music the expression in loudness Melody Structure combination of pitches (at different time) how music sections are arranged Rhythm Texture combination of long and short sounds the amount of different/similar sounds at times Harmony (chords) Timbre combination of pitches (at the same time) different shapes of waveforms Harmonic progression è different spectrum combination of chords (at different time) There can be more!
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 13 DIGITIZING SOUND AND MUSIC
How can we represent audio/sound/music in a computer? Two main ways Faithful representation of what can be heard: audio recording and playback To improve the audio representation, research is on the improving sound fidelity Reconstruction of music data (e.g. melody, rhythm, etc.): audio analysis and resynthesis To improve the symbolic representation, research is on the analysis and synthesis methods
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 14 STORING AUDIO DATA ADC DAC
To represent audio signals in the digital world, digitizing is needed ADC (analog-digital conversion) for recording DAC (digital-analog conversion) for playing back Sampling: the representation in time The higher sampling rate the better CD quality: 44.1 kHz è 44100 samples every second Quantization: the representation in amplitude The more dynamic levels the better CD quality: 16-bit è 216=65536 different levels of amplitude
Image from: Comp. Music Instruments, p.21
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 15 SAMPLING RATE
How many samples per second is the best? Storage requirements vs. sound quality Sampling Theorem 440Hz signal at 22,050Hz sampling rate Signal reconstructable if frequency is not higher than Nyquist frequency: Ω = <=/2, where FS is the sampling rate Otherwise artifacts called aliasing will form (an extra freq. component) Common rates Analog telephone: 8,000 Hz Pro digital video equipment: 48,000 Hz Low quality audio: 22,050 Hz DVD-Audio, Blu-ray audio: 96,000 Hz Audio CD, MP3: 44,100 Hz Super-Audio CD: 2,822,400 Hz
Image from: Fund. of Music Processing, p.62 Aliasing occurs at (b) and (c) DYNAMIC LEVELS
The dynamic level is controlled by the bit depth Typically 8-, 16-, and 24-bit, corresponding to 28, 216, and 224 levels Quantization error between analog value and quantized value Represented as “Amplitude” Integer vs. floating Signed vs unsigned Common bit depths Audio CD: 16-bit (~96 dB of range) Image from: Fund. of Rounding errors in quantization Pro audio: 24-bit (~144 dB) Music Processing, p.61
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 17 AUDIO CHANNELS
Different audio perception on our left and right ear helps determining the location of the sound source “Stereo” sound, with 2 channels Some professional recordings are shaped with Head Related Transfer Function (HRTF) Surround sound systems 5.1 or 7.1 with subwoofer Extra channel = extra storage Dummy head microphone for immersive sound experience Image from: https://lmkprod.com/what-is- HKBU LIATe theatre with hrtf-brief-explanation/
24.2-channel sound system AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 18 Image from: http://liate.hkbu.edu.hk/theatre.html STORING AUDIO DATA
The raw audio data, bit by bit, is called linear pulse-code modulation “linear PCM” encoding (usually as .wav files) Good enough for sound editing and manipulation Consuming too much storage CD quality: 44.1kHz 16-bit 2-channel = 80 minutes consuming ~700 MB Compressed audio can fulfil requirements of the general public Lossy: MP3, AAC, WMA, … Lossless: FLAC, ALAC (.m4a), … Audio quality often measured as bandwidth requirement Spotify “normal quality”: AAC format at ~96 kbps
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 19 Image from: https://en.wikipedia.org/wiki/MIDI
DEALING WITH MUSIC DATA
Musical Instrument Digital Interface (MIDI) Standardized in 1983 Widespread technology to connect various musical hardware and software Control data instead of actual audio data Symbolic representation Playback = Resynthesis on-the-fly Basic messages: “Note On/Off, note number, velocity” Only decides when and what should happen, but not HOW Synthesizer will make the actual sound (e.g. a violin sound, or a piano sound)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 20 BASIC MIDI MESSAGES
“Note On” denotes the starting point, where “Note Off” is to stop Calculating the note number (0-127) D > = 69 + 12×log ( ) 2 EE4 $F MIDI 22 25 27 30 32 34 37 39 42 44 46 49 51 54 56 58 61 63 66 68 70 73 75 78 80 82 85 87 90 92 94 97 99 102 104 106
C1 C2 C3 D3 E3 F3 G3 A3 B3 C4 D4 E4 F4 G4 A4 B4 C5 C6 C7 C8 MIDI 21 23 24 26 28 29 31 33 35 36 38 40 41 43 45 47 48 50 52 53 55 57 59 60 62 64 65 67 69 71 72 74 76 77 79 81 83 84 86 88 89 91 93 95 96 98 100 101 103 105 107 108
Velocity (0-127): How fast the key is pressed (correlates to playing intensity)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 21 MIDI CHANNELS
16 MIDI channels on a MIDI connection Driving 16 different instruments simultaneously Each having their own set of notes (note number, velocity, on/off) More than one note can happen at one time in a channel, but note on/off will correspond to note number “Polyphony” What if you want a whole orchestra?
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 22 DEALING WITH MUSIC DATA
After “noteon”, the note can be further modified, e.g. with 7-bit values of Pitch bend Modulation As always, the result depends on the synthesizer! Aftertouch/Pressure Control change (CC) Damper pedal, sostenuto, expression, portamento, … https://www.midi.org/midi/specifications-old/category/reference-tables MIDI 2.0 draft is just announced in January 2019… Increasing bit precision: finer control of music expression
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 23 ...
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 24 REAL WORLD MUSICAL INSTRUMENTS
How are audio signals generated in real life? Always vibrations… Musical instruments are made to create regulated vibrations Usually with a controllable pitch (periodic waveform) Different instruments show different spectra (frequency components) The vibration may be amplified with an acoustic body, e.g. Sound box of a violin Sound board of the piano
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 25 REAL WORLD MUSICAL INSTRUMENTS
Piano è Chordophones Keys driving hammers to hit steel strings and make them vibrate
Strings è Chordophones Note: We are not Friction between bow and strings create vibrations going to cover physics in this course! Winds è Aerophones Air flow creates vibrations inside the tube of various length Percussion è Membranophones/Idiophones Too many possibilities! In general, an object vibrates when being struck
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 26 ELECTRONIC MUSIC INSTRUMENTS
Since the development of electricity, musicians started to explore into new ways of expressing themselves, e.g. Telharmonium (1897) Tonewheels generates electric musical notes with electromagnets Theremin (1920) Frequency/amplitude of the sound is controlled using the capacitance between hands and antennae Analog synthesizers (1960 beyond) Sounds are generated and modified using op-amp, resistors, filters, and other electronics Video from: https://youtu.be/pSzTPGlNa5U
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 27 INTERFACING WITH COMPUTERS
Audio interfaces “Sound cards”: providing signal conversion from analog to digital, and vice versa DAC + ADC MIDI interfaces Input/output of MIDI messages between computers and instruments Control surfaces Providing faders and buttons for direct control over DAW software UI
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 28 MIDI CONTROLLERS
With computers, now we can easily make music with software programs MIDI = prominent tool for music MIDI controllers are the interface for musicians to make MIDI messages Most controllers are in the shape of a piano keyboard 88-key, 72-key, … Weighted keys for realism of touching and expressiveness Other shapes Winds, mallet percussion, drums, strings, …
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 29 NOVEL DESIGN OF INSTRUMENTS
There is a research conference for everything… “NIME — The International Conference on New Interfaces for Musical Expression” (since 2001): https://www.facebook.com/NIME2019 Check out publications if you are curious https://www.nime.org/archives
Roli Seaboard, LinnStrument, … Multi-dimensional, polyphonic
Image from: https://www.nime.org/past-nimes/
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 30 MUSIC SOFTWARE
Synthesizers Earliest software synthesizers simply emulate the analog synthesizers Now there are more complex kinds of synthesis which is only made possible by computer programs Keyboard workstations are controllers with an embedded system for synthesis Sometimes exists as plugins for DAWs Vocaloid A special kind of sound synthesizer, which can mimic human voice by speech synthesis with controls of pitch and other vocal expressions
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 31 MUSIC SOFTWARE
Digital Audio Workstation (DAW) Allowing musicians to put together the tasks for recording, editing, and producing of audio data Most of DAW supports VST plugins as a standard way to extend features To name a few: Pro Tools, Logic Pro, Reason, GarageBand Live Performance Besides using keyboard workstations, a computer with proper software can be used for live performance with maximum flexibility with MIDI controllers and VST plugins Examples: MainStage, Ableton Live
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 32 MUSIC NOTATION AND RECOGNITION
Sheet music is still a major medium among musicians Notation software allow easy typesetting with computer keyboard or MIDI controllers Examples: Sibelius, Finale, MuseScore Some DAW can also output typeset music While text can be recognized by OCR technology, handwritten or printed music can also be imported with proper recognition software More and more popular on mobile devices
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 33 LECTURE REVIEW
We have discussed: How sounds are produced, and perceived by human Ways to digitize sound and music into computer systems Audio vs. symbolic representations Music instruments and controllers Music hardware and software
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 34 READ FURTHER
Section 1.3 “Audio Representation”, Fundamentals of Music Processing Chapter 1 “Audio and Music Signals”, Computer Music Instruments Section 8.1.1 “The MIDI protocol”, Computer Music Instruments
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 35