This document is downloaded from DR‑NTU (https://dr.ntu.edu.sg) Nanyang Technological University, Singapore. The psychoacoustics and synthesis of singing harmony Chan, Paul Yaozhu 2020 Chan, P. Y. (2020). The psychoacoustics and synthesis of singing harmony. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/142516 https://doi.org/10.32657/10356/142516 This work is licensed under a Creative Commons Attribution‑NonCommercial 4.0 International License (CC BY‑NC 4.0). Downloaded on 10 Oct 2021 01:29:42 SGT The Psychoacoustics and Synthesis of Singing Harmony Paul Yaozhu Chan School of Computer Science and Engineering 2020 The Psychoacoustics and Synthesis of Singing Harmony Paul Yaozhu Chan A thesis report submitted to the School of Computer Science and Engineering in partial fulfilment of the requirements for the degree of Doctor of Philosophy June 2020 Authorship Attribution Statement This thesis contains material from 1 accepted peer-reviewed journal, 3 published conference papers and 3 filed patents, where I was the first and/or corresponding author/inventor. Chapter 2 has been accepted as Paul Yaozhu Chan, Minghui Dong and Haizhou Li, \The Science of Harmony: A Psychophysical Basis for Perceptual Tensions and Resolutions in Music," in Research, Submitted Aug 2018. The contributions of the co-authors are as follows: • I proposed the idea, designed the study, wrote the manuscript, and performed the experiments. • Dr Minghui Dong co-designed the study, and revised the manuscript. • Prof Haizhou Li co-designed the study, and revised the manuscript. Part of Chapter 3 has been published as Paul Yaozhu Chan, Minghui Dong, Grace X. H. Ho, and Haizhou Li, \SERAPHIM: A Wavetable Synthesis System with 3D Lip Animation for Real-Time Speech and Singing Applications on Mobile Plat- forms," in INTERSPEECH, pp. 1225-1229, 2016. The contributions of the co- authors are as follows: • I proposed the idea, designed the study, wrote the manuscript, and performed the experiments. iii • Dr Minghui Dong co-designed the study, and revised the manuscript. • Prof Haizhou Li co-designed the study, and revised the manuscript. Part of Chapter 3 has been published as Paul Yaozhu Chan, Minghui Dong, Grace X. H. Ho, and Haizhou Li, \SERAPHIM Live! Singing Synthesis for the Performer, the Composer, and the 3D Game Developer," in INTERSPEECH, pp. 1966-1967, 2016. The contributions of the co-authors are as follows: • I proposed the idea, designed the study, wrote the manuscript, and performed the experiments. • Dr Minghui Dong co-designed the study, and revised the manuscript. • Prof Haizhou Li co-designed the study, and revised the manuscript. Part of Chapter 3 has been patented as Paul Yaozhu Chan, Minghui Dong, Haizhou Li, \A Wavetable Synthesis System with 3D Lip Animation for Real-time Speech and Singing Applications on Mobile Platforms," Singapore Patent, Pub no. SG/P/2016002, Filed 2016. The contributions of the co-authors are as follows: • I conceptualized the idea, designed the study, wrote the patent, and per- formed the experiments. • Dr Minghui Dong co-conceptualized the idea, and revised the drafts. • Prof Haizhou Li co-conceptualized the idea, and revised the drafts. Part of Chapter 4 has been published as Paul Yaozhu Chan, Minghui Dong, Siu Wa Lee, Ling Cen, and Haizhou Li, \Solo to A Capella Conversion - Synthesizing Vocal Harmony from Lead Vocals," in Proceedings - IEEE International Confer- ence on Multimedia and Expo, 20111. The contributions of the co-authors are as follows: • I proposed the idea, designed the study, wrote the drafts of the manuscript, and performed the experiments. • Dr Minghui Dong co-designed the study, and revised the manuscript. 1This work started during application for candidature, which was before (but extended be- yond) the official commencement of candidature. iv Abstract The human singing voice is a remarkable instrument that compounds an immense amount of expressivity onto a single dimension. Apart from semantics and melody (pitch, duration and dynamics), accent, age, gender and emotion are all carried in the singing voice. While a single singing voice on its own is aesthetically pleas- ing to the ear, the addition of concurrent voices of different pitch is commonly known to be capable of producing a pleasing effect far greater than the sum of that produced by each contributing voice. This motivates the use of harmony in singing. Unfortunately, accompaniment voices are difficult to sing, even for profes- sional singers. Thankfully singing synthesis has made it viable for this task to be undertaken by machines. The overall objective of this thesis is to advance today's understanding of singing harmony and ultimately develop novel techniques for its synthetic reproduction. This is broken down into three parts. The first focuses on a psychophysical basis of harmony, the second focuses on the synthesis of the singing voice, while the third combines the first two to focus on the synthesis of harmonized singing. The first contribution is an attempt to find a psychoacoustic basis of harmony and presented in chapter2. Apart from stationary harmony ( chords, or sonorities: the aesthetics of a group of concurrent notes at one point of time), this also includes transitional harmony (chord progression, or resolution: the aesthetics of a similar group of notes progressing to another). In order to explain both stationary and transitional harmony, it introduces a theory of harmony based on the notions of interharmonic and subharmonic modulations. Acoustic measures of stationary and transitional harmony are proposed and the answers to five fundamental questions vi of psychoacoustic harmony are presented, both based on this theory. Correlations with existing music theory and perception statistics support this contribution with both stationary and transitional harmony. The second contribution is in the synthesis of the singing voice and presented in chapter3. Modern singing synthesis methods are at best capable of word- level runtime synthesis, with only two known ones dedicated to realtime synthesis. This means that they are applicable only towards offline music production. A large part of the art of music and singing, however, is in realtime performance. With both of the existing realtime singing synthesis methods bounded by a phone- coverage to realtime-capability tradeoff, a need for one that overcomes it remains. A novel realtime singing synthesis system, SERAPHIM, is proposed as an answer to this. Apart from overcoming this phone-coverage to realtime-capability trade- off, subjective listening tests also showed that listeners preferred voices synthesized by SERAPHIM as opposed to other realtime systems. The third contribution is in the synthesis of singing harmony and presented in chapter4. With this contribution, a novel method for singing harmony synthesis is proposed. Current implementations can be classified into pitch-inaccurate rule- based systems, timing-inaccurate inference-based systems, and hybrid systems that trade off between pitch inaccuracies and timing inaccuracies. This means that existing systems are vulnerable to either pitch errors, timing errors or both in different degrees of compromise. The challenge in the task was to overcome this compromise to develop a robust technique that is simultaneously resilient to both pitch and timing errors while producing harmonious accompaniment. Our strategy was to leverage on the pitch-accurate inference-based method while eliminating timing inaccuracies by use of machine-synchronization. Spectrograms revealed that harmonized voices produced by this method contain the least dissonances amongst existing methods. Subjective listening tests also showed that harmonized voices produced by this method are perceived to be the best sounding, both by vocal experts and by casual listeners. All in all, the work presented in this thesis contributes to the advancement of the psychoacoustic understanding and machine synthesis of singing harmony across vii one journal paper, three conference papers and three patents. viii Acknowledgements I would like to thank my supervisors, A/Prof Eng Siong Chng (Nanyang Tech- nological University), Prof Haizhou Li (National University of Singapore) and Dr Minghui Dong (Institute for Infocomm Research) for giving me the opportunity to undertake this research and nurturing me in my career as a researcher while giving me the much-needed space to learn and grow. Further to this, I would like to thank my fellow colleagues and students across the Institute for Infocomm Research, Nanyang Technological University and National University of Singapore for the friendship and the camaraderie and for always cheering me on. Especially Ms Aiti Aw for allowing me to work in a parallel field as my research; colleagues at One North Christian Fellowship such as Ms Susan Yap, Dr Yi Yan Yang, Dr Peter Yu Chen and Dr Francois Chin for constantly upholding me in prayer; and the LabRats, the unofficial band of the Agency for Science, Technology and Research, for helping me keep my sanity though the series of gigs and events. Finally, I would like to thank my parents back home, Ron and Lili; my wife, Jing; and daughter, Dawn, for their love, support, encouragement and prayers. ix Contents Authorship Attribution Statement ................... iii Abstract ................................... vi Acknowledgements ............................. ix List of Publications ............................ xiv List of Figures ............................... xix List of Tables ................................ xx 1 Introduction1 1.1 Motivation and Scope . .2 1.2 Background . .2 1.3 Contributions . .6 1.3.1 Psychoacoustics of Harmony . .6 1.3.2 Singing Synthesis (SERAPHIM) . .7 1.3.3 Vocal Harmony Synthesis . .8 1.4 Organisation of Thesis . 10 2 The Psychoacoustics of Harmony 11 2.1 Background . 12 2.1.1 Existing Work . 13 2.1.2 Scope . 15 2.2 A Psychophysical Basis of Harmony . 17 2.3 Modulations in Sinusoidal Summation . 18 2.4 Interharmonic Modulations . 22 2.4.1 Beating Frequencies and Low-Frequency Modulations . 22 2.4.2 Perceptual Responses across the ∆f-f¯ feature Space . 23 x 2.4.3 Intervals and Second-Order Modulations on the ∆f-f¯ fea- ture Space .