Perceptual Quality Dimensions of Text-To-Speech Systems
Total Page：16
File Type：pdf, Size：1020Kb
INTERSPEECH 2011 Perceptual Quality Dimensions of Text-to-Speech Systems Florian Hinterleitner1, Sebastian Möller1, Christoph Norrenbrock2, Ulrich Heute2 1Quality and Usability Lab, Deutsche Telekom Laboratories, TU Berlin, Germany 2Digital Signal Processing and System Theory, CAU Kiel, Germany {florian.hinterleitner, sebastian.moeller}@telekom.de, {cno, uh}@tf.uni-kiel.de Abstract The aim of our research is to assess the inherent quality dimen- The aim of this paper is to analyze the perceptual quality dimen- sions of several state-of-the-art TTS systems. This will ensure sions of state-of-the-art text-to-speech systems (TTS). There- a deeper insight into how test subjects perceive modern TTS fore, several pretests were conducted to determine a suitable set quality. Our study follows the approach presented in [8] which of attribute scales. The resulting 16 scales were used in a se- analyzed preceptual quality dimensions of modern telephone mantic differential on a diverse database containing 16 different connections via different multidimensional analysis techniques. TTS systems. A subsequent multidimensional analysis (Princi- The pros and cons of these methods are discussed in Section 2. pal Axis Factor analysis with Promax rotation) resulted in three Section 3 presents the TTS database and the series of tests that underlying quality dimensions. They were labeled naturalness, were conducted. An evaluation via factor analysis of the col- disturbances, and temporal distortions. A mapping of these fac- lected data is performed in Section 4. The resulting quality di- tors onto the perceived overall quality revealed that naturalness mensions are discussed in Section 5. Finally, Section 6 summa- contributes the most to the quality of TTS signals. rizes the main results and gives a perspective to future work. Index Terms: speech synthesis, quality dimensions, multidi- mensional analysis 2. Multidimensional analysis To reveal a mapping of the perceptual space of human listen- 1. Introduction ers, different analysis methods can be used. The MDS [9] uses paired comparison tests to create a stimulus space which is then Naturalness has always been the major weakness of TTS sys- reduced in dimensionality. The drawback of this approach is tems. However, improvements over the past years have shown the constraint on a small set of stimuli. Moreover, no hints are a notable increase in quality, which allows them to be used for given for the interpretation of the resulting perceptual space. a number of applications, e.g. short message services, infor- Therefore we opted for a semantic differential (SD). It uses pre- mation systems, or smart-home assistants. Still, modern TTS defined attribute scales to measure the auditory impression of systems suffer from diverse quality constraints, ranging from the listeners. This guarantees a direct relation between the used concatenation artefacts to difficulties in word- and sentence- attribute scales and the derived quality dimensions and thus an intonations. With the rise of new applications further improve- easier interpretation. On the downside, due to the given set of ments will be necessary. Thus, methods to efficiently assess scales, this approach cannot guarantee that all relevant percep- different quality dimensions are an important tool. tual dimensions are actually solicited from the test participants. Depending on the quality aspect to be assessed different kinds 10.21437/Interspeech.2011-570 To reduce the influence of the test designers to a minimum, of listening tests can be carried out: articulation and intellige- a suitable set of scales has to be developed through several bility tests assess whether the synthetic speech signal is able pretests. In pretest 1 attributes describing the auditory impres- to carry information on a segmental or supra-segmental level sion of the listeners are collected. These terms are converted [1]; comparison tests measure if human listeners can compre- into scales and presented in a second pretest. An analysis of the hend the content provided via the presented TTS signals [2]; and second pretest data leads to a final selection of scales which are overall quality tests, as recommended in ITU-T P.85 [3], capture presented in the final SD experiment. On the basis of these at- different quality aspects of the signal, e.g. naturalness, listening tribute ratings, orthogonal factors can be derived with the help effort and overall impression. Though doubts have been casted of a factor analysis. The realisation of this test will be described on the test protocol [4] [5], the method described in ITU-T Rec. in the following section, and the results of the factor analysis P.85 is still the most common way to evaluate TTS systems. are discussed in Section 4. However, to evaluate the entire perceptive space of test sub- jects, a multidimensional analysis has to be performed. Differ- ent studies have been carried out to determine the underlying 3. Experimental-setup quality dimensions. In [6] a pilot study with multidimensional This section gives an overview of the database of speech syn- scaling (MDS) on TTS data generated by the Festival synthe- thesizers collected for the listening tests. Moreover, it describes sizer lead to a three-dimensional space. Since only stimuli of the approach used to gain a relevant set of attribute scales that one unit-selection synthesizer were presented in that test, the describe the perceptual space of TTS systems in a more-or-less results cannot be generalized. Kraft and Portele [7] evaluated complete way. five German TTS systems in a series of tests and came up with two dimensions representing prosodic and segmental attributes. 3.1. Test database Given that their study was carried out in 1995, distortions from modern TTS systems e.g. unit-selection and HMM-based syn- 10 German sentences from the EUROM.1 corpus [10] were thesizers could not be evaluated. chosen as source material. Since place names, proper names Copyright © 2011 ISCA 2177 28-31 August 2011, Florence, Italy and words from foreign languages often use special pronunci- To narrow down the number of attribute scales, we omitted ation rules and thus cause trouble for speech synthesizers, the unnatural melody vs. natural melody which correlated highly selected sentences did not contain any of these. To avoid user (R>0.60) with the other scales that rate naturalness, and fatigue but still guarantee a valid impression of the occurring thus measure similar features. Moreover, scales that were used distortions, the sentences were shortened to a length of about rather rarely were dropped. 10 s each. In order to gain a first impression of the perceptual space a To capture a broad variety of distortions we generated synthetic Principal Component Analysis (PCA) with Varimax rotation speech files from 14/15 different TTS systems for female/male was performed on the remaining scales (=items) and 3 factors speakers, for some of them with up to 6 different voices. Thus, were extracted. Subsequently all items with high loadings on data from 35/28 different configurations (female/male) could be multiple factors and items with communalities < 0.45 were produced. Besides the synthetic speech files the database also discarded. This led to the following 16 attribute scales: contains stimuli from 4/4 amateur (female/male) and 4/4 pro- artificial vs. natural, bumpy vs. not bumpy, clinking vs. not fessional (female/male) natural speakers. All speech files were clinking, distorted vs. undistorted, fast vs. slow, hissing vs. downsampled to 16 kHz and level normalized to -26 dBov us- not hissing, interrupted vs. continuous, noisy vs. not noisy, ing the speech-level meter [11]. raspy vs not raspy, several voices vs. one voice, tense vs. calm, The database contains speech material synthesized by fol- undisturbed vs. disturbed, unintelligible vs. intelligible, unnat- lowing systems: Acapela Infovox3, AT&T Natural Voice, ural accentuation vs. natural accentuation, unnatural rhythm atip Proser, BOSS, Cepstral Voices, Cereproc CereVoice, vs. natural rhythm, unpleasant vs. pleasant (translations from DRESS, Loquendo, MARY bits, MARY hmm-bits, MARY German wordings) MBROLA, NextUp Talker, NextUp TextAloud3, Nuance Re- alSpeak, SVOX, and SyRUB. Abbr. Provider Synthesizer Female Male BOS RFW Bonn BOSS 1 - 3.2. Pretest 1 DRE TU Dresden DRESS 1 1 BIT MARY bits 1 1 The objective of pretest 1 was to collect a broad basis of at- HMM MARY hmm-bits 1 1 tributes describing auditory features of synthetic speech. There- MBR MARY MBROLA 1 1 fore audio files from 12/13 different TTS systems with fe- SYR RU Bochum SyRUB - 1 male/male voices plus 2 different natural speakers per gender CS1 Commercial synthesizer 1 1 1 were presented. 12 (4 female, 8 male) expert listeners from CS2 Commercial synthesizer 2 2 1 Deutsche Telekom Laboratories in Berlin took part in the test. CS3 Commercial synthesizer 3 1 1 CS4 Commercial synthesizer 4 1 1 The stimuli were presented in a quiet conference room environ- CS5 Commercial synthesizer 5 1 1 ment via headphones (AKG K601) in randomized order. Two CS6 Commercial synthesizer 6 1 1 sessions were conducted, one with female and one with male CS7 Commercial synthesizer 7 1 1 voices, with a break of 5 min in between. Every TTS system CS8 Commercial synthesizer 8 1 1 was covered with 2 stimuli. The listeners were instructed to CS9 Commercial synthesizer 9 1 1 write down nouns, adjectives and antonym pairs describing their CS10 Commercial synthesizer 10 - 1 auditory impression. Furthermore they were asked to give an in- tensity rating for each attribute on a scale ranging from 1 to 10. Table 1: Synthesizers and voice configurations used in the main The listening test resulted in 2179 collected terms out of which test. 296 unique descriptions were found. These attributes were con- densed into 44 scales. Attribute scales that mainly rate features 3.4. Main test concerning individual voice character and accent and those that For the main test a set of 15 different synthesizer configurations rate the same perceptual features were omitted.