EBU Evaluations of Multichannel Audio Codecs
Total Page：16
File Type：pdf, Size：1020Kb
EBU – TECH 3324 EBU Evaluations of Multichannel Audio Codecs Status: Report Source: D/MAE Geneva September 2007 1 Page intentionally left blank. This document is paginated for recto-verso printing Tech 3324 EBU evaluations of multichannel audio codecs Contents 1. Introduction ................................................................................................... 5 2. Participating Test Sites ..................................................................................... 6 3. Selected Codecs for Testing ............................................................................... 6 3.1 Phase 1 ....................................................................................................... 9 3.2 Phase 2 ...................................................................................................... 10 4. Codec Parameters...........................................................................................10 5. Test Sequences ..............................................................................................10 5.1 Phase 1 ...................................................................................................... 10 5.2 Phase 2 ...................................................................................................... 11 6. Encoding Process ............................................................................................12 6.1 Codecs....................................................................................................... 12 6.2 Verification of bit-rate ................................................................................... 13 7. Experimental design ........................................................................................15 7.1 Test methodology ......................................................................................... 15 Evaluation software.......................................................................................... 16 Impairment (artefact) categories for MCA tests ........................................................ 16 7.2 Anchors...................................................................................................... 17 7.3 Evaluation Process ........................................................................................ 17 7.4 Listening conditions....................................................................................... 18 7.5 Test Sessions ............................................................................................... 18 8. Statistical Analysis and Postscreening ..................................................................19 9. Presentation of Main Results..............................................................................21 9.1 Phase 1 ...................................................................................................... 21 All Codecs averaged over all test items plus average of worse case item. ......................... 21 Average scores for items over all codecs................................................................. 21 9.2 Phase 2 ...................................................................................................... 22 All Codecs averaged over all test items plus average of worse case item. ......................... 22 Average scores for items over all codecs................................................................. 23 9.3 Phase 1 and 2 Comparison ............................................................................... 23 Average scores for Applause item for codecs common to phases 1 and 2 .......................... 23 10. Summary and Conclusions.................................................................................24 11. Acknowledgements .........................................................................................26 12. References....................................................................................................26 Appendix 1: Selection Panel Report for Phase 1 (June 2006) ..........................................27 Description of process.......................................................................................... 27 3 EBU evaluations of multichannel audio codecs Tech 3324 Submission and coding of candidate material ............................................................. 27 Presentation of coded items.................................................................................. 27 Auditioning and decision-making ............................................................................ 27 Description of listening facilities ............................................................................ 28 Problems encountered during the selection process ..................................................... 29 Reproduced Sound Level ...................................................................................... 29 Appendix 2: Instructions for Assessors ...................................................................... 31 Background ...................................................................................................... 31 Training phase .................................................................................................. 31 Grading........................................................................................................... 32 Important remarks for the test supervisor ................................................................. 33 Appendix 3: The EBU Members' Listening Rooms and Equipment Set-up ............................ 35 British Broadcasting Corporation (BBC), Kingswood Warren, UK ....................................... 35 France Telecom - Orange, Rennes, France................................................................. 36 Institut für Rundfunktechnik (IRT), Munich, Germany.................................................... 37 ORF – Austrian Broadcasting Corporation, Vienna, Austria .............................................. 38 Radio France, Paris, France .................................................................................. 40 RAI CRIT – Centro Ricerche Innovazione Tecnologica, Turin, Italy ..................................... 41 Appendix 4: Detailed test results............................................................................. 43 Phase 1 ........................................................................................................... 43 Averages for each codec over all test items - Phase 1 ................................................ 43 Average scores for each lab over all codecs - Phase 1................................................. 54 Phase 2 ........................................................................................................... 57 Averages for each codec over all test items -Phase 2 ................................................. 57 Average scores for each lab over all codecs - Phase 2................................................. 65 Appendix 5: Histograms......................................................................................... 69 Phase 1 ........................................................................................................... 69 Phase 2 ........................................................................................................... 81 4 Tech 3324 EBU evaluations of multichannel audio codecs EBU evaluations of multichannel audio codecs EBU Committee First Issued Revised Re-issued DMC 2007 Keywords: Multichannel Audio, Codec, MUSHRA 1. Introduction With the advent of High Definition television services (HDTV), the public is increasingly being exposed to surround sound presentations using so-called home theatre environments. Multichannel audio is available also from recorded media and computer games. However, the restricted bandwidth available into the home, whether by broadcast (satellite, terrestrial, or cable), or via broadband, means that there is an increasing interest in the performance of low bit rate surround sound audio coding systems for "emission" coding. The European Broadcasting Union Project Group B/MAE (Multichannel Audio Evaluations) was set up in 2006 in order to perform the assessment of the sound quality of some multichannel audio bit rate reduction codecs for broadcast applications. Codecs under test include offerings from Dolby, DTS, Microsoft, implementations of MPEG AAC and of the new MPEG Surround codec. The bit rates ranged from 64 kbit/s to 1.5 Mbit/s. The last time that the EBU conducted subjective quality tests on surround sound codecs there were few available [1]. Now the range is much wider, little test data is publicly available and so the choice is more difficult. To help broadcasters make informed decisions, an EBU technical project group has performed extensive subjective tests of a wide range of codecs at a wide range of bit rates on a wide range of test material. The test method and the results are described in this report. The choice of test method was made based on the expected range of quality. Of the two main methods, BS.1116 [2] and MUSHRA [3], it was felt that MUSHRA was more appropriate, at least for initial tests. If finer discrimination between high quality systems were to be required, then BS.1116 tests could have been conducted on a subset of the systems later. However MUSHRA was found to provide satisfactory resolution of all the test results, therefore BS. 1116 was not used at all. The choice of test material followed the process specified in the test