PROCEEDINGS OF SPIE SPIEDigitalLibrary.org/conference-proceedings-of-spie Assessment of quality of JPEG XL proposals based on subjective methodologies and objective metrics Pinar Akyazi, Touradj Ebrahimi Pinar Akyazi, Touradj Ebrahimi, "Assessment of quality of JPEG XL proposals based on subjective methodologies and objective metrics," Proc. SPIE 11137, Applications of Digital Image Processing XLII, 111370N (6 September 2019); doi: 10.1117/12.2530196 Event: SPIE Optical Engineering + Applications, 2019, San Diego, California, United States Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2019 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use Assessment of quality of JPEG XL proposals based on subjective methodologies and objective metrics Pinar Akyazi and Touradj Ebrahimi Multimedia Signal Processing Group (MMSPG) Ecole Polytechnique F´ed´erale de Lausanne CH 1015, Lausanne, Switzerland ABSTRACT The Joint Photographic Experts Group (JPEG) is currently in the process of standardizing JPEG XL, the next generation image coding standard that o↵ers substantially better compression efficiency than existing image formats. In this paper, the quality assessment framework of proposals submitted to the JPEG XL Call for Proposals is presented in details. The proponents were evaluated using objective metrics and subjective quality experiments in three di↵erent laboratories, on a dataset constructed for JPEG XL quality assessment. Subjective results were analyzed using statistical significance tests and presented with correlation measures between the results obtained from di↵erent labs. Results indicate that a number of proponents superseded the JPEG standard and performed at least as good as the state-of-the-art anchors in terms of both subjective and objective quality on SDR and HDR contents, at various bitrates. Keywords: JPEG XL, image compression, standardization, objective quality assessment, subjective quality assessment. 1. INTRODUCTION Joint Photographic Experts Group (JPEG) is a sub-group of ISO/IEC Joint Technical Committee 1, Subcom- mittee 29, Working Group 1 (ISO/IEC JTC1/SC29/WG1). The group has created the image coding standard JPEG more than two decades ago, as well as many other standards including JPEG 2000, JPEG XS, JPEG XT and JPEG XR. While maintaining the previous standards, JPEG is active in development of novel coding standards in multimedia. In 2017, JPEG published a Call for Proposals for creating the next generation image coding standard, referred to as JPEG XL.1 The Call targeted development of a standard for image coding that o↵ers substantially better compression efficiency than existing image coding formats (e.g. > 60% over JPEG), along with features desirable for web distribution and efficient compression of high-quality images. The final Call for Proposals was issued in April 2018 with a deadline for expression of interest and registration in August 2018. Submissions were gathered in September 2018 and the performance of proponents was evaluated via subjective and objective quality assessment tests, following the Recommendations in ITU-R BT.2022,2 ITU- R BT.500-133 and ITU-T P.910.4 A total of seven proponents were compared to four anchors at eight di↵erent bitrates during objective and four di↵erent bitrates during subjective evaluation. All proponents were evaluated using standard dynamic range (SDR) contents and high dynamic range (HDR) contents at various resolutions with di↵erent characteristics. This paper aims to describe the methodology and results of the quality assessment of JPEG XL, which were first published as an output document of the 81st JPEG meeting, Vancouver, Canada, 13-19 October 2018. 2. BACKGROUND Billions of images are captured, created, uploaded, and shared daily, which creates an immediate need for efficient image compression. Applications are becoming increasingly image-rich, and websites and user interfaces (UIs) rely on images for sharing experiences and stories, visual information and appealing design. On the low end of the spectrum, UIs can target devices with stringent constraints on network connection and/or power consumption. Even though network download speeds are improving globally, in many situations bandwidth is constrained Further author information: (Send correspondence to authors) E-mail: firstname.lastname@epfl.ch Applications of Digital Image Processing XLII, edited by Andrew G. Tescher, Touradj Ebrahimi, Proc. of SPIE Vol. 11137, 111370N · © 2019 SPIE · CCC code: 0277-786X/19/$21 · doi: 10.1117/12.2530196 Proc. of SPIE Vol. 11137 111370N-1 Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2019 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use to speeds that inhibit responsiveness in applications. On the high end, UIs utilize images that have larger resolutions, higher dynamic range and wider color gamut, as well as higher bit depths, which leads to a further explosion of image data.5 Standards such as JPEG and PNG are still widely used as the primary coding formats, however, reduced network transmission times are needed for more interactive applications. With websites and UIs containing up to hundreds of images or several high-resolution images, the accumulated data can amount up to several megabytes worth, which could be equivalent to more than a minute of video. While video streams can be bu↵ered before playback, image-based UIs have to be responsive and interactive, without several seconds of loading and stalling when downloading or scrolling. Recently, evidence has been presented of compression technologies that outperform image coding standards in common use.6, 7 Several metrics showed the HEVC/H.265 HM encoder with SCC extensions8 to be superior according to most metrics, and for most test images. Subjectively, Daala9 was competitive, with a limited di↵erence in MOS scores. Although there is evidence of technical advances, there is no widespread standard that has state-of-the-art compression performance, and is widely supported in consumer devices and browsers. The goal of JPEG XL is to develop a new image coding standard that provides state-of-the-art image com- pression performance, and that addresses shortcomings in current standards. The activity aims to (i) achieve significant compression efficiency improvement over coding standards in common use at equivalent subjective quality, e.g. > 60% over JPEG, (ii) o↵er features for web applications, such as support for alpha channel coding and animated image sequences, and (iii) o↵er support of high-quality image compression, including higher reso- lution, higher bit depth, higher dynamic range, very high quality and wider color gamut coding. To encourage widespread adoption, an important goal for this standard is to support a royalty-free baseline. JPEG XL targets a large variety of use cases including image-rich UIs and web pages on bandwidth- constrained connection such as social media applications, media distribution applications, cloud storage ap- plications, media web sites, animated image applications, mobile applications and games, and high quality imaging applications such as rapid photo viewing, HDR/WCG user interfaces, augmented/virtual reality, image bursts, high-end photography, image mosaics, depth images, and printing. The requirements of the final call for proposals are depicted in Tables 1-3. Table 1: Necessary attributes of uncompressed images that the targeted image coding standard is expect to support. Support uncompressed images with attributes Image resolution: from thumbnail-size images up to at least 40 MP images. Transfer functions including those listed in BT.70910 and BT.210011 Bit depth: 8-bit and 10-bit Color space: at least RGB, YCbCr, ICtCp. Input type of the encoder shall match output type of the decoder. • Internal color space conversion is permitted (as part of the proposal). Color• primaries including BT.70910 and BT.2100.11 Chrominance subsampling (where applicable): 4:0:0, 4:2:0, 4:2:2, and 4:4:4. Di↵erent types of content, including natural, synthetic, and screen content. In addition to the requirements in Tables 1-3, the proposals were expected to include a high-level description of the submission including block diagrams of encoder and decoder, as well as arguments on why the proposal is meeting the requirements. Binary encoder/decoder executables and scripts, encoded-decoded materials and results, algorithm and design description, technical documentation and complexity analysis were submitted with the proposals. The submissions were then evaluated by researchers at three di↵erent universities, namely, Ecole Polytechqnique F´ed´erale de Lausanne (EPFL), Vrije Universiteit Brussel (VUB) and T´el´ecom ParisTech (TPT), using the contents and methodologies defined in the Call. Proc. of SPIE Vol. 11137 111370N-2 Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 12 Sep 2019 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use Table 2: Compressed bitstream requirements that submissions are required to cover. Core requirements Significant compression efficiency improvement over coding standards in common use at equivalent subjective quality. Hardware/software implementation-friendly encoding and decoding (in terms of parallelization, memory, complexity, power consumption) Support for alpha channel / transparency coding. Support for animation image sequences. Support for 8-bit and 10-bit bit depth. Support for high dynamic range coding. Support