Impact of Adaptation Dimensions on Video Quality
Jens Brandt, Lars Wolf
Institute of Operating Systems and Computer Networks Technische Universit¨atBraunschweig
14th IEEE International Symposium on Consumer Electronics, June 07-10, 2010, Braunschweig, Germany
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 1 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Introduction
Scope Video adaptation for mobile devices Compressed domain video transcoding
Problem How to adapt a video stream to meet a certain bit rate?
Approach Analysis of the produced quality Three main adaptation dimensions
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 2 ⇒ The bit rate of a video stream mainly depends on the spatial, temporal, and detail resolution of the stream.
Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Video Adaptation for Mobile Devices
Screen ⇒ Frame size (resolution) Processor ⇒ Frame size, frame rate and detail resolution Memory ⇒ Frame size Network ⇒ Bit rate
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 3 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Video Adaptation for Mobile Devices
Screen ⇒ Frame size (resolution) Processor ⇒ Frame size, frame rate and detail resolution Memory ⇒ Frame size Network ⇒ Bit rate
⇒ The bit rate of a video stream mainly depends on the spatial, temporal, and detail resolution of the stream.
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 3 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
How to Adapt?
Which dimensions should be adapted? Reduce the temporal resolution and keep the detail quality? Reduce the detail quality while keeping the frame rate? Reduce the spatial resolution while keeping the detail quality? ... How much should each dimension be adapted? As much as needed by the device? Further than needed to keep another resolution higher? ... ⇒ Several different combinations exist
⇒ Which one produces the best quality?
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 4 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Spatial Adaptation
Meet a display resolution as well as a certain bit rate Reducing the spatial resolution also reduces the bit rate Additional bit rate reduction by reducing the detail resolution Three possibilities for the spatial target resolution: 1 The target resolution is higher than the display resolution. 2 The target resolution is the same as the display resolution. 3 The target resolution is lower than the display resolution.
⇒ Which target resolution produces the best quality?
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 5 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Evaluation Process
10 video sequences Resolution: CIF, 264×216 pixels and QCIF Bit rate: 40 kbit/s – 480 kbit/s Evaluation of average Y-PSNR values Target resolution: CIF and 264×216 pixels
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 6 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Results - Spatial Adaptation
Sequence: deadline Sequence: mobile 38 24 36 34 22 cif (target: cif) 32 cif (target:264x216) 30 264x216 (target: both) 20 qcif (target: both) 28 cif (target: cif) cif (target:264x216)
Average Y-PSNR 26 Average Y-PSNR 18 264x216 (target: both) 24 qcif (target: both) 22 16 0 100 200 300 400 500 0 100 200 300 400 500 Bit Rate (kbit/s) Bit Rate (kbit/s) Similar results for all test sequences, also for higher resolutions ⇒ Resolution reduction other than needed is not beneficial
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 7 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Temporal Adaptation
Bit rate reduction by reducing the frame rate of a stream Which frame rate produces the best quality? PSNR values are not sufficient for quality evaluation Interpolation of missing frames produces poor PSNR values PSNR values of remaining frames do not reflect visual quality ⇒ User interviews Four test sequences Encoded at different frame rates Presented on a mobile device in changing order Participants should chose a preferred version
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 8 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Test Sequences
Encoded at 5, 12, and 24/25 frames per second (fps) Constant bit rate of 180 kbit/s Fixed spatial resolution (fit to 320×240 pixels) Duration of 75 and 90 seconds
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 9 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Results - Temporal Adaptation 50 non-expert participants with ages between 21 and 59 years
100 100
80 80 3 no difference 4 60 25/24 fps 60 3 2 12 fps 1 40 5 fps 40 2 average Fraction in % Fraction in %
20 20
0 0 1 Average Grade (1=best, 6=worst)
News News Soccer Soccer Animation Animation Natural Movie Natural Movie Dominance of 12 fps as the preferred rate No clear dominance for high amount of motion and scene cuts Good grades on average for the preferred version ⇒ Users accept lower frame rates to get more details per frame
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 10 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Detail Adaptation
High detail quality is important for users of mobile devices Reduction of the detail quality reduces the bit rate Is there a maximum detail quality? ⇒ User interviews Test sequence encoded at different bit rates Pairwise presentation on a mobile device Participants should chose a version they liked more
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 11 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Results - Detail Adaptation
Percentages of 41 people preferring version A over B: Version A 500 kbit/s 700 kbit/s 1500 kbit/s 300 kbit/s 87.80 % 85.37 % 90.24 % Version B 500 kbit/s 70.73 % 73.17 % 700 kbit/s 60.98 %
78.04 % of the users preferred the higher bit rate. 23.57 % of all comparisons were chose randomly. In 60.34 % of these cases the users chose the higher bit rate.
⇒ The higher quality is still noticeable ⇒ There is no upper bound for the quality level below 1500 kbit/s
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 12 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Conclusion
Spatial dimension Quality evaluation at different spatial resolutions ⇒ No benefit from resolution reduction other than needed Temporal dimension Subjective quality tests on a mobile device ⇒ Users preferred lower frame rates Detail dimension Subjective quality tests on a mobile device ⇒ No upper bound observable
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 13 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Combined Adaptation
1 Spatial dimension Spatial downscaling to the screen resolution of the client 2 Temporal dimension Reduction of the frame rate if necessary 3 Detail dimension Reduction of the detail resolution to fine tune the bit rate
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 14 Questions?
Jens Brandt [email protected]
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 15 Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 16 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Combined Adaptation
i) adaptation in the detail dimension ii) adaptation in the temporal and detail dimension iii) adaptation in the spatial and detail dimension iv) adaptation in the spatial, temporal and detail dimension
akiyo mobile
400 8000 350 7000 300 6000 250 5000 200 4000 150 3000 100 2000 Bit Rate (kbit/s) Bit Rate (kbit/s) 50 1000 0 CIF 25fps 0 CIF 25fps CIF 12.5fps CIF 12.5fps 5 10 QCIF 25fps 5 10 QCIF 25fps 15 20 15 20 25 30 QCIF 12.5fps 25 30 QCIF 12.5fps Quantization Scale Value Quantization Scale Value
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 17 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Test Sequences
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 18 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Spatial Adaptation - Akiyo and Foreman Sequence
Sequence: akiyo Sequence: foreman 42 34
40 32 38 30 36 28 34 cif (target: cif) cif (target: cif)
Average Y-PSNR cif (target:264x216) Average Y-PSNR cif (target:264x216) 32 264x216 (target: both) 26 264x216 (target: both) qcif (target: both) qcif (target: both) 30 24 0 100 200 300 400 500 0 100 200 300 400 500 Bit Rate (kbit/s) Bit Rate (kbit/s)
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 19 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Spatial Adaptation for Higher Resolution Sequences
harbour - Target Resolution: 528x432 soccer - Target Resolution: 528x432 33 36 704x576 704x576 32 616x504 35 616x504 528x432 528x432 31 34 30 33 29 32 28 31 27 30
Average Y-PSNR 26 Average Y-PSNR 29 25 28 24 27 0 1000 2000 3000 4000 5000 6000 0 1000 2000 3000 4000 5000 6000 Bit Rate (kbit/s) Bit Rate (kbit/s)
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 20 Introduction Spatial Adaptation Temporal Adaptation Detail Adaptation Conclusion
Results Percentages of 41 people preferring version A over B: Version A 500 kbit/s 700 kbit/s 1500 kbit/s 300 kbit/s 87.80 % (36) 85.37 % (35) 90.24 % (37) Version B 500 kbit/s 70.73 % (29) 73.17 % (30) 700 kbit/s 60.98 % (25)
For all six comparisons, 78.04 % of the users chose the higher bit rate. Percentages of 41 people randomly chose one version: Version A 500 kbit/s 700 kbit/s 1500 kbit/s 300 kbit/s 7.32 % (3/1) 9.76 % (4/2) 9.76 % (4/2) Version B 500 kbit/s 34.15 % (14/9) 39.02 % (16/10) 700 kbit/s 41.46 % (17/11)
In 23.57 % of all comparisons the user randomly chose the preferred version. In 60.34 % of these cases the users intuitively chose the version with a higher bit rate.
Jens Brandt, Lars Wolf Impact of Adaptation Dimensions on Video Quality 21