PRACTICAL METHODS TO VALIDATE ULTRA HD 4K CONTENT Sean McCarthy, Ph.D. ARRIS Abstract Yet now, before we hardly got used to HD, we are talking about Ultra HD (UHD) with at Ultra High Definition television is many least four times as many pixels as HD. In things: more pixels, more color, more addition to and along with UHD, we are contrast, and higher frame rates. Of these getting a brand new wave of television parameters, “more pixels” is much more viewing options. The Internet has become a mature commercially and Ultra HD 4k TVs rival of legacy managed television distribution are taking their place in peoples’ homes. Yet, pipes. Over-the-top (OTT) bandwidth is now Ultra HD content and service offerings are often large enough to support 4k UHD playing catch-up. We don’t yet have enough exploration. New compression technologies experience to know what good Ultra HD 4k such as HEVC are now available to make content is nor do we know how much better use of video distribution channels. And bandwidth to allocate to deliver great Ultra the television itself is no longer confined to HD experiences to consumers. In this paper, the home. Every tablet, notebook, PC, and we describe techniques and tools that could smartphone now has a part time job as a TV be used to validate the quality of screen; and more and more of those evolved- uncompressed and compressed Ultra HD 4k from-computer TVs have pixel density and content so that we can plan bandwidth resolution to rival dedicated TV displays. resources with confidence. We will describe the statistical methods we use to validate Is all that resolution going to make a Ultra HD 4k content, and will present some of difference to consumers? If yes, what our results. We will also explore the impact bandwidth will 4k UHD programming need? of high-efficiency video coding (HEVC) Those are two big questions our industry is compression on the statistics of Ultra HD 4k exploring with respect to planning UHD content. The data and analysis we present are services; yet they are not independent intended to provide tools and data that could questions. be used to optimize bandwidth allocation and design Ultra HD 4k service offerings. 4k UHD is still new enough in the studios and post-production houses that 4k-capable cameras, lenses, image sensors, and INTRODUCTION downstream processing are still being optimized. Can we be sure yet that the optics Only a decade ago, high definition HD was and post processing are preserving every bit the big new thing. With it came new wider of “4k” detail? On the distribution side, could 16:9 aspect ratio flat screen TVs that made the video compression and multi-bitrate adaptive living room stylish in a way that old CRTs streaming protocols change the amount of couldn’t match. Consumers delighted in the visual detail to an extent that it could new better television experience. Studios, conceivably turn “4k” quality into something programmers, cable, telco, and satellite video more like “HD” or even less? providers delivered a new golden-age of television. HD is now table stakes most If the 4k content we have available today places, and where that is not yet the case, it for bandwidth and video quality testing does will be soon enough. not truly have a “4k”-level of detail, then we could go astray and plan for less bandwidth 2015 Spring Technical Forum Proceedings than we might need for future 4k UHD CAMERA CONSIDERATIONS services. If the 4k content we have available today is truly “4k”, then we should also want The quality of 4k content depends on the to be sure that we do not over compress and quality of the camera, the particulars of the turn 4k UHD into something less impressive. post processing such as filtering and compression, and the skill of the camera Indeed, during our UHD 4k testing, we operator and crew. have found several candidate test sequences that appeared normal to the eye but turned out 4k-capable cameras available today range to have unusual properties when examined from consumer camcorders to cream-of-the- mathematically. Such content could lead to crop professional 4k-cameras that are used to wrong conclusions when planning for UHD create premium cinema and television 4k bandwidth and services. content. Even some smartphones boast 4k cameras. In this paper, we present mathematical techniques to help answer the question “How Lens quality and image sensor size are key 4k is it?” Our method examines 4k UHD issues in 4k capture. Obviously, consumer video to see if it has a statistically expectable and prosumer grade cameras should not be distribution of spatial detail as a function of 2- expected to have the top-quality lenses and dimensional spatial frequency. The image sensors found in high-end professional benchmark for our statistical expectations is cameras. Yet, even in high-end cameras one drawn from numerous studies of the statistics needs to consider the interaction between the of natural scenes. lens and image sensor. At this point in time, the image sensors found in many 4k cameras Our main objective in writing this paper is are larger than HD image sensors. As a result, to describe methodology that might be useful depth-of-field tends to be shallower. in helping to decide which 4k UHD content Background and foreground details that are should be included in the video test library out of the plane of focus can be softer than intended to be used for bandwidth and video they are in HD. Depth-of-field can be quality planning. increased by decreasing the aperture, but at the expense on less light which can result in SOURCES OF 4K UHD CONTENT noisier video because of sensor noise. Longer exposure times could improve the amount of There are many online places from which light captured, but then motion blur could to obtain 4k content that could be considered become an issue. All of these opto-electrical for testing purposes. Industry-focused items are capable of producing 4k content that sources include the European Broadcasting has less spatial detail in the subject matter and Union1,2 (EBU), CableLabs3, and more noise than would otherwise be expected. blenderfoundation4. Stock footage typically More important, such content could lead to intended for promotional projects, but which wrong conclusions about the amount of might also be considered for testing purposes, bandwidth that will be needed to deliver great is available online sites such as Shutterstock5, 4k experiences to consumers. NYC B.Roll6, NatureFootage7, and others that can be found by searching keywords such as COMPRESSION CONSIDERATIONS “4k stock.” Video-sharing sites such as YouTube8 and Vimeo9 host compressed 4k Video compression works mainly by UHD content that could be candidates for strategically reducing the amount of spatial testing certain kinds of 4k UHD services. detail in video. Each compressed video frame 2015 Spring Technical Forum Proceedings is predicted from previously stored frames as 4k. Thus, any test content derived from such a much as possible. Whatever is unpredictable source could impact how test results should be is packaged as a residual signal and sent to interpreted. decoders, but not before the residual is further refined by being converted into a signal SPATIAL FREQUENCY having less numerical precision through a technique called quantization. In the MPEG An image is normally thought of as a 2- family of compression standards (MPEG-2, dimensional array of pixels with each pixel AVC/H.264, and HEVC/H.265), quantization being represented by red, green, and blue has the effect of preferentially reducing the values (RGB) or luma and 2 chrominance spatial details that are associated with higher channels (for example, YUV or YCbCr). An spatial frequencies. In addition, AVC and image can also be represented as a 2- HEVC employ spatial blurring filters that dimensional array of spatial-frequency reduce the noticeability of spatial components as illustrated in Figure 1. The discontinuities between coding blocks. Both visual pixel-based image and the spatial- frequency-sensitive quantization and spatial frequency representation of the visual image blurring can reduce the kind of fine detail that are interchangeable mathematically. They 4k aims to show off. Without that “4k” detail have identical information, just organized in our test content our bandwidth predictions differently. could be off when the next even-better generation of 4k camera arrives. ADAPTIVE STREAMING CONSIDERATIONS Over-the-top streaming services have taken the lead in delivering 4k content to consumers. Such services typically employ one of several adaptive streaming protocols that enable video to play smoothly even when the consumer’s bandwidth fluctuates significantly. Each adaptive-streaming video player senses its available bandwidth and requests a segment of programming that fits within its capabilities. If enough bandwidth is available, a 4k-capable video player would select lightly-compressed high bitrate segments having full 4k resolution (3840x2160). More restricted bandwidth could force selection of more aggressively compressed versions of the content though still at full 4k resolution. Even more restricted bandwidth can force selection of aggressively compressed sub-4k resolution (for example, 1920x1080 or 960x540 etc.). The modulation of the resolution and the compression level mean that 4k adaptive- streaming services might not always be fully 2015 Spring Technical Forum Proceedings Figure 1. Illustration the representation of an image in terms of spatial frequencies. The visual pixel-based image (A) can be represented as a 2-dimensional array of complex numbers using Fourier transform techniques. The absolute-value of the complex numbers is shown as a 2- dimensional magnitude spectrum (B) in which brighter areas correspond to larger magnitude values.