Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

By Huawei, BOE, and CAICT Contents

Flat-Panel Display Approximat- ing the Human Visual Limit

4.1 Flat-Panel Display Technology Suits Human Eyes

4.1.1 Industry Development History Visual Experience Re- quirements Drive Display 4.1.2 Screen Parameter Requirements Technology Upgrade 4.1.3 Mainstream View Points in the Industry

3.1 Human Visual Features 4.2 Technologies Such as Cloud Computing and AI Help High-Quality Content Production 3.2 Continuously Increasing Human 01 02 Visual Experience Requirements 4.3 Development Trend of Flat-Panel Display Preface Overview 03 04 Challenges for Net- works

7.1 Network Upgrades Driven by Video Experience

7.2 Network KPI Requirements of Various Video Experience Phases

7.3 Target Network for Superb 05 06 Visual Experience 08 Immersive 3D VR Display Light Field Is the Ideal Future Resolution Requires Im- Form of Future Display Prospects provement Technology 5.1 Steady Development of VR 6.1 Light-Field Display Technology 07 Display Technologies Is in the Initial Stage

5.1.1 VR Display Principle 6.2 Light Field Content Production Has High Requirements on 5.1.2 High Deﬁnition Is the Greatest Computing Capabilities Challenge for VR Display Technologies 6.3 Development Trend of 04 5.1.3 Increasing Deﬁnition of VR Light-Field Display Technologies Screens

5.2 Diverse VR Content Display Modes

5.3 Development Trend of Immersive 3D VR Display Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

01 Preface

Digitalization and intelligentization are fostering a new information revolution. Ubiquitous display and connectivity have become important elements in the ICT infrastructure industry, and consumers' increasing demands for experience accelerate the transformation.

The visual system is humanity's dominant medium of information reception. Therefore, improving visual experience is the top priority in display technology innovation. The industry has progressed from traditional SD to HD and then to UHD 4K. Displayed colors have become more vivid and the pictures are clearer, even allowing ﬁne details to be seen. 8K large screens with further improved resolution are entering the market, leading industry development toward ultimate and ubiquitous display.

However, even ultimate 2D experience pales in comparison to genuine human experience. Immersive virtual reality (VR) will become the entrance to the digital world. Consumers now demand 360° panoramic observation, free movement, and interaction in a virtual environment for the ultimate VR experience. VR is still in the initial development stage and is far from providing ultimate experience. To build a truly lifelike virtual world, the VR industry must collaborate with the wider technological industry.

In the double G (gigabit home broadband and 5G) era, networks will accelerate business digitalization with faster transmission speed and ultra-low latency.

The double G era will be characterized by four key trends. First, big data techniques allow gleaning insights from massive data, making information reﬁning results more regular and universal. Second, the

01 Preface Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

edge computing architecture will eliminate barriers to cross-region data sharing and provide a basic environment for widespread application of machine learning. Third, the rapid development of data openness platforms promotes information aggregation and sharing, generating more value. Fourth, as the ﬁnal part of information transmission, data visualiza- tion demands will increase rapidly. To integrate naturally with human behavior, human-machine interaction will evolve from single-screen display to multi-screen/cross-screen display, from SD to UHD (4K/8K), and from purely visual interaction to multisensory interaction.

With the development of communications technologies, China's network has made great strides under the drive of innovation and efficiency. The enterprises represented by Huawei Technologies Co., Ltd. (Huawei) have made unremitting efforts to develop communications technologies, and are the cornerstone of rapid economic and technological transformation. Continued development of cutting-edge technologies such as edge computing, big data, and artiﬁcial intelligence will reshape people's lives with the support of stronger communications infrastructure. Simultaneously, the development of display technologies has been silently fueling China's communications industry. In just 20 years, China has leaped from relying on import of cathode ray tubes (CRT) to mass production of LCD products and now to ﬂexible OLED technologies and independent R&D. As a key display technology company in China, BOE Technology Group Co., Ltd. (BOE) developed multiple display technologies and actively explores man-machine interaction to support the steady rise of China's broader ICT industry. Communications technology and human-machine interaction technology complement each other. In the double G era, ultra-high bandwidth and display interaction will become the mainstay of social progress and business transformation.

This white paper is jointly produced by BOE, China Academy of Information and Communications Technology (CAICT), and Huawei. It explores the limits of visual information development from the perspective of visual experience requirements, provides insights and forecasts for future development trends, and explores the probability of network indicator requirements and solutions from the perspective of service scenarios. This white paper will help industry partners conceptualize visual experience information and increase product competitiveness. Meanwhile, Huawei iLab is willing to work with industry partners to push the visual limit all the way to the information limit.

Preface 02 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

02 Overview

Most of information humans receive comes from vision. Progressing toward natural interfacing with human vision, display technology may be divided into three phases: ﬂat-panel display, immersive 3D VR display, and light-ﬁeld display.

In the ﬂat-panel display phase, VR provides immersive 3D visual Light ﬁeld technology can be the 4K/8K resolution nearly experience. However, resolution used to restore the actual reaches the limit of human eyes. remains a bottleneck in develop- scenario and improve the visual In the future, the color gamut, ment. Ultimate VR experience experience. Although this frame rate, and 3D display requires at least 60 pixels per technology is not yet mature, the technologies can be further degree (PPD). That is, the resolution whole industry has a great optimized to provide better visual of the display terminal must reach development potential in terms experience. 16K (binocular), and the full-view of content collection, production, resolution of the content must and terminal display. reach 24K. Current headsets can provide a PPD of only 20.

This white paper collects industry insights and data and ﬁnds that the entire display industry is booming.

If the price remains unchanged, product performance more than doubles every 36 months, and this period is shorten- ing. In the communications field, optical fiber capacity doubles every 36 months, and IP network performance doubles every 18 months. The display and communications fields are redefining Moore's Law and challenging the Shannon limit, as digital visual experience continues to accelerate toward the human visual limit.

Throughout the development history of the industry, full introduction of a new product stage takes about four years: The period from 2006 to 2010 saw the introduction of FHD 2K. From 2012 to 2016, 4K was introduced and the penetration rate of UHD 4K climbed to 40%. 2018 was the start of a new epoch of 8K. It is estimated that the penetration rate of UHD 8K in displays over 65 inches will exceed 20% by 2022.

In the double G era, improved communications capabilities will support myriad UHD display applications. 2K will become the baseline video and display requirement, 4K will become the mainstream requirement, and 8K will become mainstream for sports matches, hot event live streams, and movies. 8K display will also become a new benchmark for high-end consumer electronics products.

By 2020, fast LCD will be the optimal display solution for VR. With a 60 Hz refresh rate and less than 5 ms response time, latency-induced dizziness can be effectively mitigated, meeting VR display requirements and reducing VR hardware costs. Later, with the development of optical technologies, micro OLED will become the best solution for VR display. The response time of less than 0.1 ms minimizes dizziness. The wide color gamut of over 80% and 10000:1 high contrast bring perfect display effects. The weight of the devices will be greatly reduced, improving user comfort.

03 Overview Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

From flat panel to VR/AR and then to light field, the more advanced a display technology, the lower its maturity and the farther it is from meeting the visual limit of human eyes. However, VR/AR and light field are without doubt ideal display technologies for the future. VR with higher resolution and light field with more layers of depth information will bring users a visual experience closer to the real world. Meanwhile, ultimate visual experience brings massive data, posing challenges for network transmission.

Flat panel phase: Carrier-grade 4K requires 50 Mbit/s bandwidth, and ultra-high 4K requires 75 Mbit/s bandwidth. Considering multi-screen concurrency in homes, the required bandwidth will reach 200 Mbit/s.

As ﬂat panel visual experience evolves towards immersive visual experience and Cloud VR starts from full-view 8K resolution, home broadband needs to evolve to gigabit-level, the existing network faces challenges, home Wi-Fi needs to be optimized, and GPON needs to be upgraded to 10G PON. For strong interaction services, CDN downshifting and edge computing must be used to keep latency low, and the network architecture needs to be reconstructed.

In the future, the evolution of immersive visual experience to holographic light ﬁelds will increase data volume by an order of magnitude. Therefore, networks of the post–double G era, such as 10 gigabit or terabit HBB/6G, need to be supported.

Overview 04 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Visual Experience Requirements 03 Drive Display Technology Upgrade

About 83% of information humans receive comes from vision. With the popularity of video entertainment, various digital images fill people's lives. Visual information requires massive data, from images, videos, and VR to light fields, and from SD to HD and then to UHD. Humans' pursuit of visual experience is gradually increasing, so what level is required to approximate the human visual limit? Benchmarking to the experience of human eyes looking at the real world, the human visual limit can be defined as a state when human eyes cannot distinguish between digital content and real-world content in terms of image definition, color saturation, and display latency. This is closely related to human visual features.

3.1 Human Visual Features

Color perception

There are two types of photoreceptors in the human retina: rods and cones. Rods provide Human eyes' scotopic vision, and cones provide color relative sensitivity vision. Human eyes can perceive visible light V（λ） at wavelengths ranging from 380 nm to 780 [%] nm and have different sensitivity to light at 100 different wavelengths. Experiments show 80 Night vision Daytime vision that human eyes are most sensitive to light at the 555 nm wavelength during the day- 60 time. Therefore, in the daytime, red and 40 purple colors perceived by human eyes 20 appear slightly dark and the brightest colors λ（nm) 0 sensed by human eyes are yellow and green. 400 450 500 550 600 650 700 750 800 As shown in the following ﬁgure, the visual Figure 3-1 Visual sensitivity of human eyes sensitivity curve moves leftwards slightly at night.

05 Visual Experience Requirements Drive Display Technology Upgrade Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Color depth is a representation of the number of visible colors. The color depth of 8 bits (that is, the grayscale value between 0 and 255 is divided into 2 to the 8th power) allows representation of 256 distinct colors. Human eyes can distinguish more than 10 million colors. Therefore, a 24-bit color depth is required to achieve the color resolution limit of human eyes.

Brightness perception

In addition to color classiﬁcation, human eyes can perceive color intensity. Light intensity of a unit area of an object is represented by brightness (unit: nit), and human eyes' perception of brightness ranges from 10-3 to 106 nits. When the brightness is 3000 nits, which is close to the brightness of the natural environment during a sunny day, human eyes can view objects comfortably.

Contrast perception

Human eyes can perceive a wide range of brightness, but they cannot view the same image with extremely high and low brightness simultaneously. This introduces the concept of contrast, that is, the ratio of the maximum brightness to the minimum brightness of a luminous surface. Human eyes can perceive images with a contrast range of about 1000:1 (appropriate brightness) to 10:1 (extremely high or low brightness). The contrast of a projection screen is about 100:1.

FOV and visual discrimination

The ﬁeld of view (FOV) of human eyes can reach about124 ° . The FOV of the area that human eyes can focus on is about 25°, and an FOV of 60° brings the most comfortable visual perception.

Human eyes have a strong ability to discriminate object details. The discrimination of human eyes with 1.0 vision can reach 60 PPD. That is, human eyes can discriminate two objects 30 cm away from each other at a distance of 1 km.

Persistence of vision

The persistence of human visual perception is the premise of active images. When light stimulates human eyes, it takes a certain period of time from the establishment of visual images to the disappearance of these visual images. This lag is referred to as persistence of vision, which in humans is about 0.05 to 0.2s. Therefore, as long as the image refresh frequency is greater than 20 Hz, the human visual system perceives continuous active images. This is also the basis for the typical movie frame rate of 24 FPS or higher, which is sufficient to ensure continuous image perception. To provide good visual experience, the frame rate should reach 60 FPS, 90 FPS, or even 120 FPS. For example, for latency-sensitive services such as VR and AR, the frame rate must be greater than 60 FPS.

Depth perception

Depth perception is one of the most important features of human eyes. Human eyes can perceive stereoscopic space or objects because there is a parallax in human eye imaging. After two images with a parallax are synthe- sized in the brain, stereoscopic vision will be presented so the depth information of the object can be perceived.

Visual Experience Requirements Drive Display Technology Upgrade 06 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

3.2 Continuously Increasing Human Visual Ex- perience Requirements

Industry research shows that human eyes can perceive stereoscopic vision mainly due to psychological perception, motion parallax, and accommodation.

Psychological perception: Psychological perception means that what human eyes perceive may not be a 3D object, but a 3D "illusion" produced by the brain when it is "deceived" by some means. The means include:

Affine: When human eyes look at two objects of the same size, the image 01 of the nearer object is larger.

Occlusion: On the same line of sight, a near object can occlude a distant object, and the human visual system can determine the relative position 02 of these objects based on the occlusion relationship.

Shadow: When unidirectional light is present, the shadow of an object can help the human visual system to determine the stereoscopic shape of the object. For example, for either a sphere or a cylinder, human eyes obtain a 03 round image when looking at it directly, and cannot discriminate the actual shape. However, if light shines on the side of the object to produce a shadow, human eyes can easily discriminate the sphere and cylinder.

Transcendental prophet: Human brains have the ability to learn and remember. For objects that have been encountered in the real world, 04 even if they are displayed in 2D images, human brains can automatically perceive their 3D shape.

Binocular parallax: The image locations of an object seen by the left and right eyes are different. This feature deceives human brains to produce stereo vision effects as long as there is a parallax between the image loca- 05 tions of a plane image seen by the left and right eyes. A larger parallax brings a more obvious stereo effect. This principle is used in 3D movies.

07 Visual Experience Requirements Drive Display Technology Upgrade Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Through these psychological effects, ﬂat-panel displays (such as mobile phone screens and TV screens) can produce 3D effects without changing the display mode itself. That is, no 3D object or model is displayed.

Motion parallax Accommodation In the real world, human eyes can obtain different With both motion parallax and binocular parallax, visual images of objects by observing them from the visual experience of VR is extremely close to the different angles. Using motion parallax, human real world. However, in real-world scenarios, when brains can determine the shape and size of objects. human eyes focus on near objects, distant objects The most typical application is VR, although VR become blurred, and vice versa. VR cannot simulate imaging still uses binocular parallax. In a 3D theatre, this feature, called accommodation. In VR, users no matter how you move your position, the images always focus on the VR screen regardless of whether you see are always at the same angle of view. In VR, they are focusing on near or distant objects. This however, you can move your body freely to see produces a depth perception conflict in users' brains, images from different angles. causing effects such as dizziness. Light-field display is the best solution to this problem, because the light field can record the direction information of light, so as to provide images of different depths.

Psychological perception can produce 3D illusions, while motion parallax and accommodation can provide stereo vision approximating real-world perception. Therefore, from the perspective of visual experience, display technology development can be divided into three phases of development toward the human visual limit: ﬂat-panel display, immersive 3D VR display, and light-ﬁeld display.

Accommodation

Motion parallax

Psychological perception

Light-ﬁeld display

Immersive 3D VR display

Flat-panel display

Figure 3-2 Three phases of display technology development

Visual Experience Requirements Drive Display Technology Upgrade 08 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Flat-Panel Display Approximating 04 the Human Visual Limit

For flat-panel display, a resolution of 2K/2.5K for small screens such as those of mobile phones and tablets can reach the human visual limit, whereas large screens such as those of TVs require a resolution of 4K/8K. In terms of resolution, flat-panel display is close to the human visual limit. However, other parameters, such as wide color gamut, high refresh rate, and contrast, are still being optimized for flat-panel display to approximate the human visual limit from various dimensions.

4.1 Flat-Panel Display Technology Suits Human Eyes

4.1.1 Industry Development History

Flat-panel displays are widely used. Their earliest application was in film. Films are played by projecting images on screens using light. If price remains unchanged, the perfor- With the emergence of cathode ray tubes (CRTs), images are digitalized. The mance of display products most typical application of CRTs is TVs and outdated computer monitors. must be at least doubled Their screens are curved in both the horizontal and vertical directions, causing every 36 months. This image distortion, reflection, and poor visual experience. period is now being Currently, the most common display technologies include light emitting diode shortened. (LED) and liquid crystal display (LCD). LED allows display of images by con- trolling light emissions of semiconductor light-emitting diodes. The LED dis- Wang Dongsheng, play has high brightness, long service life, and large angle of view. It can be chairman and founder of flexibly extended and seamlessly stitched. It mainly applies to indoor and out- BOE door large-screen billboards. LCD displays different images by rotating liquid crystal molecules in liquid crystal pixels. The modular design and high-integration chip design directly reduce circuit power consumption and heat emission. In addition, LCDs are light and provide high-quality images. Currently, LCD is the mainstream display technology in the market and is widely used in screens of mobile phones, tablets, TVs, and computers.

Human eyes' pursuit of visual experience is endless. For small-screen products such as mobile phones, the organic light-emitting diode (OLED) technology can provide a wider color gamut and higher contrast than LCD, and is gradually applied to high-end ﬂagship phones. For large screens, the micro-LED technology (LED miniaturization and matrix technology) implements micrometer-level pixel spacing. Each pixel can be controlled and driven inde- pendently. The resolution, brightness, contrast, and color gamut of each pixel are better. However, the micro-LED technology is not mature or applied on large scale.

09 Flat-Panel Display Approximating the Human Visual Limit Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Film

CRT display

LCD and LED display

OLED and LTD display

Figure 4-1 Development history of the ﬂat-panel display technology

4.1.2 Screen Parameter Requirements

According to industry research, when the number of pixels exceeds 60 at each angle of view, human eyes will not feel the ﬁlm grain effect or screen-door effect. That is, the value of PPD is greater than 60. The value of PPD depends on the FOV and the number of pixels on the screen. px PPD= ⁄FOV Here FOV indicates the FOV of human eyes in the horizontal or vertical direction, and px indicates the number of pixels in the direction.

Flat-panel display is mostly common in mobile phones and TVs. Generally, phone screens are viewed from about 30 cm away from human eyes, whereas TV screens are viewed from about 2.5 m away. The following table lists the relevant speciﬁcations.

Terminal Mainstream Watch Distance Horizontal Mainstream Horizontal Size Type (Unit: Inch) (Unit: m) FOV Resolution PPD

Mobile phone 6 0.3 25° 2560*1440 102

TV 55 2.5 27.5° 4096*2160 148

Table 4-1 Screen speciﬁcations of mobile phones and TVs in typical scenarios

Flat-Panel Display Approximating the Human Visual Limit 10 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

In typical scenarios, the PPD of mainstream 6-inch 2.5K mobile phones can reach 102, and the PPD of 55-inch 4K TVs even reaches 148, both exceeding the resolution requirement of 60 PPD. Of course, for TVs, the 55-inch size cannot provide ultimate home theatre experience. TVs with 100-inch and 120-inch screens will gradually become available. This requires 8K resolution to achieve the vision effect of 55-inch 4K TVs. To promote the development of the 8K industry, BOE proposed a strategy of simultaneously promoting 8K, popularizing 4K, replacing 2K, and making good use of 5G. This accelerates the construction of the 8K UHD industry ecosystem. In 2017, BOE started delivering 110-inch 8K and 75-inch 8K UHD displays to customers including Huawei.

In terms of resolution, both small-screen products and large-screen products can reach the human resolution limit. However, from the perspective of visual experience, parameters including the color gamut, refresh rate, and contrast require further optimization. For example, the high dynamic range imaging (HDR) technology and 4K resolution at the 144 Hz refresh rate can be used to further improve visual experience.

2018 was the start of a new epoch of HDR display. In 2017, to pursue ultimate experience, the electric HDR is an image enhancement technology, which competition industry launched a display with a makes the dark part of images deeper without refresh rate of 1080P@240 Hz. In 2018, the refresh losing details and makes the bright part brighter rate for high resolution displays even reached while retaining image authenticity. In addition, 4K@144 Hz. HDR increases the contrast and color gamut of images.

4.1.3 Mainstream View Points in the Industry

The active-matrix organic light-emitting diode (AMOLED) technology is gradually being applied to mobile phone displays. However, thin-film transistor LCD (TFT-LCD) remains the dominant technology in these displays. In recent years, AMOLED has developed rapidly in the mobile phone field and has become popular in the high-end mobile phone market. Currently, mid-range and high-end phone models of all brands are adopting AMOLED. Simulta- neously, AMOLED has the potential for flexible display. With the gradual improvement of peripheral accessories, AMOLED flexible display will have a larger space for rapid growth in the future.

LCD is still the mainstream technology of TV displays. TFT-LCD is the mainstream technology, comprising almost all market share. The new technology, AMOLED, offers good color performance but remains at the exploration phase in the consumer market due to factors such as yield rate and cost. The price of a device using the AMOLED technology is much higher than that of the device using LCD technology. Therefore, large-scale AMOLED application in the terminal market is distant.

In the foreseeable future in the TV ﬁeld, TFT-LCD will become the mainstream technology and continue to improve. 2018 marked the start of a new epoch of 8K entering the introduction stage. Throughout the development history of the industry, full introduction of a new product stage takes about four years: The period from 2006 to 2010 saw the introduction of FHD 2K. From 2012 to 2016, 4K was introduced and the penetration rate of UHD 4K climbed to 40%. It is estimated that the penetration rate of UHD 8K in displays over 65 inches will exceed 20% by 2022.

11 Flat-Panel Display Approximating the Human Visual Limit Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

4.2 Technologies Such as Cloud Computing and AI Help High-Quality Content Production

The ﬂat-panel display technology is mature, but corresponding high-quality content is lacking. This is because high-quality content is slow and expensive to produce. Rendering of computer graphics (CG) is especially costly, consuming a lot of computing resources. For example, image rendering for the ﬁlm Avatar used 4000 servers and took up to one year.

The popularization of UHD technologies such as 4K and 8K shocks the traditional media production industry. Media production requires technical innovation from the aspects of on-site photography, storage, transmission, and distribution. To address the challenges of traditional media production, cloudiﬁcation of media production is inevitable.

In cloudiﬁcation of media production, original audio and video information is uploaded to the cloud through private lines. Cloud servers provide powerful cluster computing services, which enable the photographed 4K/8K videos to be efficiently edited on the cloud and ﬁnally transmitted to user terminals.

In addition to content editing on the cloud, low resolution and frame rate of original ﬁlming can be improved using cloud-based image enhancement to enhance viewing experience.

Video image enhancement improves video visual effects by frame insertion and super-resolution. Frame insertion uses time domain information between adjacent frames to calculate a transition frame, improving video smoothness. Super-resolution uses spatial information to calculate interpolation pixels, improving resolution.

3D display can bring better visual experience than 2D display, and it is an important direction for future development of ﬂat-panel display technology. 3D content will primarily be produced by using 3D media sources, but some classical 2D media sources can be augmented with 3D visual effects using 2D-to-3D video conversion technology.

The ﬁlm Titanic is a typical case of 2D-to-3D video conversion. The 2D ﬁlm was well received by audiences. However, if most scenarios use 3D display technology, it would bring more vivid visual effects. In 2012, the 3D Titanic produced using the 2D-to-3D video conversion technology was released. It took more than two years to turn 2D into 3D, because it was time-consuming and cumbersome to capture the depth information of images frame by frame. With the recent rapid development of AI technologies, algorithms can be applied to 2D-to-3D video conversion. This will greatly improve conversion efficiency and release more high-quality 3D media sources.

The data volume of 3D media sources is twice that of 2D media sources of the same time duration. Using special compression algorithms, the network bandwidth requirement of 3D media sources can be reduced to only about 1.5 times that of 2D media sources.

Flat-Panel Display Approximating the Human Visual Limit 12 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

With new technologies such as cloud computing and AI, complex computing can be migrated to the cloud, reducing content production cost and improving efficiency. However, uploading raw materials to cloud servers brings huge uplink network requirements.

4.3 Development Trend of Flat-Panel Display

Billy Lynn's Long Halftime Walk, a 2016 war drama directed by Ang Lee, was unique for using the 4K 3D 120 FPS form, which set a new bar for the film industry. The frame rate of typical films is only 2K 25 FPS, but in this film, Lee chose the higher specifications, not only to improve image quality, but also to provide a high-definition scenario allowing audiences to immerse themselves in the vivid scenes. This form displays the fear and death of war to the fullest extent and brings a new understanding for wider audiences.

In 2019 MWC, BOE and Huawei jointly conducted foldable phone projection and 8K+5G live display. They used Huawei's foldable phone, the Mate X, to project online 4K content through 5G signals to BOE 110-inch 8K UHD display in real time. They also used 8K UHD cameras to shoot the scenery on the coast of Barcelona in real time and transmitted the captured images to BOE 110-inch 8K UHD display in 5G ultra-high speed long-distance mode, demonstrating ultimate visual experience.

Figure 4-2 Huawei Mate X + BOE 110-inch Figure 4-3 Huawei 5G CPE Pro + BOE 110-inch 8K UHD projection 8K live broadcast

The high-resolution, high-frame-rate, and 3D display form not only brings better visual experience, but also helps to express the concept of video content transmission. This is the key trend of future plane vision development. The ﬂat-panel display technology continues to mature. With the help of cloud computing, high resolution and high frame rate content like Billy Lynn's Long Halftime Walk will become more prevalent, and ﬂat-panel display will gradually approximate the human visual limit. In the double G era, improved communications capabilities will support myriad UHD display applications. 2K will become the baseline video and display requirement, 4K will become the mainstream requirement, and 8K will become mainstream for sports matches, hot event live streams, and movies. 8K display will also become a new benchmark for high-end consumer electronics products.

13 Flat-Panel Display Approximating the Human Visual Limit Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

05 Immersive 3D VR Display Resolu- tion Requires Improvement

Although ﬂat-panel technology supports 3D imaging, it does not provide mobile parallax. With users viewing the same image at different angles, and the experience lacks the full sensation of reality. VR provides users with 3D immersive experience and creates a path to the visual limit. Many factors affect VR experience, including the resolution, FOV, color gamut, refresh rate, screen response time, and degrees of freedom (DOF). Resolution is the primary determinant, and VR poses high requirements on it. Full-view 8K VR content is equivalent to only 480p traditional video. To achieve 4K video-level resolution, the full-view resolution of VR must reach 24K.

5.1 Steady Development of VR Display Technologies

5.1.1 VR Display Principle

The two core components in a VR HMD are its lens and screen. By 2020, fast LCD will be the opti- When using a convex lens, if the object is within the focal length mal display solution for VR. With a of the lens, a magniﬁed image is displayed. VR display is based 60 Hz refresh rate and less than 5 on the magnifying glass principle. The screen is within the focal ms response time, latency-induced dizziness can be effectively mitigat- length of the lens. In this way, the user's eyes can see the ed, meeting VR display requirements magniﬁed content on the screen on the other side of the lens. and reducing VR hardware costs. Later, with the development of VR content optical technologies, micro OLED will become the best solution for VR display. The response time of less VR screen Lens than 0.1 ms minimizes dizziness. The wide color gamut of over 80% and 10000:1 high contrast bring perfect 1 x focal length display effect. The weight of the Human eye devices will be greatly reduced, improving user comfort.

Xiao Shuang, VR/AR Department of BOE Figure 5-1 VR display principle

Immersive 3D VR Display Resolution Requires Improvement 14 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

5.1.2 High Deﬁnition Is the Greatest Challenge for VR Display Technologies

Due to its special display mode, VR poses higher requirements on screen refresh rate, response delay, FOV, and resolution.

If the screen refresh rate is too low or the response time is too long, the user may feel dizzy. The screen refresh rate of entry-level VR must be at least 60 Hz and the response delay must not exceed 5 ms. Currently, most HMDs offer refresh rates of 90 to 120 Hz, and the response delay can be less than 3 ms, meeting requirements.

VR is intended to be an immersive experience, so the FOV is key. If the FOV of the VR header is less than that of human eyes, about 110°, black edges appear around the image, which severely affects the visual experience. Currently, the FOV of most HMDs can reach 100°, and some even reach 200°. Therefore, the FOV has little impact on VR experience.

The current bottleneck to improving VR experience is the definition, especially granules and screen-door effect on images. The reasons for unclear VR images are as follows: First, the content of VR needs to be evenly distributed in 360°, but the FOV of human eyes is only about 110°; therefore, some pixels are outside the FOV at any given time. Second, the VR image should fill the entire FOV, making the pixels per degree (PPD) lower than traditional display products. Moreover, it is technically difficult to achieve high definition on the normally small screens of HMDs. Currently, VR screens have only 3K to 4K resolution, which can display 8K content. However, the visual experience is equivalent to only 480p traditional video.

Binocular Resolution Horizontal Resolution of 360° Resolution of Equivalent of VR Screens PPD VR Content Video Experience

2K (1920*960) 10 4K 240P

4K (3840*1920) 21 8K 480P

8K (7680*3840) 32 12K 720P

16K (15360*7680) 64 24K 4K

Table 5-1 Equivalent viewing experience of VR and traditional video in various resolutions

The PPD of 8K 360° VR content is only 21, whereas human eyes need more than 60 PPD to perceive granule-less pictures. Therefore, the target binocular resolution of VR screens is 16K. At this resolution, VR viewed deﬁnition is equivalent to that of traditional 4K videos.

15 Immersive 3D VR Display Resolution Requires Improvement Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Display

Lens

Now Target PPD=px/fov

20 60

PPD Gap

Figure 5-2 PPD requirements for VR screens

In addition to improving overall screen definition, the display effect can be optimized according to the special physiological structure of human eyes. The fovea centralis in the macula is the most sensitive area in the retina. The fovea centralis has a stronger resolving ability than the surrounding area, and its sensitive angle is about 5°. In view of this, it is sensible to improve the definition of the middle area of the VR screen at the expense of the definition of the surrounding area and edges, improving visual experience and reducing the technical requirements of the screens.

5.1.3 Increasing Deﬁnition of VR Screens

The screen pixel density of entry-level VR devices is about 500 PPI, and that of consumer-level devices is about 800 PPI. Both OLED and LCD screens can meet the requirements. However, to reach the professional level, the screen pixel density must be at least 1600 PPI, making LCD advantageous. The micro OLED technology will be applied in the future. It can exceed 3000 PPI and the response time is within 1 ms, meeting development requirements of the VR/AR industry.

Display technology speciﬁcation Market positioning

~3000 PPI Professional-level: Micro OLED RT < 1 ms; 90 Hz to 120 Hz

~1600 PPI

RT < 3 ms; 90 Hz to 120 Hz Consumer-level: LCD

~800 PPI

~500 PPI Entry-level: LCD and OLED RT < 3 ms; 90 Hz

Market capacity Figure 5-3 Technical requirements for VR display in different market positioning

Immersive 3D VR Display Resolution Requires Improvement 16 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

The whole VR/AR industry began developing rapidly around 2016. By 2017, the pixel density of VR/AR small screens was 500–700 PPI, and the resolution was 2K; in 2018, the resolution was 800–900 PPI, and the resolution was 3K. Now, VR/AR screens have reached and in some cases exceeded 1000 PPI, the resolution has reached 4K, and the refresh frequency has increased to 120 Hz. As a leader in the display industry, BOE has released near-eye small screens with 5600 PPI using Micro OLED. This will be applied to VR/AR products in the future, further improving Micro OLED's visual experience.

2K 3K 4K 500–700 PPI 800–900 PPI Over 1000 PPI

Figure 5-4 Development trend of VR screen resolutions

5.2 Diverse VR Content Display Modes

Unlike traditional video content, VR brings users a pure immersive virtual experience. The following types of VR content need to be displayed: traditional video sources, splicing of real-word shooting, and 3D model rendering.

Traditional 2D/3D sources can also be viewed in VR. This is VR IMAX. Although this display mode depends on existing sources, it can provide large-screen visual experience that surpasses traditional video experience. VR IMAX has become a basic function of HMDs on the market.

Figure 5-5 VR IMAX

17 Immersive 3D VR Display Resolution Requires Improvement Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Splicing of real-word shooting is mainly used for VR 360° video production. The principle is to distribute multiple cameras on a spherical surface, control all cameras to perform synchronized shooting at the same frame rate, and ﬁnally combine pictures taken by each camera to form a spherical video. Currently, VR 360° cameras can shoot in full-view 24K resolution. However, due to limited decoding capability of terminal chips, most VR content is still in 8K.

Distribution through networks

Shooting Splicing Stream servers Display on terminals

Figure 5-6 VR 360° video production process

VR 360° videos differ from traditional videos only in terms of viewing mode. Traditional video formats are still used. Viewing VR 360° content on a traditional video player simply compacts the 360 spherical picture into a rectangular picture. You can only truly view VR 360° video on VR devices.

Figure 5-7 VR video spherical picture and rectangular picture display

VR 360° videos can bring immersive video experience, but require high video resolution. The full-view 8K video watching experience is equivalent to that of traditional 480p video. To achieve 4K TV experience, the full-view resolution must reach 24K, requiring Gbps-level network transmission rate.

3D model rendering is mainly used for VR games or animation production. A material model is built in the client in advance. When a user experiences VR, the HMD or external locator collects information such as actions, gestures, and instructions in real time, and then renders and displays the image within the current FOV. In this manner, immersive experience with 6 DOFs can be provided, immersing the user in the virtual world.

Immersive 3D VR Display Resolution Requires Improvement 18 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

3D model

Rendering Display on terminals

User instruction Figure 5-8 VR game making process

5.3 Development Trend of Immersive 3D VR Display

VR can be applied to myriad scenarios, such as live sports event broadcast, VR cinema, VR 360° video, VR social networking, and VR gaming. VR has a broad market development space, and hardware sales are one major component of this space. In addition, companies can proﬁt from providing value-added services (for example, additional charging for VR sports event broadcast) and advertisements. Therefore, the VR industry has a promising future.

VR provides users with 3D immersive experience, but the current VR visual experience is still entry-level. Both the screen and content resolution require improvement.

At Mobile World Congress (MWC) 2018, BOE and Huawei jointly demonstrated the 8K VR surround effect, the world's ﬁrst mature 8K VR display solution. The solution eliminates the screen-door effect by improving resolution and creates a more natural and realistic immersion feeling, greatly improving VR user experience. Figure 5-9 8K VR display experience of BOE at MWC 2018

19 Immersive 3D VR Display Resolution Requires Improvement Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

At CES 2019, many VR vendors demonstrated their new-generation products, such as V901 of Skyworth, G2 4K of PICO, and Odin of DEUS. All had 4K screen resolutions, a large improvement over 3K in 2018. According to Huawei iLab research, 4K screens need to display full-view 8K contents. Therefore, with the increase of VR screen resolution this year, more 8K VR content will be produced.

During production of VR content with higher resolution, the splicing, modeling, and rendering steps have high requirements on terminal computing capabilities. Cloud computing is the best choice for VR content production.

In addition to head-mounted VR, naked eye VR will be gradually used in public spaces, such as in shopping malls. Naked eye VR is a closed three-dimensional space that is conﬁned through a three-sided or ﬁve-sided LED screen. In this space, people feel the 3D immersive visual effect without using a device such as an HMD.

Figure 5-10 Naked eye VR

Immersive 3D VR Display Resolution Requires Improvement 20 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Light Field Is the Ideal Form of 06 Future Display Technology

2D images can only generate a stereoscopic "illusion" for human eyes by blocking, shadow, and affine. VR/AR can achieve 3D visual experience using binocular parallax. However, VR/AR displays can only focus on a plane with a ﬁxed distance, regardless of the intended distance. Long-time use may cause weariness of human eyes. In the future, as the amount of information displayed in the display interaction increases, holographic interaction will become the display mode of choice. Among present technologies, holographic display is most likely to be realized using light ﬁeld technology.

A light ﬁeld refers to a ﬁeld in which light passes through each point in each direction, which can record and present different depths of different objects in actual scenarios, reproducing the 3D scenario visually.

6.1 Light-Field Display Technology Is in the Initial Stage

A light ﬁeld has a large amount of data to be collected, including the intensity and direction information of the light reﬂected by the surface of objects. Therefore, there are many possibilities for rendering and image generation, and images of the object in different angles and light conditions can be obtained.

The light ﬁeld content may be displayed in an ordinary screen, or may be displayed by using a VR/AR or even in naked eye hologram mode.

Flat-panel display

Ordinary flat-panel display is the simplest display mode of the light field. It can display images of different depth and view from one or more flat-panel screens. The required volume of data is determined by multiplying the number of screens by the data volume required for a common flat-panel video. This has relatively low requirements on the rendering and display technologies, and is easy to implement in light field applications. For example, in the security field, a light field camera may be used instead of a traditional camera. Compared with a single picture, a multi-view multi-depth picture may better assist in manual or intelligent monitoring.

VR/AR light-ﬁeld display

With the help of VR/AR, a 3D effect can be presented in a light ﬁeld. Using multiple cameras to shoot objects from different directions, 3D modeling can be performed, and light information of objects on different angles can be obtained by using a light ﬁeld rendering algorithm. Using VR/AR, users can view the object from various angles and feel the real sense of light change.

Light-field display can be used in scenarios such as product marketing, goods (such as antique) display, and Internet celebrity live broadcasts. If a user is interested in a product, they can learn about it through light-field display. Similarly, in the live broadcast scenario, with the light-field display, the audience no longer watches a celebrity on the screen, but can inter- act with them from various angles.

21 Light Field Is the Ideal Form of Future Display Technology Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

In addition to the inside-to-outside light-field display, the most typical application is the outside-to-inside Welcome to light fields released by Google. Users can view the contents in a spherical area by using an HMD. This spherical region is the space occupied by the cameras that collect the light field information. When a user changes their angle of view, the light changes accordingly. This light-field display may be used for VR tourism to provide a more realistic visual experience to users.

Figure 6-1 VR light-ﬁeld display (source: Welcome to light ﬁelds)

Using the VR/AR display technology, if all data recorded in the light ﬁeld is transmitted to the VR/AR HMD or local computer and the local terminal performs real-time rendering and calculation, this imposes high requirements on the storage space and real-time rendering capability of the local terminal. Therefore, cloud-based storage and rendering will become mainstream for this light-ﬁeld display mode. Cloud servers perform real-time rendering of the user's current perspective, and transmit the rendered image to their VR/AR terminal in real time. This transmission requires Mbps-level network bandwidth. Due to real-time interaction with the cloud, the rendered picture of a new perspective must be presented quickly after the user turns their head, posing high requirements for network delay. A cloud-to-terminal network round-trip time (RTT) of 20 ms, excluding delay from the collection end to the cloud, is a basic initial requirement.

Holographic light-ﬁeld display

VR/AR can help achieve three-dimensional multi-view light-ﬁeld display. However, there remains a gap in experience between wearing an HMD and seeing the real world with naked eyes. In the future, naked eye holography will be the best form of light-ﬁeld display. However, this technology is not yet mature. It remains in the demo phase in labs and is far from commercial use.

Currently, holographic light-field display mode mainly includes holographic film and multi-view light-field display. For clear display, the amount of data required by the holographic film display is much larger than that of traditional flat-panel display. According to research by DGene, the data of a 4K resolution hologram reaches 600 Mbps. The multi-view light-field display technology presents 3D effects through multiple sub-views. The required data volume is the number of sub-views multiplied by the data volume of a single sub-view, the latter being related to the resolution of the sub-view. If the resolution reaches 4K, the data volume in each sub-view reaches dozens of Mbit/s. In the laboratory, the data volume required for multi-view projection display and tensor light-field display is similar. Certainly, there is redundant overlapping data between multiple sub-views. With the improvement of compression, the amount of data transmitted over the network can be greatly reduced, but this poses higher requirements on the decoding capability of terminals. To achieve ultra-high definition 3D holographic imaging as shown in Avatar, the data will reach Tbps level.

Display technology companies are proactively investing in holographic display technology. In the current phase, technologies such as light ﬁeld, space optical modulation, optical diffraction, and eye tracking are being combined. This allows naked-eye simple multi-view, multi-intersection, and 3D display technology to be achieved on LCD screens, approaching the theoretical effect quality of holographic display interaction.

Light Field Is the Ideal Form of Future Display Technology 22 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

6.2 Light Field Content Production Has High Requirements on Computing Capabilities

The production of light field content requires a professional light field camera. A traditional camera can only record (x,y,z,λ,t) information of a light ray, where (x,y,z) indicates the position, λ indicates the wavelength, and t indicates that the light changes over time. In addition to recording information in these five dimensions, a light field camera can also record direction information of a light ray, that is, information of seven dimensions in (x,y,z,θ,φ,λ,t), where (θ,φ) indicate the included angles between the light ray and the horizontal plane and the vertical plane. Therefore, compared with traditional content production, light field content production involves recording and processing more data.

In addition to traditional indicators such as image resolution and FOV, the main indicators of light ﬁeld camera imaging performance include angle resolution and ﬁeld of parallax (FOP). FOP indicates the angle range of the aperture from which light emitted by a point on the object can enter the camera, and the angle resolution indicates the number of beams that are split in the FOP range.

Currently, mainstream consumer-level light field cameras use light field collection based on a microlens array, which is added between the imaging sensor and the lens of a traditional camera. As shown in Figure 6-2, the light of a point on an object in an FOP is divided into 4 x 4 beams by a microlens; that is, the angle resolution of the light field camera is 4 x 4. The camera collects light in 16 directions at the same time, and records 16 pixels.

The resolution of the view image depends on the angle resolution of the light ﬁeld camera. The angle resolution represents the discretization of the scene that is collected. The larger the angle resolution, the more detail the camera collects. However, the resolution of the view image is reduced. Therefore, considering the resolution of the view image and the detail of the object, both the resolving capability of the imaging sensors and the number of microlenses must be improved. However, it is difficult to improve the resolution of the imaging sensors of a single camera, with 8K being the general limit. Therefore, the resolution of each view is about 480p. Limited by the resolution of the imaging sensors of a single camera, the amount of data collected by light ﬁeld cameras based on the microlens array is equivalent to that of traditional cameras. Camera Imaging sensor lens FOP Microlens array

Angle View image resolution resolution

Figure 6-2 Schematic diagram of light ﬁeld data collection based on microlens array

The volume of the consumer-level light ﬁeld camera is generally small and limited to space. The FOP, angle resolution, and view image resolution of these cameras are relatively small, making it difficult to record objects in detail. Therefore, a professional light ﬁeld camera generally uses multiple cameras arranged in an array structure to replace the original microlens. This improves the FOP and makes available enough space for arranging the cameras, improving the angle resolution.

23 Light Field Is the Ideal Form of Future Display Technology Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Imaging 4 x 4 camera array sensor Lens

FOP

Figure 6-3 Schematic diagram of light ﬁeld data collection based on camera array

Using the camera array, the angle resolution is the number of cameras. The resolution of the view image is the resolution of a single camera. The resolution of the view image can be high (such as 2K or 4K). Much information of the actual object can be recorded, but the amount of data is huge, being the number of cameras times the data volume of a single camera. There can be dozens to hundreds of cameras, and the data of a single camera can reach 6–12 Gbps. If content is modeled locally, the original data can be compressed by thousands of times, to Gbit/s or even hundreds of Mbit/s. However, the required local servers are expensive. Dozens of servers with high-end graphics cards are required, costing over USD100,000. In the future, local and cloud collaborative production can be used to reduce the cost of local servers, in turn posing great challenges to network bandwidth and latency.

6.3 Development Trend of Light-Field Display Technologies

Currently, light-field display technologies are limited to single-layer screen displays, such as a mobile phone screen and a VR/AR screen, and cannot manifest the advantages of multiple viewpoints and multi-layer focusing of the light field technology. For example, when human eyes look at a near object and a distant object, the focus planes are different. However, when displayed on a single-layer screen, the two objects can only be shown on the same plane, potentially causing conflicts with human brain perception, causing visual fatigue after long-time viewing. Multi-plane display is the trend of light field technology. A close-up view of an image can be displayed on a plane near the user's eyes, and a distant view can be displayed on a farther plane, precisely restoring the real world that the human eyes see. This technology is gradually being applied to VR/AR screens, such as the Magic Leap One.

In addition to multi-plane display, light-field display can also realize multi-view display. Current TVs can only display specific views. Users see the same picture from different angles. With light field technology, when a user changes their line of sight, they can feel the change in the angle of the image, making the visual experience closer to real-life experience. The industry is currently able to display 45 viewpoints. With the development of display technologies, the multi-view light-field display technology will continue to mature. In the future, living rooms will no longer have regular 4K/8K TVs, but 4K/8K multi-view TVs. The network requirements will double as the number of viewpoints increase.

Light Field Is the Ideal Form of Future Display Technology 24 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

One-to-one mapping

Cameras Corresponding sub-pixel

…… Data volume of N channels of videos Collection end Display end

Figure 6-4 Schematic diagram of multi-view display

The best display form of the light field technology is naked eye hologram. The 3D holographic image can be seen directly without using auxiliary devices such as VR/AR HMDs. Avatar is a classic science fiction movie. In it, the 3D holographic sand table was extremely impressive. This is the holographic display form and the best visual experience of the light field technology.

Figure 6-5 3D holographic sandbox in Avatar (Source: Avatar)

Holographic display has a wide range of application scenarios, such as in conferencing, other communication, and concerts.

Currently, teleconferencing mainly includes telephone and video conferencing, and there is a lack of presence. With holographic technology, the holographic images of participants around the world can be transmitted to the conference room, as if they were discussing face to face. The technology can also be applied in communications, bringing distant relatives closer. There are already attempts to apply the technology to concerts, such as the Michael Jackson, Hatsune Miku, and Deng Lijun concerts. However, these concerts were displayed by holographic ﬁlm projection instead of "true" holography. With the development of holographic technology, in the future, holographic ﬁlm will be replaced by directly displaying the images. This method is more stereoscopic and closer to the actual scene in projection mode, bringing users a more vivid visual experience. Figure 6-6 Holographic remote conferencing

25 Light Field Is the Ideal Form of Future Display Technology Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

07 Challenges for Networks

To improve visual experience, be it 4K/8K, VR/AR, light ﬁeld, or naked eye holographic imaging, data volume is increasing in both content production and presentation, making cloud computing and rendering a trend. Therefore, fast network transmission is essential. The requirements for networks may differ by expected video experience. For example, in the early stage of IPTV, 10 Mbit/s could support SD video quality. However, after HD and UHD display emerged, home broadband required upgrade to 20–100 Mbit/s. Now, with the emergence of VR, networks require a new round of transformation.

Network Upgrades Driven by Video 7.1 Experience

In the late 1990s, telecom operators in China launched home broadband services. From dial-up Internet access in the early stage, China experienced several generations of network upgrades from copper lines to fiber to the home (FTTH). By June 2018, China had 830 million fixed broadband access ports nationwide, covering all cities, towns, and over 96% of villages. FTTH has become the mainstream broadband access technology, and the optical fiber ports share of all fixed broadband access ports has increased to over 86%, indicating that fixed broadband networks in China have entered the fiber era.

During the network rate evolution, service development and network construction complement each other. Before the emergence of IPTV, the major home broadband service was Internet access, and the mainstream broadband rate was lower than 10 Mbit/s. In 2004, video services were popularized after a pilot deployment of IPTV in Heilongjiang prov- ince, leading to the popularization of video boxes and driving network upgrades. The Broadband China Strategy and the issuance of IPTV licenses catalyzed network construction and the popularization of FTTx. In the meantime, TV sizes and resolutions are increasing, driving further upgrade of home broadband rate. According to China Broadband Development White Paper released by China Academy of Information and Communications Technology (CAICT), around 80% of users have a bandwidth of 50 Mbit/s or higher, and over half of users have a bandwidth of 100 Mbit/s or higher, indicating that 100 Mbit/s is a strict minimum requirement in the 2D video era.

In 2018, the Cloud VR service, facilitated by the VR OpenLab industry alliance led by Huawei iLab, has been successfully implemented in operators. It enables them to extend their video services from 2D to 3D immersive services. This also means that the home broadband rate will be further increased to gigabit-level.

Challenges for Networks 26 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Voice/email HSI/SD Video HD Video UHD 4K Cloud VR POTS/ISDN ADSL VDSL/Vec/SV 64Kbps 10Mbps 30～100Mbps GPON 10G PON 100～300Mbps 300Mbps ~ 1Gbps PSTN DSLAM FTTC/FTTB FTTH/FTTdp FTTH

2000 ~ 2002～ 2008～ 2010～ 2018～

Figure 7-1 Development history of home broadband

As for mobile networks, the prevalence of smartphones has created a new era of communications, and 4G network pipes are ﬁlled with mobile application data. If video is a typical 4G network application, VR/AR will be a typical service for 5G.

1G 2G 3G 4G 5G

communication

Text messaging Text

messaging

Multimedia

VR/AR/MR

Video Analog

Figure 7-2 Development history of mobile Internet

7.2 Network KPI Requirements of Various Video Experience Phases

The industry chain of UHD 4K videos is mature, and the industry has reached consensus in the transmission requirements for operators' networks. According to the data, 100 Mbit/s can meet the requirement of commercial-level 4K video experience. To provide superb 4K experience, the bandwidth needs to be upgraded to 200 Mbit/s or higher, considering network transmission overhead and the bandwidth consumed by other applications.

27 Challenges for Networks Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Phase Entry-Level 4K Commercial-Level 4K Superb 4K

Color depth (bits) 8 10 12

Frame rate (FPS) 30 60 120

Average bitrate (Mbit/s) 20 35 55

Bandwidth requirement 30 50 75 (Mbit/s)

Table 7-1 Network requirements of 4K videos in different experience phases

The size of most 8K TVs is twice that of 4K TVs. At the same viewing distance, 8K TVs offer a greater field of view and sense of immersion. Due to the increase in definition, the data volume of 8K TVs is four times that of a 4K TVs of the same screen size. At present, the content production and playing of 8K video are still burgeoning, and the industry lacks a unified codec standard. In addition to H.265, which supports 8K video encoding and decoding, AVS3 is also considered an 8K standard. In March 2019, the Audio Video Coding Standard (AVS) Workgroup released its draft of AVS3, which offers twice the encoding efficiency of AVS2. Therefore, the bandwidth needed for 8K video is at least twice that needed for 4K video.

For VR content, panoramic 8K video experience is equivalent to 480P experience on a traditional TV. To achieve superb 4K TV experience, the VR panoramic video resolution must reach 24K, increasing network bandwidth requirements exponentially.

Comfortable-Expe- Phase Fair-Experience Phase Ideal-Experience Phase rience Phase Typical full-view 4K (3840 x 1920) 8K (7680 x 3840) 12K (11,520 x 5760)/24K (23,040 x 11,520) resolution Typical terminal 2K 4K 8K resolution

FOV 90° to 110° 120° 120° to 140°

Color depth (bits) 8 8 10–12

Coding standard H.264/H.265 H.265 H.265/ H.266

Frame rate 30 30 60–120

Compression ratio 133 230 410 (12K), 1050 (24K)

Typical Full-view: 90 Mbit/s Full-view: 290 Mbit/s (12K) 1090 Mbit/s (24K) 40 Mbit/s VR bitrate FOV: 50 Mbit/s FOV: 155 Mbit/s (12K) 580 Mbit/s (24K) video service Typical Full-view: 140 Mbit/s Full-view: 440 Mbit/s (12K) 1.6 Gbit/s (24K) bandwidth 60 Mbit/s requirement FOV: 75 Mbit/s FOV: 230 Mbit/s (12K) 870 Mbit/s (24K)

Table 7-2 Network requirements of cloud VR video services in different experience phases

Challenges for Networks 28 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

In addition to bandwidth, strong interaction of VR also challenges network latency. According to an analysis by Huawei iLab, the MTP must be within 20 ms to ensure that users do not become dizzy. In the case of cloud VR, VR content is stored and rendered in the cloud and then streamed to terminals over a network, increasing E2E latency. Terminal-cloud asynchronous rendering can mitigate latency caused by streaming.

The device-cloud asynchronous rendering technology converts two serial processes, namely, cloud rendering and streaming, and terminal refreshing, to parallel processing. That is, when refreshing the picture, the VR terminal uses the nth rendered frame sent by the cloud rendering platform as the basic frame to perform secondary projection. Meanwhile, the cloud rendering platform processes frame n+1 in parallel with the VR terminal. In this case, the motion-to-photon (MTP) latency is determined by the terminal, not cloud rendering or streaming, meeting the MTP latency requirements.

Cloud rendering platform Computing, rendering, and encoding Rendering of frame n+1 Parallel Processing

Rendered/Basic frame n Latest posture VR terminal Secondary Display and location (Posture and position, depth projection frame N information map, motion vector)

Figure 7-3 Terminal-cloud asynchronous rendering technology

Although cloud rendering and network latency does not affect MTP latency, the latency still affects the visual experience of users. Excessive latency can cause black edges and image distortion. Therefore, the cloud processing and network transmission latency still need to be minimized. According to lab tests, the cloud rendering and streaming latency must be limited to 30 ms to achieve ideal experience.

Cloud Processing Network Transmission Terminal Processing Transport Home Wi-Fi network network Terminal Content cloud/ Motion GPU engine cloud capture Processing in the cloud (logic computing, Network Decoding and content rendering, transmission synchronization Asynchro- Anti- Screen coding, data sending, etc.) Fair-experience nous time distortion lighting ≤30 ms ≤20 ms ≤20 ms phase warping Comfortable-ex- ≤25 ms ≤15 ms ≤10 ms perience phase ≤16 ms ≤8 ms ≤6 ms Ideal-experience phase Fair-experience phase: ≤ 70 ms

Latency Comfortable-experience phase: ≤ 50 ms planning Ideal-experience phase: ≤ 30 ms ≤20 ms

Cloud rendering and streaming latency MTP

Figure 7-4 Recommended E2E latency planning for strong-interaction VR services

29 Challenges for Networks Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

The superb visual experience of light field video requires multiple layers to display the depth information. Therefore, under the same resolution and frame rate, the network bandwidth required by light-field display is several times or even dozens of times that required by normal video. If VR/AR, which already contains a large amount of information, is used to display light field content, the required network bandwidth will be even larger; if holographic field display is used, 6G networks (operating at terahertz) or 10G or terabit broadband networks are required. For interactive services, E2E network latency cannot exceed 40 ms, and for strong-interaction services such as games, the latency cannot exceed 20 ms.

100M–1000M/4G–5G 10G–terabit broadband Network networks networks/6G networks /Mbps

24K 1.6Gbps

1000

500Mbps + 500 8K 140Mbps 8K 100 4K 140Mbps 2K 37.5Mbps 12.5Mbps 4K 37.5Mbps

Flat-panel display VR/AR Light-ﬁeld display Service

5 years 0–5 years

10 years 0–10 years

5–10 years

Maturity time Figure 7-5 Maturity and network requirements of various visual experience phases

Challenges for Networks 30 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

7.3 Target Network for Superb Visual Experience

To achieve superb 2D 4K/8K video experience, the minimum home network bandwidth is 200 Mbit/s. On the user side, in addition to access bandwidth, Wi-Fi speed is also an important factor affecting user experience. The 802.11ac standard uses a wider radio frequency (RF) bandwidth and higher-order modulation technologies to increase the transmission speed to 1.73 Gbit/s, ensuring the Wi-Fi network throughput. On the access side, according to Table 1-3, GPON can provide only 156 Mbit/s for each user even at a split ratio of 1:32. Therefore, in the long run, upgrading GPON to 10G PON is inevitable.

PON Capacity Split Ratio Convergence Ratio Attainable Bandwidth

1:64 50% 32 Mbps EPON 1 Gbps 1:32 50% 64 Mbps

1:64 50% 78 Mbps GPON 2.5 Gbps 1:32 50% 156 Mbps

10G EPON/ 1:64 50% 312 Mbps GPON 10 Gbps 1:32 50% 625 Mbps

Table 7-3 Bandwidth available to FTTH users

For Cloud VR services, network requirements in the fair-experience phase approximate those of 2D 4K/8K video, and reusing the existing network architecture suffices. However, with the popularity of Cloud VR applications, consumers will pursue superb experience, requiring higher deﬁnition, zero latency, and no artifacts. Services that can offer superb experience will be more competitive. In this case, Cloud VR has higher requirements for network bandwidth and latency, and home broadband plans need to be upgraded to 1000 Mbit/s. In addition, to ensure ultra-low latency, the network architecture needs to be upgraded. CDN needs to be moved down to be closer to users, and edge computing is needed. Determin- istic low-latency Wi-Fi and all-optical networks are goals for future superb Cloud VR experience.

31 Challenges for Networks Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

3 Intelligent O&M system based on latency and bandwidth

Regional cloud Center Economical IP network cloud IP Internet BRAS CR Regional cloud: 20 ms latency circle All-optical quality network

OLT 2 Deterministic low- All-optical high-quality network latency Wi-Fi 1 Internet access/Video service Competitive air-interface Strong-interaction Cloud VR service ONT slicing /AP Guaranteed air-interface slicing

Figure 7-6 Target network for superb-experience Cloud VR

In terms of home Wi-Fi, 802.11ac improves the air interface rate to Gbit/s. Unfortunately, only the 5 GHz frequency band is supported. Moreover, Wi-Fi performance is prone to deterioration due to the attenuation caused by complex home environments, and when home Wi-Fi is shared by multiple users, even less bandwidth is available for each.

Room 5

0 Mbps

Room 3 Room 4

20 Mbps 10 Mbps

Room 1 Room 2

100 Mbps 20 Mbps

Figure 7-7 Wi-Fi performance issues

Challenges for Networks 32 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

The next-generation Wi-Fi needs to solve the efficiency problem of Wi-Fi networks caused by multi-user access. The Wi-Fi 6 (802.11ax) standard introduces technologies such as uplink MU-MIMO, OFDMA frequency division multiplexing, and 1024-QAM higher-order coding to solve network capacity and transmission efficiency problems. In dense user environments, the per-user throughput of Wi-Fi 6 is at least four times that of 802.11ac, and supports three times the number of concurrent users. Therefore, Wi-Fi 6 is more competitive for large-scale commercial use of Cloud VR.

In addition, network slicing on air interfaces on the AP and ONT side also effectively ensures service experience. Air interface slices can be classiﬁed into contention-based slices and guaranteed slices based on their service priority. For Internet access services, contention-based air interface slices can be used; for high-priority services, guaranteed air interface slices can be used to ensure deterministic latency and good user experience.

In the access segment, with large-scale deployment of Cloud VR and popularization of gigabit home broadband, GPON access cannot easily meet Cloud VR service experience requirements. GPON should be upgraded to 10G PON, which supports 12K Cloud VR at a split ratio of 1:32 and meets network requirements for the coming 3-5 years.

On the bearer network side, the future network may be diversiﬁed. For highly latency-sensitive strong-interaction Cloud VR services, the network transmission latency cannot exceed 8 ms. According to the Low Latency Optical Network Technical White Paper of China Telecom, optical network is the lowest-latency mainstream communications technology, and the latency is predictable. The latency comparison diagram shows that a lower layer introduces a smaller processing latency. In terms of magnitude, the introduced processing latency is ns-level on L0, μs-level on L1, and ms-level on L2 and L3. The latency introduced in the transmission over ﬁbers depends on the transmission distance (5 μs/km).

More latency 7 Application Presentation Data 6 Session 5 TCP windowing Segments 4 Flow control Packet re-send

DSI Layer Address lookup Packet 3 forwarding Routing Packets

Store-Forward, Line 2 coding Switching Frames Data Unit

Framing 1 FEC Bits

Less Latency

Figure 7-8 latency magnitudes of 7 OSI layers

33 Challenges for Networks Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

An OTN device, which uses wavelength division multiplexing, is on the bottom L0 and L1. Therefore, the latency introduced by OTN devices approaches the physical limit, and a stable latency can be ensured. In all-optical network transmission, the latency can meet the requirements for an ideal Cloud VR service experience and even services that require lower latency.

In the light-ﬁeld display phase, massive volumes of data can be used to replicate the actual scenario and provide superb visual experience, which is more challenging for network transmission. With the rapid development of science and technology and the decreasing cost of optoelectronic devices, the entire network architecture may change radically and demon- strates the following features:

Bandwidth increases according to Moore's Law The capacity of all-optical network devices increases exponentially. Moreover, the per-bit cost tends to decrease through technology innovation to meet the bandwidth requirements of innovative services.

Simplified sites Networks need to be flattened and site integration needs to be improved to significantly lower network construction costs, including those incurred from equipment room space, equipment and air conditioning power consumption, and manual fiber connections and grooming.

Evolution to autonomous driving networks Networks must be autonomous and intelligent to support agile provisioning of new services and shorten their time to market (TTM). In addition, networks must support intelligent O&M to enable accurate fault prediction and automatic fault locating, minimizing OPEX and assuring E2E user experience.

Technically, in the home segment, FTTH cannot support Terabit capacity. In this case, ﬁber to the desktop and ﬁber to the room are required. In addition, home Wi-Fi needs to operate at a frequency band of 60 GHz or higher.

In the metro network, the ITU-T G.698.4 (formerly known as G.metro) is a much-anticipated access technology. It uses low-cost tunable lasers and is port-agnostic and wavelength-adaptive. This greatly simpliﬁes network construction and O&M and reduces the TCO. It also supports wavelength-level connection; physical isolation; highly available, independent, and secure connections; and independent bandwidth upgrade.

On the bearer network, the transmission distance is typically far from the transmission limit of optical ﬁbers, especially in real-world networks. There remains signiﬁcant room for improvement in capacity, transmission distance, and power consumption. The oDSP algorithm, as the most important part, is also far from full potential. In the future, more advanced chip manufacturing techniques and continuous optimization of the CMS algorithm will increase the transmission capability of networks and reduce transmission power consumption. In addition, lossless rate adjustment, dynamic bandwidth setting, and chip-level AI functions will become important parts of autonomous driving networks.

Challenges for Networks 34 Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

08 Future Prospects

Looking back on the development of the communications and electronics industries, a major disruption has occurred every 10 to 15 years, from PC release by IBM to PC popularization, the upsurge of the Internet at various portals and network forums, mobile Internet spreading worldwide, and now to popularization of apps such as Facebook and Netﬂix. New terminal platforms breed new application platforms, and new application platforms increase terminal platform value. In the near future, VR/AR has the potential to become the next terminal platform.

However, current VR/AR devices suffer problems, and user experience needs to be improved. A network with higher performance will enable the development of VR/AR. First, the rate is greatly improved. VR/AR terminals no longer have high local computing requirements, and the cloud can complete most data processing, saving hardware requirements for local storage and computing and greatly reducing the volume, weight, and cost of devices. In addition, product forms will be improved, which are more suitable for long-term use and wear.

Second, the change of terminals will inevitably lead to an upgrade of interaction. The next-generation computing platform needs a new human-machine interaction interface, replacing the keyboard and mouse of PCs and touchscreen of mobile phones. VR/AR applications will create a more natural human-machine interaction mode.

Third, the center of interaction will also be transferred, and the new interaction mode will be people-oriented. The human-machine interface goes through the command character interface, to the Windows graphical interface, and then to the current Android/iOS graphical user interface of mobile terminals. From the perspective of display, human-machine interaction has developed from no graphics to 2D graphics. The 2D graphics display quality will continue to improve and will evolve towards 3D holographic display in the near future, which is more in line with the natural visual experience of humankind. The traditional mode of human operation (limited instructions + machine's passive response to limited information) will transform into a new mode of digital simulation (all information + people's active selection of required content). Traditional single-screen and desktop displays will disappear. The obvious change is the operation interface. In addition, human-machine interaction does not require special learning, and simply uses natural interaction habits, such as dialogue, gesture, and visual manners, greatly improving interaction efficiency. In the future, with the development of communications and display technologies, content will be redeﬁned and developed towards multi-dimensional and multi-directional interaction, building a 3D IoT. In the 3D IoT future, the digital world will break through the restraints of mobile phones and computer screens, integrating naturally with the real world. 3D IoT not only brings about changes in content forms, but also enables the Internet to evolve into a space for sharing, interaction, and collaboration.

From black-and-white to color, and from SD to HD, the display industry has undergone technological revolutions. As we enter the double G era, future home displays will vitally be ultra HD and large in size. Flexible AMOLED and VR/AR are key to future display technologies. Light ﬁeld and holography will further increase network requirements and may require 10 gigabit to terabit broadband or 6G mobile networks in the future. At that time, the virtual and real world will coexist as in Ready Player One, and the 3D holographic sandbox in Avatar will become reality.

As a world-leader in semiconductor display, BOE has always been committed to leading display port innovation. BOE seeks to open applications and technologies, and will collaborate with industry partners to promote technological innovation and application in the display and electronic technology industries for everyone's beneﬁt. iLab is an ultra-broadband network innovation lab of Huawei. It is committed to researching scenarios, experience, ecosys- tems, and friendly networks. It studies network friendliness from the perspective of user scenario and experience, and studies the industry and ecosystem from the perspective of network impact. Huawei iLab is working with industry partners to facilitate business and technical innovation, industry development, and promotion of an open industry ecosystem for a better connected world.

35 Future Prospects Ubiquitous Display: Visual Experience Improvement Drives Explosive Data Growth

Prepared By:

BOE Yuan Feng, Luo Huan, Xiao Shuang, Si Jie, Li Yunshuang, Chen Jie

Huawei iLab Hu Yan, Xu Lingling, Gao Li, Lin Bizhu

CAICT Chen Xi

36 Copyright © Huawei Technologies Co., Ltd. and Boe Technology Group Co., Ltd. 2018. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. and other BOE trademarks are trademarks of Boe Technology Group Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the commercial contract made by Huawei and BOE. All or partial products, services and features described in this document may not be within the purchased scope or the usage scope. Unless otherwise speciﬁed in the contract, all statements, information, and recom- mendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recom- mendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd. BOE Technology Group Co., Ltd. Address: Bantian, Longgang District Shenzhen 518129, P. R. China Address: No.12 Xihuanzhong RD, BDA, Beijing 100176, P. R. China Website: http://www.huawei.com Website: http://www.boe.com Email: [email protected] Hotline: 01060965000 Hotline: 4008302118