Annual Report NHK 2019 Science & Technology Research Laboratories 2019 NHK Science & Technology December 2020 Research Laboratories NHK Science & Technology Research Laboratories 2019 Nippon Hoso Kyokai [Japan Broadcasting Corporation] Table of Contents Greetings ············································· 1 Accomplishments in FY 2019 ···············2 1 Reality Imaging - Spatial imaging ···········4 5 Smart Production - Universal service ···36 1.1 3D imaging technology 4 5.1 Automatic closed-captioning technology 36 1.2 AR (Augmented reality) / VR (Virtual reality) 7 5.2 Audio description technology 36 1.3 3D imaging devices 8 5.3 Machine translation technology 37 5.4 Information presentation technology 38 2 Reality Imaging - 8K Super Hi-Vision ···10 6 2.1 Video systems 10 Devices and materials for next-generation broadcasting ··············41 2.2 Cameras 11 6.1 Imaging technologies 41 2.3 Displays 12 6.2 Recording technologies 43 2.4 Sound systems 13 6.3 Display technologies 44 2.5 Video coding 15 2.6 Satellite broadcasting technology 17 2.7 Terrestrial broadcasting transmission 7 Research-related work ·························46 technology 18 7.1 Joint activities with other organizations 46 2.8 Wireless transmission technology for program contributions 20 7.2 Publication of research results 49 2.9 Wired transmission technology 22 7.3 Applications of research results 52 3 Connected Media ································· 23 3.1 Content provision platform 23 3.2 Content-matching technology for daily life 25 3.3 IP content delivery platform 26 3.4 TV-watching robot 27 3.5 Security 29 3.6 IP-based production platform technology 29 4 Smart Production - Intelligent program production ··········································· 31 4.1 Text big data analysis technology 31 4.2 Image analysis technology 32 4.3 Speech transcription technology 33 4.4 New image representation technique using real-space sensing 34 4.5 Promotion of the use of AI technology 35 NHK Science & Technology Research Laboratories Outline ···················54 Greetings Kohji MITANI Director of NHK Science & Technology Research Laboratories HK Science & Technology Research Laboratories (STRL), a part of the public broadcaster NHK and the Nsole research facility in Japan specializing in broadcasting technology, is working to create a rich broadcasting culture through its world-leading R&D on broadcasting technologies. Fiscal year 2019 marked one year since 4K and 8K satellite broadcasting was launched in December 2018 after many years of R&D. It also saw the start of the test delivery of the simultaneous continuous online streaming and program catch- up service NHK Plus, which was officially launched in April 2020. It was a year that gave us the realization that NHK is making steady progress toward the goal of becoming a “public service media.” We have been driving our research activities according to the NHK STRL 3-Year R&D Plan (FY 2018–2020), which is aimed at creating new broadcasting technologies and services for the future around 2030 and 2040. We will continue to push ahead with our R&D under the three pillar concepts of “Reality Imaging” technologies to deliver video and audio with a higher sense of presence and reality, “Connected Media” technologies to achieve more convenient broadcasting and services through the use of the internet, and “Smart Production” technologies to utilize artificial intelligence (AI) to support program production and expand universal services. On June 1, 2020, NHK STRL celebrated the 90th anniversary of its foundation. We are deeply grateful to all those concerned, and the viewers and residents of neighboring communities who have supported us up to today. As in the past, the entire STRL will work as one to engage in R&D to play a leading role in the R&D on broadcasting technologies and services. This annual report summarizes our research results in FY 2019. It is my hope that this report will serve as an impetus for you to better understand NHK STRL’s R&D activities. I also hope it will help us build collaborative relationships that promote R&D and opportunities for cooperative creation utilizing the results of our efforts. Finally, I would like to express my sincere gratitude for your support and look forward to your continued cooperation in the future. 90th anniversary NHK STRL ANNUAL REPORT 2019 | 1 Accomplishments in FY 2019 Reality Imaging - Spatial imaging NHK STRL is researching three-dimensional (3D) imaging technology that offers more natural 3D images without the use of special glasses with the goal of realizing a future 3D image broadcasting service. In FY 2019, we made progress in our research on a method for improving the resolution of 3D images and studied the color moiré reduction of 3D images for portable terminals. We also investigated a technology that achieves both the reduction of the amount of information of 3D images, which contain a huge amount of depth information, and an effective image representation method based on human vision property. In our work on the application of augmented reality (AR) and virtual reality (VR), we developed prototypes that allow people to try new viewing High-resolution VR images experiences and services and promoted the service concept through exhibitions such as projected on a large cylindrical the NHK STRL Open House 2019. To achieve a device that displays more natural 3D images, we conducted fundamental experiments on spatial light modulators and screen researched the display of 2D images using narrow-pixel-pitch liquid crystal and high- speed optical beam scanning with a narrow-pitch optical phased array. →See p. 4 for details. Reality Imaging - 8K Super Hi-Vision We are conducting R&D on a production system to enable program production in full- featured 8K, which is the ultimate format of Super Hi-Vision (SHV), and research to identify synergy effects between the parameter sets. For imaging technologies, we investigated an autofocus function suitable for 8K three-chip cameras and a method for suppressing image degradation caused by higher frame rates. We also researched ways to increase the sensitivity of solid-state image sensors. For display technologies, we developed a 4K flexible OLED display formed on a plastic film and researched technologies for increasing the image quality of OLED displays. For audio technologies, we researched next-generation object-based audio and technologies for enhancing the spatial sound 4K flexible OLED display representation as well as technologies for reproducing 22.2 multichannel sound at formed on a plastic film home. In addition, we conducted full-featured 8K live production and transmission experiments using a 21-GHz-band broadcasting satellite as the verification of the full- featured 8K video coding standard and technologies for expanding the transmission capacity of satellite transmission systems. With the goal of realizing terrestrial 4K/8K broadcasting following the 4K/8K UHDTV satellite broadcasting launched in December 2018, we researched functional additions to the transmission system and conducted large-scale field experiments as part of Technical Examination Services of the Ministry of Internal Affairs and Communications. We also researched an IP multicast distribution method that uses commercial closed networks of FTTH owned by cable TV service providers. To achieve a wireless transmitter for producing a wide variety of 4K/8K programs, we worked on the development of ARIB Standard for 1.2-GHz/2.3-GHz-band mobile relay FPUs and researched a low latency wireless camera using a millimeter- wave band. →See p. 10 for details. Connected Media As a study of technologies for providing content linking broadcasting and broadband networks, we researched Hybridcast Connect and other technologies that connect broadcast content with smartphones and IoT-enabled devices. We released a software development kit (SDK) and other tools for Hybridcast Connect as open source software to support service providers in developing device linkage applications. We also investigated the effectiveness of the utilization of viewers’ viewing history through demonstration experiments linking broadcasting and various services. In our research on IP content delivery platform technology that allows viewers to view content by using the internet, we researched a quick-response delivery technology to stabilize the viewing quality of video delivered on the internet and achieve smooth viewing operation and a technology for delivering the same content to many terminals stably and efficiently in a space crowded with users. We also continued with our research on TV-watching robots, which are expected to become partners to make TV watching more enjoyable. We Viewing experiments using incorporated a function for personal information and privacy protection into a robot that TV-watching robot we developed previously and also conducted viewing experiments using the robot. In our research on cryptography and information security, which are essential to ensure high security and reliability of services in the internet age, we studied cryptography algorithms including one that can be used for post-quantum computer measures and one for integrated broadcast-broadband services. In our research on IP-based production platform technology to realize efficient program production utilizing IP, we prototyped multiformat IP transmission equipment that transmits material in different formats (2K/4K/8K) from a venue to a broadcast station efficiently and developed a system monitoring