Spring 2015 STRL Bulletin NHK Science & Technology Research Laboratories ISSN 1345-4099 “Video File Transmission” no. 60 -Automatic transmission of recorded video to CONTENTS a broadcasting station he Science & Technology Research Laboratories (STRL) has been working on improving the transmission environment so that news video materials can be Tgathered from locations that would normally present problems. One recent accomplishment involves the construction of “video file transmission” technology that “Video File Transmission” is capable of delivering camera footage to a broadcasting station via a public network -Automatic transmission of recorded video to a broadcasting station ........1 such as the Internet. NHK has been working to enhance news reporting through the development of a Trends in Advanced Video Representation Techniques .............2 skip-back recorder, which automatically records footage before and after an earthquake by detecting a tremor. Our new technology will also enable priority-based automatic Video Analysis Techniques that Enhance Video Retrieval ....................9 transmission of weather-related information. This technology records second-time scale video signals in a storage media as file data. These files are combined after simultaneous transmission. By retransmitting some of the files on a different line, the transmission will reach the broadcasting station Challenge / R&D / Treatise / even if some of the transmissions are interrupted. This enables reliable, high-quality About us / NHK Technology video transmissions on interruption-prone lines such as mobile phone networks. A transmission device incorporating this technology and controlled from the broadcasting station can crop out video segments from constantly updated stored video. Because simultaneous transmission of video signals from multiple locations may use up the available bandwidth capacity, the number of files sent at one time can be varied in order to control the transmission rate. For instance, a remote command from the broadcasting station can prioritize video footage from locations experiencing more severe tremors during an earthquake (Figure). STRL will examine the practicality of automatic transmission of footage from remotely located cameras via mobile phone networks during natural disasters, including earthquakes, torrential rain, flooding, and volcanic eruptions. We will continue our studies on automatic monitoring at more locations in order to promptly deliver local video coverage during an emergency. Remotely located cameras Reception system and video file transmission system at a broadcasting Fewer simultaneous station Areas with transmissions smaller tremors Areas with larger tremors Received video footage is used in news programs Increased number of simultaneous Video signals are divided transmissions enables prioritized into small segments reception of important video data Figure: Video collection using a video file transmission technology STRL Broadcast Technology No.60, Spring 2015 ● C NHK STRL 1 Trends in Advanced Video Representation Techniques The recent applications of computer vision and applications of video analysis in program production. computer graphics (CG) have caused a dramatic expansion in the variety of video representations in TV programs and the like. This paper describes the trends 2. Issues of video production faced by broadcasters in representation techniques that use video analysis in Program production essentially consists of research, program production. We also comment on issues affecting planning, materials recording, post-production, and video production, present approaches to their resolution, transmission. Image processings such as VFX are and give examples of video representation applications additional steps in this process. using video analysis with real-time processing. As shown in Table 1, the objective of materials management is not simply to collect raw video footage, but also to enable the efficient handling of large volumes 1. Introduction of video footage by providing functions to list and search A wide variety of video representations are the available materials. At present, however, except for incorporated in TV programs. In TV drama in particular, labels indicating temporal information, users must add it is becoming commonplace to see highly realistic their own labels to enable the content of the video to representations created by adding visual effects (VFX) to be searched. Given the constraints of production, it is digital video footage during the processing and editing impractical to manually add detailed labels to a huge stage (called post-production, or “postpro”). amount of video. Moreover, once the desired video images VFX is used for a wide variety of purposes, from the have been retrieved, the job of adding information for striking, e.g., the depiction of scenes that are impossible processing that video footage has just begun. in real life, to the inconspicuous, such as the removal of The pre-visualization (“previz”) step in this table create manmade structures that would not have existed during a rough video for imagining what the finished video the times in which a period drama is set. However, while would be like3). Pre-visualizations are used for selecting VFX satisfies various production needs, the content lenses or for showing producers and performers, by way creator must invest a great deal of effort in using it. This of example, what the finished video would look like. The labor-intensive aspect is a major issue because content CGs of pre-visualizations are simplified, but still require creators also have to do all of the program production substantial knowledge and effort to produce. Hence, the work they would normally do and productions are previz step is omitted in TV program productions when constrained in terms of time and cost. the preparation period before filming is too short. In such Under these circumstances, there has been growing cases, although producers can use storyboards to convey recognition of the importance of process management in their ideas about the finished video to the performers and program production and proper management of video staff, this form of expression does not facilitate intuitive materials. For example, new workflows have been proposed, comprehension like previz does. such as the scene-linear workflow in which the original As regards the subject area extraction step listed in data (RAW data) of the captured video is left untouched the table, it often happens that is it not possible to use and is saved together with related metadata, such as lens chroma key on location. In such a case, it is necessary information acquired during filming. This video information to create subject area information (matte generation, is then handled in a consistent way, i.e., “linearly”, within also called key generation) by hand. This task takes a same color space1)2). While such management makes it considerable time and requires experience and skill. It is unlikely that video materials will be damaged or degraded, especially difficult to perform in TV program productions it also removes much of the subjectivity of the creator from that have many time-related constraints. the production environment. That is, while the aim is to How to obtain camera movement information is make the work involved in VFX more efficient to enable another issue. This information is used to composite the creator to concentrate on purely creative activities, in CGs and live-action images. To ensure that composited order to be creative it is necessary to have technology that image appears natural, it is necessary to match the supports the complicated work that is inherent to each environment information of the live-action filming process. This paper describes issues faced by broadcasters in space with that of the CG space, as shown in Table video production and shows how analysis techniques such 2. In particular, in order to make natural appearing as computer vision*1can support VFX. It also describes new composited images, the movements of the camera and the lighting conditions during the live-action filming have to be recorded and used to make the CGs. However, *1 A technique that uses a computer to analyze images captured acquiring this information places further constraints by a camera to elucidate the world of the subject and the on the production. For instance, it may require special relationship between that world and the camera. equipment and measurements to be made on the set. 2 Broadcast Technology No.60, Spring 2015 ● C NHK STRL Feature Table 1: List of VFX tasks Task Objective and details Typical technique Materials management Version management, searching, etc., of Version management and searching of materials materials using a video database (labels attached manually) Pre-visualization (previz) A simple visualization Imagining the finished CG production using authoring tools (software video before filming starts for creating information such as the shape and movements of CG subjects) Post-visualization (postviz) A simple visualization of finished video made Provisional composite of simple CG and directly after filming live-action video Extraction and compositing of subject area by Chroma key (technique of filming using a using video analysis background such as a blue screen, then using color information to separate the areas of subject Extraction and compositing and background in the video images) of subject area Specifying subject areas by hand