PROXCHANGE WHITEPAPER Active Storage for Digital Content™ Transcoding on a Grid Computing Framework Simplifying the Delivery of Multiple Media Formats Simon Eldridge Product Manager Omneon, Inc. www.omneon.com (WP/02/08) PROXCHANGE WHITEPAPER Abstract Beyond the advantages of flexible central storage, the grid storage systems also offer spare processing power that can be harnessed for other applications. The ContentServers are X86-based servers that run standard Intel® processors and have a specific amount of Broadcasters and media organizations have various requirements for converting media between formats. Numerous video system system memory. These resources are used to manage the disks and files slices in storage, but there is, in fact, more power available than is required purely for storage management. vendors, multiple video formats and countless wrapper types have created a landscape of non-interoperable systems. With the move to file-based workflows and the requirements to move media between systems, it is necessary to have a system for format conversion Through a software Applications Programming Interface (API), applications can use the spare processing capacity of ContentServers. This is ideally suited to computationally-intensive content processing applications as the ability to distribute this processing across in order to ensure that content can be accessed on the destination device. The ideal transcoding system would be configurable to a large number of grid nodes vastly increases the performance of the application. automatically deliver content in the correct format to any destination device and be able to do it faster than real time. There are a number of components to the GPF, each designed to provide a specific function within the framework – these functions This paper provides an overview of a transcoding application running on a distributed grid processing architecture and the benefits of that include allocating ContentServer nodes to applications when they are requested, allowing foreign applications and processes to be launched on the ContentServers, and a sub-grid pool of processors within a grid storage system to process the media as necessary. approach over traditional transcoding solutions. It also discusses the unique aspects of such an approach and examines some cases of how the technology may be used by broadcasters to improve their workflow and provide a platform for new business opportunities such as Two components key to the GPF are supplied by the application provider – the application controller, which is the core application utilizing the GPF, and the application process that runs on the ContentServer. content distribution for web and mobile TV. FIGURE 1 How grid processing works I. Transcoding in a Broadcast Environment With the increased move towards file-based workflows, transcoding has become an integral part of broadcast operations in order to allow Grid Resource Manager • Software running on ContentDirectors content interchange between systems from different vendors, re-formatting for different internal purposes and creation of new formats for Application Controller • Manages Grid Processing on MediaGrid distribution on new platforms. Historically, content would be moved between systems by simply playing it out of one device in format A and • Application server that connects to MediaGrid • Responds to requests from Application Controller recording it in to another as format B, thus mirroring long established workflows of tape to tape operations. While proven and reliable, this • Establishes environment on ContentServer for application process methodology is well suited to tape based operations and is always limited by physical infrastructure and a real-time transfer. • Uses MediaGrid API for communication Traditionally, transcoding was the converting of video or audio material from one format to another, but now the term encompasses numerous content re-formatting functionality from converting the video or audio format from one to another, changing wrapper types, bit-rates, resolutions, converting between SD and HD standards, altering aspect ratios or even standards conversion between NTSC and PAL. With a plethora of formats, wrappers and third-party applications, transcoding is something that must happen in practically all broadcast operations, but should it be a function that requires interaction? Surely the interchange of content between systems should be based on configurable rules based on what formats the receiving device can handle and what the most efficient format Grid Application Loader for the chosen use is. One thing is certain: with the explosive growth of file-based workflows and new media formats, the need for • Software running on ContentServers Application Process Sub-Grid • Sets up application process and launches appliacations transcoding and the high levels of processing power it requires is only going to get bigger and more intensive. • Linux process in ContentServer that • Subset of ContentServers • Responds to requests from Application Controller executes the application job used by a given application II. Introduction to the Grid Processing Framework Applications that utilize the GPF are able to spread content processing across all available ContentServers Today it is possible to transcode multiple content streams simultaneously at faster than real time from within a single transcode in systems thus enabling distributed parallel processing in order to achieve must faster results. application – provided that the application is harnessing the power of a Grid Processing Framework (GPF), or Grid Computer. The grid processing framework is a new concept that is a derivative of grid storage systems. Details about grid storage can be found III. Grid Transcoding Overview elsewhere, but for the purpose of this paper it is necessary to understand just a few basic concepts. In the grid storage system that is currently commercially available in the broadcast industry, the basic storage building block is a ContentServer, which houses Content format conversion is traditionally a very computationally intensive task, and so the ability to spread it across multiple nodes and manages a number of disk drives. Any number of ContentServers, along with some additional processing components, together in order to increase performance is a very attractive proposition. Simply put, transcoding was an obvious target application for use form the grid storage system. Currently these grid storage systems are being deployed as centralized shared storage for large media on the grid processing framework. facilities. In addition to improving a facility’s workflow by allowing shared access to all material in the facility, grid storage also offers the benefit of easy expansion of storage space and bandwidth (figure1). The transcode application described here is divided into two components – JobDirectors and JobScalers. Each of these is capable of initiating transcode jobs, and the JobDirector is also responsible for storing the job queue and managing the priorities of tasks. JobScalers are added to the system to increase processing capabilities for larger grid systems. PROXCHANGE WHITEPAPER ProXchange JobDirectors Users can define transcode presets that can be used multiple times throughout the system. Each preset determines the parameters The JobDirector is the core component of the transcoding system. It is a server connected directly to the grid and runs the core required to create the desired output format such as wrapper type, format, bit-rate, resolution and audio format. These presets can transcoding software. Functionality includes the job management and prioritization application and the logic used to split, distribute, then be applied by dragging them to watch folders on the grid. When doing so, users select an output folder (also on the grid) where and join content throughout the grid. Content re-wrapping is also undertaken on the JobDirector. Each JobDirector is capable of the transcoded content will be placed. Folder priorities can also be set as normal or high. Multiple transcode presets can be applied addressing 12 grid nodes allowing a single transcode job to be spread across up to 12 individual processors. to a single folder if incoming content needs to be transcoded to multiple formats. When a task is received, the target file is unwrapped and split into smaller parts within the JobDirector based on GOP boundaries of Job status can be viewed through a web application. Users can navigate to the appropriate web address for the JobDirector or click the content. Each of these components represent jobs within the system master job queue. The JobDirector then distributes these through from the Manager application. The interface also provides information on the performance of each job with a simple value jobs to each ContentServer available within the sub-grid assigned to the task. Each ContentServer converts the component it has to represent how fast that job was processed in relation to the length of the input content. For example, if a 30 minute clip took 15 been assigned to the destination format specified. The resulting file components are returned to the JobDirector, and the output file minutes to complete, the interface will report that the job was processed in 2x – that is, two times faster than real-time. is constructed in the desired output format and wrapped in the wrapper format determined by the transcode rule. The JobDirector also checks the final output file and verifies