UPTEC IT 11 016 Examensarbete 30 hp December 2011

An Evaluation of the PandaBoard as a Set-Top-Box in an Android Environment

Torbjörn Svangård 2 Abstract An Evaluation of the PandaBoard as a Set-Top-Box in an Android Enviroment Torbjörn Svangård

Teknisk- naturvetenskaplig fakultet UTH-enheten The aim of this thesis is to evaluate the potential of the PandaBoard to take the role as an Android based Set-Top-Box (STB). A STB is a device that converts an external Besöksadress: signal into viewable video on a connected television. The PandaBoard is a low-cost Ångströmlaboratoriet Lägerhyddsvägen 1 development platform intended for mobile software development and is based on a Hus 4, Plan 0 chipset, OMAP4, developed and manufactured by Texas Instruments.

Postadress: During the thesis implementation phase a PandaBoard was configured to run Android; Box 536 751 21 Uppsala a popular operating system mainly used in smartphones. A DVB-T2 device was attached to one of the USB-ports on the PandaBoard in order to recieve terrestrial Telefon: digital television broadcasts. 018 – 471 30 03

Telefax: The main obstacle was the fact that despite very promising video accelerated 018 – 471 30 00 hardware specifications on the PandaBoard, they were not available for evalutaion. During the thesis pilot study it became clear that hardware accelerated video playback Hemsida: was not yet supported for the PandaBoard Android-port. Programmable OpenGL ES http://www.teknat.uu.se/student 2.0 shaders was used instead in order to offload the PandaBoard CPU as much as possible. Although this thesis may not be conclusive in terms of the impressive video accelration hardware present on the PandaBoard, the thesis will demonstrate what can be achieved with a dual core ARM Cortex-A9 processor together with OpenGL ES 2.0 on a system running the Android operating system. The thesis work was performed at the Syntronic Software Innovation office in Kista, Stockholm.

Handledare: David Näslund Ämnesgranskare: Arnold Pears Examinator: Anders Jansson ISSN: 1401-5749, UPTEC IT11 016 Tryckt av: Reprocentralen ITC 4 Sammanfattning

Syftet med denna rapport ¨ar att utv¨ardera potentialen av utvecklingskortet PandaBoard i rollen som en Androidbaserad set-top-box (STB). En STB ¨ar en enhet som anv¨ands f¨or att omvandla en extern signal till en videostr¨om som kan visas p˚aen ansluten TV. En PandaBoard ¨ar ett utvecklingskort som ¨ar avsedd f¨or utveckling av programvara f¨or s˚a kallade smartphones. Pandaboardkortet baseras p˚aett chipset, OMAP4, som utvecklats och tillverkas av Texas Instruments. Under examensarbetets implementationsfas konfigurerades ett PandaBoardkort f¨or att anv¨anda Android. Android ¨ar ett popul¨art operativsystem som ˚aterfinns fr¨amst i smarta mobiltelefoner. En DVB-T2 dongle kopplades in p˚aen av PandaBoardkortets USB-portar f¨or att f˚atillg˚ang till marks¨and digital-TV. Det st¨orsta hindret under arbetest g˚ang var det faktum att trots mycket lovande specifikationer f¨or h˚ardvaruaccelereread videouppspelning p˚aPandaBoardkortet, s˚avar drivrutiner f¨or Android dessv¨arre inte tillg¨angliga f¨or utv¨ardering. Under examensarbetets f¨orstudie blev det uppenbart att h˚ardvaruaccelererad videouppspelning ¨annu inte st¨oddes f¨or PandaBoardkortets androidport. Programmerbara OpenGL ES 2.0 shaders anv¨andes i st¨allet f¨or att avlasta PandaBoardkortets CPU s˚amycket som m¨ojligt. Aven¨ om denna uppsats inte ¨ar utt¨ommande vad g¨aller den imponerande h˚ardvara f¨or video acceleration p˚a PandaBoardkortet, s˚akommer rapporten redog¨ora vad som kan uppn˚asmed en dual core ARM Cortex-A9-processor tillsammans med OpenGL ES 2.0 i ett system med linuxdistributionen Android som operativsystem. Examensarbetet utf¨ordes p˚aplats hos Syntronic Software Innovation i Kista, Stockholm.

5 6 Contents

1 Background 11 1.1 The PandaBoard ...... 11 1.1.1 Ducati Sub System ...... 11 1.1.2 Pandroid ...... 11 1.2 The DVB-project ...... 13 1.3 Digital Terrestrial Television in Sweden ...... 13 1.3.1 Past and present...... 13 1.3.2 DTTV in the future ...... 13 1.4 Content protection...... 14 1.5 Set-Top-Box requirements ...... 15

2 Problem description 17 2.1 Linux kernel DVB-stack ...... 17 2.2 DVB-T2 ...... 18 2.3 Decoding Audio and Video ...... 18

3 Methods 19

4 Implementation 21 4.1 System peripherals...... 21 4.1.1 Usb devices ...... 21 4.1.2 HDMI ...... 23 4.1.3 The SD-Card ...... 23 4.2 System software ...... 24 4.2.1 DVB Utils Library...... 24 4.2.2 FFMpeg Library...... 25 4.2.3 Color space conversion ...... 26 4.2.4 OpenGL ES 2.0 ...... 28 4.3 Results and evaluation ...... 28 4.3.1 System performance ...... 28 4.3.2 Power consumption ...... 29

5 Conclusions 31 5.1 Unsolved problems...... 31 5.1.1 Audio playback ...... 31 5.1.2 Accelerated video playback ...... 31

6 Future work 33

Bibliography 35

7 CONTENTS CONTENTS

A PCTV nanoStick T2 290e PCB 37

8 Glossary

ADB Android Debug Bridge AVC Advanced Video Coding ATSC Advanced Television Systems Committee COFDM Coded Orthogonal Frequency Division Modulation DVB-T Digital Video Broadcasting - Terrestrial DVB-T2 Digital Video Broadcasting - Second Generation Terrestrial DTTV Digital Terrestrial Television EPG Electronic Program Guide GLSL OpenGL Shading Language GPU Graphics Processing Unit HDTV High Definition Television IVA-HD Image Video Accelerator - High Definition ISS Imaging Sub System MPEG Moving Picture Experts Group MUX Multiplexer OEM Original Equipment Manufacturers OFDM Orthogonal Frequency Division Modulation OMAP Open Multimedia Application Platform OpenGL ES 2.0 Open Graphics Library for Embedded Systems PCB Printed Circuit Board SDTV Standard Definition Television SIMD Single Instruction Multiple Data SoC System-on-a-Chip STB Set-Top-Box TI Texas Instruments

9 CONTENTS CONTENTS

10 Chapter 1

Background

1.1 The PandaBoard

The PandaBoard is a low-power and low-cost platform intended for mobile software development. It is equipped with a 4th generation Open Multimedia Application Platform (OMAP4) chip. The OMAP brand name is a family of System on Chips (SoC) for mobile multimedia applications developed and manufactured by the American semiconductor and computer technology company Texas Instruments, widely known as TI. The PandaBoard is equipped with the OMAP4430 SoC, which houses a dual-core ARM Cortex-A9 processor with each core running at 1GHz, a PowerVR SGX540 Graphics Processing Unit (GPU) and an Image Video Accelerator-High Definition (IVA-HD) Sub System. The OMAP4 family is one of the first dual-core Cortex-A9-based architectures on the market. A community of Linux experts and enthusiasts supports the PandaBord via the website PandaBoard.org. The PandaBoard supports playback of high definition video content at the HDTV standard 1080p, perhaps more known by its marketing term ”Full HD”. The PandaBoard’s high performance video media support is made possible thanks to its dedicated hardware; the Ducati Sub System.

1.1.1 Ducati Sub System The Ducati sub system consists of two ARM Cortex-M3 processors, the Image Video Accelerator - High Definition (IVA-HD) sub system and the Imaging Sub System (ISS). The IVA-HD sub system is the hardware for encoding and decoding video up to 1080p resolution at 30 frames per second. It supports multiple video codecs such as the Mpeg-4 AVC, also known as H.264, codec. The ISS processes pixel data coming from an external camera sensor or from memory. The purpose of these sub systems is to offload video and image processing from the main ARM Cortex-A9 CPU.

1.1.2 Pandroid Pandroid is a Texas Instruments supported port of Android for the PandaBoard. Android is an operating system for mobile devices, developed and maintained by Google. It consists of a complete system based on the open source Linux kernel, middleware i.e. libraries that can be used by developers and applications. The applications are the actual programs that the user sees on a display and interacts with. The difference between Pandroid and standard Android is a number of patches that enables support for hardware found on the PandaBoard; such as TI specific hardware not

11 1.1. THE PANDABOARD CHAPTER 1. BACKGROUND

Figure 1.1: PandaBoard Layout from pandaboard.org (CC BY-SA 3.0)

yet included in the Linux kernel. Other more directly practical patches are also included such as support for an USB mouse. Standard Android is designed for touch screen usage and lacks default USB mouse support. For the implementation the Pandroid L27.10.2 released in March 2011 is used. It is based on code from the Android version 2.2 named Froyo. This Pandroid release has the following main features:

External monitor via HDMI/DVI • USB Keyboard • USB Mouse with mouse cursor • Wifi/802.11n • Bluetooth • Ethernet • PowerVR SGX540 GPU Drivers • Note that neither the OMAP4 Ducati Sub System nor audio playback/recording is available in this release. However OpenGL ES 2.0 is supported through the PowerVR GPU drivers.

12 CHAPTER 1. BACKGROUND 1.2. THE DVB-PROJECT

1.2 The DVB-project

The Digital Video Broadcast (DVB) project was established in September 1993 to develop a common video broadcasting standard. It is a consortium consisting of over 250 different companies and organizations from 35 countries with interest in open standards for delivery of digital media services[16]. Although DVB usage is the most widespread DTTV standard, some alternatives exists. To name a few countries that do not use the DVB standards; USA uses the Advanced Television Systems Committee (ATSC) standard, Japan use the Integrated Services Digital Broadcasting (ISDB) standard. See 1.2 for an DTTV standards overview.

Figure 1.2: DTTV world coverage[1]

Currently there are over 500 million receivers in the world, compatible with a standard from the DVB-project. The DVB-project has produced a number of widely used standards. The most important are called DVB-S, DVB-C and DVB-T. TheS stands for satellite, th eC for cable and the T for terrestrial and, as the reader might assume, represents broadcasting standards for these mediums. The focus for this thesis will be towards DVB-T and DVB-T2, the latter is the next generation terrestrial standard.

1.3 Digital Terrestrial Television in Sweden

1.3.1 Past and present The political discussions regarding an introduction of Digital Terrestrial Television (DTTV) in Sweden started in the early 1990s. In June 1995 Lars Jeding was appointed by the Swedish government, to investigate the country’s possibilities for a transition from analog to digital television. Jeding proposed in his report, From Massmedia to Multimedia – Digitalisation of Swedish Television[11], that Sweden should switch over to digital transmissions as soon as possible. He concluded that DTTV would provide an “alternative to digital broadcasting by satellite or cable” and that it would “attract commercial players”. The report generated

13 1.4. CONTENT PROTECTION CHAPTER 1. BACKGROUND

TTV broadcasting issions launched

Lars Jedings reportDecisionDTTV to establish is launched D Analog switch-off DVB-T2 mpeg4 transm

1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014

Figure 1.3: Historical timeline of DTTV introduction in Sweden

a heated debate[4] whether DTTV should be introduced in Sweden or not. The decision was however taken 1 November 1997, by the Swedish Riksdag, to go over to DTTV. According to the final report Digital TV commission[5]

1.3.2 DTTV in the future A standard for mobile TV; DVB-H, emerged in November 2004 and trials have been conducted in Sweden back in 2006. Despite the positive outcome[14] of the trials, DVB-H is not currently provided in Sweden. Why was it not deployed? It seems, contradictively, that it was due to lack of public interest[18] and that Teracom needed all the frequency space they could get for HDTV. The DVB-project is currently developing future standards for 3DTV, IPTV, and the next generation hand-held TV (DVB-NGH).

1.4 Content protection

The DVB-CSA (Common Scrambling Algorithm) is a cryptographic method used to secure the Mpeg-2 transport stream payloads. Its main purpose is to support the use of scrambling and Conditional Access (CA) for digitally transmitted Pay-TV mainly in Europe. DVB-CSA was specified by the European Telecommunications Standards Institute (ETSI) and adopted by the DVB consortium in May 1994. The algorithm was designed to last for at least ten years. Nearly twenty years later it still remains a robust solution. Threats to DVB-CSA has however increased along with technology and computing power advancements. Original Equipment Manufacturers (OEM) were not allowed to implement the algorithm in software and it was only available under a Non-Disclosure Agreement (NDA) from an ETSI custodian. This was due to security reasons. As a result, very little was publicly known about the CSA algorithm for almost a decade. This changed in the autumn of 2002 when a Windows program called FreeDec emerged which implemented the CSA in software. FreeDec was reverse–engineered and the results were published on the web. The publication meant that the details about the algorithm became available to the public. Even now when the algorithm is known, no attack has been successful. The details of CSA and are beyond the scope of this thesis. They vary between broadcasters and are usually implemented on a smart card which is required to view encrypted pay-TV transmissions. The actual key for the CSA is called common key and is usually changed every 10–120 seconds. The great relevance of CSA lies in the fact, that every encrypted digital Pay-TV transmission in Europe is secured using

14 CHAPTER 1. BACKGROUND 1.5. SET-TOP-BOX REQUIREMENTS

CSA. A practical break of CSA would therefore affect all broadcasters which would have to exchange the hardware used to decrypt the transport streams. [3]

1.5 Set-Top-Box requirements

A Set-Top-Box (STB) is a device that converts an external signal into viewable video on a connected television. The signal source is usually a satellite dish (DVB-S), an ethernet cable (IPTV), UHF/VHF antenna (DVB-T) or simply a coaxial cable (cable television, DVB-C). There are also hybrid STB’s which allow reception from multiple sources. Such as a combination of DVB-S, DVB-T and DVB-C reception in the same device. A traditional STB have in general only one additional feature other than providing the connected television with audio and video; they are also capable of presenting Electronic Program Guide (EPG) in a more or less understandable way. The EPG information is normally distributed via specific EPG-packets in the DVB Transport Stream. STB’s can be equipped with a Common Interface (DVB-CI[7]) this interface is designed so that it is possible for service providers to use custom encryption algorithms for their transmissions. Details of DVB decryption mechanisms are beyond the scope of this thesis as the focus only lies on displaying unencrypted channels, so called Free-To-Air (FTA) channels.

15 1.5. SET-TOP-BOX REQUIREMENTS CHAPTER 1. BACKGROUND

16 Chapter 2

Problem description

The main question for this thesis is if the PandaBoard, and the OMAP4 in particular, is powerful enough in terms of performance to be used as a Set-Top-Box, (STB). For this thesis the external signal is the European digital terrestrial television standard DVB-T and its successor DVB-T2. The targeted operating system is Android, developed and maintained by Google. Android is an increasingly popular operating system found primarily in smartphones and tablets. The ARM processors main advantage is its low power consumption, this is the reason why it is found in mobile devices and embedded systems. The weakness of the ARM architecture has until recently been performance. Is it feasible to construct a low power ARM based STB for multimedia consumption? In order to reach the thesis goals the following tasks were considered for the implemen- tation phase:

1. Setup the devlopment enviroment

2. Boot Pandroid on the PandaBoard using an SDCard

3. Add DVB support to the Pandroid Linux kernel

4. Create an application that can tune to a channel

5. Extract the transport stream from the DVB device

6. Split the transport stream into a Video and Audio stream

7. Decode the video and audio streams

8. Synchronized audio and video playback

2.1 Linux kernel DVB-stack

In order to be able to use a USB DVB-T dongle in Android the DVB-stack has to be activated in the Android Linux kernel. Reconfiguration and recompilation of the kernel will be needed. Compilation of Linux kernel drivers for the DVB-T and/or DVB-T2 USB dongle is also essential. When the DVB-stack has been enabled in the kernel, software will be needed in order to interface with the Linux kernel DVB-API. The DVB-API is used for operations such as tuning and scanning channels. The Linux DVB-API is developed, maintained and documented by the LinuxTv.org[17] Project.

17 2.2. DVB-T2 CHAPTER 2. PROBLEM DESCRIPTION

2.2 DVB-T2

One of the implementation goals is to be able to receive and display the new DVB-T2 transmissions that were introduced in Sweden on November 2010 by Teracom. The problem is that since the standard is new and few countries have adopted it, the available consumer hardware supporting DVB-T2 is limited. In fact during the time this thesis was written, only one DVB-T2 compatible USB device existed; the PCTV Systems nanoStick T2 290e. Device drivers is only officially supported for Windows operating system, this means no drivers for the Linux kernel exist. The DVB-T2 goal is directly dependant on Linux kernel drivers for the nanoStick. If no drivers would emerge during the time reserved for the implementation phase, a DVB-T USB device with drivers already is in the Linux kernel will be used instead. It turned out that linux drivers for the nanoStick device emerged in April 2011, this meant that it would be possible to compile these drivers into the Pandroid linux kernel before the thesis implementation phase ended. Once we are able to get the datastream from a DVB-T2 or a DVB-T signal, the problem that remains is to extract the different packages contained in the transport stream, this process is called demultiplexing.

2.3 Decoding Audio and Video

The audio and video packets in the Transport Stream are encoded depending on the DVB standard used. In Sweden the video stream is encoded using MPEG-2 and the audio stream is encoded with MPEG Audio Layer II (MP2) or Dolby Digital (AC-3), when sent using DVB-T. The video stream sent using DVB-T2 is encoded with the newer MPEG-4 AVC, also known as H.264, standard and audio is encoded with the MPEG-4 HE-AAC standard. Unfortunately Texas Instruments was not able to enable support for the Ducati Sub System in Pandroid when the implementation was conducted. This meant that the dedicated hardware on the PandaBoard that are supposed to be used when decoding video streams like the two mentioned above were not available. How can video decoding be made possible under these circumstances? Due to lack of support by Texas Instruments, audio playback is not enabled in Pandroid during the implementation phase. If this issue is not solved by Texas Instruments it will not be possible to playback the audio contained in the Transport Stream.

18 Chapter 3

Methods

The work on this thesis was performed at the Syntronic Software Innovation office in Kista, Stockholm. During the first 3-4 weeks work was focused on finding information about the PandaBoard and setting up the development enviroment. The development enviroment included a desktop computer with Ubuntu 10.10 (Maverick Meerkat) installed. Ubuntu was a natural operating system choice, since the vast majority of the documentation for the PandaBoard assumed Ubuntu as the host system. It was decided that a version control system would be needed, not only for the implementation code, but also for keeping track of changes in this written report. GIT was chosen as the primary system because it has gained much popularity among the open source community. The open mobile software development platform; PandaBoard was released in October 2010. The primary source of information about it can be found at the PandaBoard.org web page. This web page is also intended to work as a community page and to connect software developers with each other. Information was also found on their mailing list and IRC chat room. One of the main goals for this thesis was to receive and decode the new DVB-T2 transmissions, however Linux drivers for the PCTV nanoStick 290e usb dongle did not emerge until 8 April 2011. News about the development progress for the nanoStick Linux drivers could be found at a blog by a man called Steve Kerrison[12]. Once it was confirmed that the nanoStick actually was working in Linux one of these devices was ordered by Syntronic and work was made on patching the Linux kernel in order to get it working properly on the existing system.

19 CHAPTER 3. METHODS

20 Chapter 4

Implementation

4.1 System peripherals

The most significant piece of external system hardware is of course the nanoStick DVB-T2 usb dongle. However in order to interact with the system in real time, such as changing channel, an usb mouse is used. An usb cable connecting the PandaBoard with the host system is convenient for development and debugging purposes. A screen must of course be attached to the system and the system itself is running from and stored on a memory stick inserted into the PandaBoad. An overview of the system peripherals can be seen in figure 4.1.

Figure 4.1: Peripherals connected to the PandaBoard

4.1.1 Usb devices The preferred way to test programs for Android is by transferring and debugging the application via the Android Debug Bridge, or ADB in short. An ADB-connection is

21 4.1. SYSTEM PERIPHERALS CHAPTER 4. IMPLEMENTATION

established via an usb cable connected to an usb-port on the host system and a mini-usb connector on the PandaBoard. There is also a RS-232 port on the PandaBoard that can be used for debugging, a serial-to-usb converter was used for this project however only to a limited extent, mostly because ADB was more convenient. The only HID-device, (Human Interface Device), used in this implementation is an usb-mouse. A standard Android system do does not include mouse support since the conventional way of interaction with an Android device is by using a touch screen interface. The Pandroid distribution is patched so that it supports a mouse. Instructions on how to patch Pandroid is documented on the Pandroid homepage. In order for the system to receive DVB-T and DVB-T2 signals a PCTV nanoStick T2 290e usb dongle is connected to an usb port. This device is the first consumer product on the market to support the DVB-T2 standard and as of April 2011, the first DVB-T2 device with Linux kernel driver support. The nanoStick was shipped with a small antenna, however reception is poor with this kind of small antennas. For the implementation a different antenna was used instead.

The PCTV nanoStick T2 290e The 1st November 2010 Swedish terrestrial broadcast service company, Teracom, started sending with the new DVB-T2 standard in Sweden. A few weeks later, in November 25th 2010, the PCTV nanoStick T2 290e was released. It was promoted with slogans such as ”World’s first DVB-T2 receiver for PC”, even today, a half year later, few consumer DVB-T2 devices for PCs are out on the market.

Figure 4.2: The PCTV nanoStick T2 290e

The nanoStick’s main components is a NXP TDA18271HDC2 tuner, an Empia em28174 controller chip and a Sony CXD2820R demodulator. The tuner and the controller chip drivers was implemented previously in the kernel, since they are used in other DVB-T devices. The problem was the Sony demodulator. New code needed to be written for it to work in Linux. Pictures of the nanoStick PCB can be found in appendix A. Below is an excerpt from an entry from Steve’s blog when an important milestone was reached. It was early April when the last missing piece of the driver puzzle was laid by the Finnish Linux kernel hacker Antti Palosaari.

8th April 2011 - And so it begins Another LinuxTV developer enters the fray - Antti Palosaari - bringing with him a CXD2820R module! There are still a lot of details to work out when it

22 CHAPTER 4. IMPLEMENTATION 4.1. SYSTEM PERIPHERALS

comes to T2 support, but this should give us a working device. I cannot test until some time next week, but the keen hackers amongst you might want to head over to Antti’s git repository to check it out. Of course, those of you already on the linux-media mailing list will already know this. For those who don’t, here’s the message.

4.1.2 HDMI A LCD screen is connected using a HDMI to DVI adaptor cable. The PandaBoard has two HDMI-connector slots, one of them hosting a HDMI-signal and the other a DVI-signal. The HDMI-connector with the DVI-signal was not enabled in Pandroid at the time the implementation was performed. There where no problems using the HDMI-signal to a LCD display with a DVI-port.

4.1.3 The SD-Card The PandaBoard default boot device is the Secure Digital card slot. A Secure Digital flash memory card, SD-card, is needed to be formatted in a proper configuration and have the necessary boot-images in order to boot the Linux Kernel and Pandroid. The SD-card is partitioned according to instructions on the PandaBoard homepage. Basically the SD-card is partitioned with one boot partition and one rootfs partition. The boot partition contains three important binary images. The X-Loader, U-Boot and uImage. The Android file system resides on the rootfs partition.

Figure 4.3: Content on the SD-card

X-loader, is a small first stage boot loader originating from the u-boot source code and is loaded into the internal static ram. The internal static ram is very small, this is why the

23 4.2. SYSTEM SOFTWARE CHAPTER 4. IMPLEMENTATION

x-loader is stripped down to a minimum and is used to initialize memory and the peripheral devices needed to access and load the second stage loader, U-Boot, into main memory. Universal Bootloader, or Das U-Boot, is a widely used open source boot loader for embedded devices. It’s name comes from an abbreviation of Das Unterseeboot, which is German for ”the submarine”. The U-Boots primary function is to load the Linux kernel uImage. The Linux kernel 2.6.35 is configured and patched for use with Android according to instructions on the Pandroid web page. In order to activate the DVB-stack custom configurations has to be made in the Linux compilation configuration. When using a DVB-T usb dongle with drivers included in Linux 2.6.35, it is only a matter of using the kernel menuconfig-tool, and enabling DVB-T support in the kernel. However, it is not this simple when trying to use a brand new DVB-T2 usb dongle with Linux. First of all the kernel driver source code was released in April 2011, and this highly experimental driver is at the time of writing this report, not officially included in the Linux kernel yet. There is however a script provided by linuxtv.org that facilitates compilation of experimental multimedia drivers for the Linux kernel. The LinuxTV project, hosted at linuxtv.org, develops and maintains the DVB driver subsystem which is included in the Linux 2.6.x kernel.

4.2 System software

The main system covers a wide spectrum of technical areas. Some of these areas include programming fragment shaders for OpenGL ES 2.0 hardware, utilizing 3rd party libraries (such as FFMpeg), patching, configuring and compiling of the Linux kernel to name a few. The main challenge has been to find a solution for each area and connect them with each other into a usable system.

4.2.1 DVB Utils Library In order to control a DVB device in Linux the DVB-stack must be included in the kernel when compiling. The DVB-stack is developed and maintained by the LinuxTV project. The code in the DVB Utils library is partly based on example code from the dvb-apps[17] tree, provided by LinuxTV. When a DVB device has locked to a MUX a transport stream can be extracted from the device. The transport stream is part of the MPEG-2 specification.

4.2.2 FFMpeg Library The FFMpeg project is an open source and cross-platform project that host several libraries for multimedia processing. Two of its most widely used libraries are libavcodec and libavformat. Many media formats don’t specify which codecs should be used to encode audio and video data. These media formats are containers that define how audio and video streams should be combined into a single stream. In the case of DVB the container format is called Transport Stream and is specified in the MPEG-2 standard. The libavformat library deals with parsing the transport stream and separating, or demuxing, the streams contained in it. The libavcodec library deals with decoding the audio and video streams. In order to use FFMpeg in the thesis implementation, the following features has been activated when configuring and compiling FFMpeg:

Video decoders: MPEG-2 and MPEG-4 AVC (aka h.264) •

24 CHAPTER 4. IMPLEMENTATION 4.2. SYSTEM SOFTWARE

Figure 4.4: Simplified system structure

Audio decoders: MPEG-2 Audio Layer II (mp2) and MPEG-1 Layer 3 (mp3) • Demuxer: MPEG-2 Transport Stream • NEON optimizations • Once the DVB device is locked to a mux, the Transport Stream can be read from the device. Each mux usually contains several standard definition television channels and it is needed to specify what channel we are interested in reading data from. When demuxing is done it is possible to extract the separate media elements, such as the video and the audio stream. For DVB-T the video stream is encoded with the MPEG-2 video format and for DVB-T2 the video format is MPEG-4 AVC (also known as h.264). The audio used in DVB broadcasting is usually encoded with MPEG-2 Audio Layer II however audio suppport is not included in the thesis implementation.

4.2.3 Color space conversion When the video stream is decoded the resulting product is video frames in a format called YUV420P.This format is used for video compression and human perception reasons. The Y in YUV is the frame luminance component. It measures 720x576 pixels and represents the frame brightness with values 0-255. The U and the V is two chrominance components that only is one fourth the size of the Y component. The reason for this is that the human eye is much more sensitive to brightness compared to color information, it is therefore possible to reduce the resolution of the color components with no loss in perceived quality. The numbers 420 (sometimes written 4:2:0) in YUV420P represents the chroma sub-sampling

25 4.2. SYSTEM SOFTWARE CHAPTER 4. IMPLEMENTATION

rates for Y, U and V component. This basically means that for each 4x4 pixel square area with Y components there are two 1x1 pixel chrominance componens, U and V. This relationship is illustrated in Figure 4.5. Finally the letter P needs an explanation, it stands for planar and means that the three components Y, U and V are grouped in three separate planes as Figure 4.5 shows.

Figure 4.5: YUV420P SDTV frame layout

In order to render an YUV420P frame to a screen it has to be converted to RGB format. This conversion process needs to be efficient and fast. At an early stage of the implementation process the FFMpeg project was evaluated and some experiments was conducted using a library called libswscale, part of FFMpeg. Libswscale has some functions for converting YUV frames to RGB, however results from the experiments showed that it was not fast enough for real-time video rendering on the PandaBoard. At that point a distinct problem was identified; how should color space conversion be done? The following conclusions was made:

No support yet for the dedicated Ducati subsystem in Pandroid • libswscale in FFMpeg was not fast enough • It was noted that some work had been done in adding NEON optimizations for the libswscale color space conversion routines[2]. However the code seemed highly experimental and was quickly dismissed as a viable solution. Instead focus went to the hardware that was supported in Pandroid; OpenGL ES 2.0. It was chosen to do the color space conversion since it has a fragment shader that can be programmed, demonstrated in Listings 4.1, to perform arbitrary operations on individual pixels in a texture - very fast. Much effort went into researching how this could be implemented in the fragment shader using GLSL, a C-like OpenGL programming language. The actual mathematical conversion is done according to recommendations specified by the International Telecommunications Union. There are differences in the conversion details depending on if the transmission is SDTV or HDTV. ITU-R BT.601[8] is used for SDTV and ITU-R BT.1847[9] is used for HDTV[13]. Figure 4.6 illustrate the conversion

26 CHAPTER 4. IMPLEMENTATION 4.2. SYSTEM SOFTWARE

matrix corresponding to ITU-R BT.601, according to [10] pages 19-20. In order to minimize implementation code complexity, a decision was made to treat HDTV as a SDTV transmission as far as color space conversion is concerned.

R 1.164 0 1.596 Y 16 601 − G = 1.164 0.391 0.813 U 128  601  − −   −  B 1.164 2.018 0 V 128 601 −       Figure 4.6: Color space conversion matrix according to ITU-R BT.601 used for SDTV

The YUV values in the above matrix are in the range of a byte i.e. 0-255. The pixel values in OpenGL ranges between 0.0 and 1.0 so the YUV values need to be divided with 256 as seen in Figure 4.7.

Y 16 Y 0.0625 − 256 − U 128 = U 0.5  − 256   −  V 128 V 0.5 − 256 −     Figure 4.7: Conversion from samplesize 0-256 to 0.0-1.0

1 Y = texture2D(Ytex , v texCoord).r 0.0625; − 2 U = texture2D(Utex , v texCoord).r 0.5; − 3 V = texture2D(Vtex , v texCoord).r 0.5; − 4 5 mat3 m = mat3(1 .164, 1.164, 1. 1 6 4 , 6 0 . 0 , 0.392, 2.017, − 7 1 . 5 9 6 , 0.813, 0.0); − 8 vec3 rgb = m vec3(Y, U, V); ∗ 9 g l FragColor = vec4 ( rgb , 1 . 0 ) ; Listing 4.1: GLSL code from the implementation of YUV to RGB color space conversion in the OpenGL ES 2.0 fragment shader

4.2.4 OpenGL ES 2.0 The lack of support for the Ducati sub system, used for hardware accelerated video decoding, lead to the use of OpenGL ES 2.0 fragment shader. Inspiration for this kind of solution came from a number of open source projects. One project called Dead Penguin[6]. The Dead Penguin project has also, apart from having color space conversion in the fragment shader, deinterlacing in the fragment shader. A very interesting feature that could be a possible future improvement for this project. Another example is the perhaps more influential Chromium project[15]. They have also done an implementation of color space conversion in an OpenGL fragment shader. Shortly after the decision was made to leverage on the OpenGL ES 2.0 acceleration hardware, the idea came to life that it would be interesting to present the decoded video stream as a cube that could be rotated. Each face of the cube rendering video except the top and bottom face. The top was given a static texture with the Syntronic logo and the

27 4.3. RESULTS AND EVALUATION CHAPTER 4. IMPLEMENTATION

bottom face the Uppsala University logo. The cube can be rotated using an usb mouse connected to the PandaBoard.

4.3 Results and evaluation

The research reported in this thesis has investigated if and how the PandaBoard mobile software development platform can be used as a STB. A prototype system has been developed demonstrating reception and displaying of digital terrestrial television. The prototype system is based on Android and uses an external usb device for DVB-T and DVB-T2 reception. The system displays unencrypted channels sent over the Swedish digital terrestrial television network. Dedicated PandaBoard hardware for accelerated video decoding was not used due to TI lack of driver support for Pandroid. Another way of rendering the video stream was developed using the FFMPEG library. It was discovered that the OpenGL ES 2.0 fragment shader could be used for accelerated color space conversion, this technique is implemented in the prototype system.

Figure 4.8: Prototype system

4.3.1 System performance The system performance was evaluated by measuring the amount of frames per seconds (FPS) rendered on to the screen. For this evaluation the channels SVT1 (SD) and SVT1 HD was compared to give an indication on the system video rendering performance. No deinterlacing or audio decoding was done when measuring the FPS.

28 CHAPTER 4. IMPLEMENTATION 4.3. RESULTS AND EVALUATION

Tests showed that when viewing SVT1 (SD) an average of 70 FPS was reached. 70 FPS is good since the video is only encoded with 25 FPS i.e. 280% more fps than required. The extra processing power could be used for deinterlacing and audio decoding. When SVT1 HD was viewed an average of 14,8 FPS was measured. This value is 59% less than the video stream at 25 FPS. All frames are not displayed resulting in a poor viewing experience.

4.3.2 Power consumption The system power consumption was measured using an ammeter . In an idle state the system consumes 3,15 Watts without the nanoStick DVB-T2 USB device connected. After connecting it, the power consumption increased to 4 Watts revealing that the nanoStick itself consume 0,5 Watts in an idle state. When a SDTV channel is tuned and rendered on a screen 5,75 Watts is consumed and the CPU load goes up to an average of 37%. When viewing a HDTV channel 7 Watts is used by the system and the CPU goes up to 95%-100%.

29 4.3. RESULTS AND EVALUATION CHAPTER 4. IMPLEMENTATION

30 Chapter 5

Conclusions

5.1 Unsolved problems

5.1.1 Audio playback One of the issues has been audio playback in Pandroid. The FFMPEG library, used in this project, includes codecs for the audio streams used in DVB broadcasts. This fuctionallity was however never used since there was no ability to output the audio from the PandaBoard hardware. Audio support was eventually added to Pandroid, unfortunately this was done after the implementation phase of this thesis. Below is an excerpt from the changelog on the Pandroid webpage.

6/03/11 * Audio is working to some degree.

5.1.2 Accelerated video playback The Pandaboard has great hardware for video playback, however it is not worth much when it is not possible to use it. TI did not manage to support accelerated video playback for Pandroid during the implementation phase of this thesis. A different solution was found in order to render video. The FFMPEG library supports the needed video codecs used in the DVB-T and DVB-T2 standards. It was configured to take advantage of the advanced SIMD (Single Instruction Multiple Data), perhaps more known as NEON, support in the ARM Cortex-A9 CPU. NEON provides standardized acceleration for media and signal processing applications. The PandaBoard performed well using the FFMPEG library. SDTV signals recieved using DVB-T was rendered to a screen in satisfying way. HDTV video received using DVB-T2 was however too demanding for the system and video was not rendered fast enough, resulting in dropped and corrupt frames.

31 5.1. UNSOLVED PROBLEMS CHAPTER 5. CONCLUSIONS

32 Chapter 6

Future work

Unfortunately hardware accelerated video playback was not possible to implement in the prototype system for this thesis. Further evaluation of performance of this dedicated hardware for video processing on the PandaBoard would be a very interesting development direction to pursue in the future. When/if TI manages to enable such support in future versions of Pandroid. Audio support in a STB is of course essential and it should be included in a future system, it seems that TI has managed to enable audio in Pandroid, however this was after the prototype system for this thesis was completed. Other things that are missing in the prototype,that are needed in a STB, is EPG support (Electronic Program Guide). EPG data is sent in the MPEG2 Transport Stream and should be relatively easy to implement from a technical standpoint. It is however non-trivial to present this EPG information to the consumer in a satisfactory manner. The information should be easy to read and navigation through the data should be very intuitive. A remote control is the traditional and probably most common device for system interaction with a STB. There is an open source project called LIRC (Linux Infrared Remote Control) that could be useful in a future implementation. A remote control may not, however, be the best method. It would be interesting to do further research into what interaction device is most suitable for a future STB-device.

33 CHAPTER 6. FUTURE WORK

34 Bibliography

[1] Wikipedia, November 2010. http://en.wikipedia.org/wiki/File:Digital_ broadcast_standards.svg.

[2] Allmann, J. FFMpeg fork with NEON optimizations for YUV to RGB color space conversion. https://github.com/j0sh/ffmpeg/tree/swscale-neon/libswscale/ arm.

[3] Benoit, H. Digital television: MPEG-1, MPEG-2 and principles of the DVB system, 2. ed. ed. Focal press/Elsevier, Burlington, MA, 2004.

[4] Brown, A. Sweden: The digital threat to cultural sovereignty. In Digital Terrestrial Television in Europe, A. Brown and R. G. Picard, Eds. Lawrence Erlbaum Associates Publishers, Mahwah, New Jersy, 2005, pp. 203–221.

[5] Digital TV Commission. Digital TV switchover : 2005.09.19 - 2007.10.15 : The Digital TV Commission’s final report, KU 2004:04., http://www.sweden.gov.se/ sb/d/574/a/113008 visited February 2011 ed. Swedish Government Official Report, 2008:35. Fritze, Stockholm, 2008.

[6] Essen, L. Dead Penguin - the set top box. http://www.deadpenguin.tv/.

[7] European Telecommunications Standards Institute (ETSI). EN 50221: ”Common Interface Specification for Conditional Access and other Digital Video Broad- casting Decoder Applications”, February 1997. http://www.dvb.org/technology/ standards/En50221.V1.pdf.

[8] International Telecommunications Union. Recommendation ITU-R BT.601, Encoding Parameters of Digital Television for Studios. Geneva, 1992.

[9] International Telecommunications Union. Recommendation ITU-R BT.1847, 1280 x 720, 16:9 progressively-captured image format for production and international programme exchange in the 50 Hz environment. Geneva, 2009.

[10] Jack, K. Video Demystified: A Handbook for the Digital Engineer, 4th Edition, 4th ed. Newnes, Newton, MA, USA, 2005.

[11] Jeding, L. From Massmedia to Multimedia – Digitalisation of Swedish Television, http://www.sweden.gov.se/sb/d/8430/a/737 visited February 2011 ed. Swedish Government Official Report, 1996:25. Fritze, Stockholm, 1996.

[12] Kerrison, S. PCTV nanoStick T2 290e for Linux. http://stevekerrison.com/ 290e/index.html.

35 BIBLIOGRAPHY BIBLIOGRAPHY

[13] Sveriges Television. HDTV - Expertinformation. http://svt.se/2.152197/1. 2408782/expert.

[14] Teracom. Konsumenterna efterfr˚agarmobil tv. http://www.dvb-h.org/PDF/ Stockholm-Trial-Release.pdf, March 2007.

[15] The Chromium Authors. The Chromium projects. http://src.chromium.org/ svn/trunk/src/media/tools/player_x11/gles_video_renderer.cc.

[16] The DVB Project Office. Introduction to the dvb project - creating global standards for digital television. Fact Sheet http://www.dvb.org/technology/fact_ sheets/DVB-Project_Factsheet.pdf. visited February 2011.

[17] The LinuxTV project. The LinuxTV project develops and maintains the DVB driver subsystem which is included in the Linux 2.6.x kernel. http://linuxtv.org/repo/.

[18] Zirn, T. Teracom kan tvingas offra mobil-tv-n¨at. http://computersweden.idg.se/ 2.2683/1.132101, November 2007.

36 Appendix A

PCTV nanoStick T2 290e PCB

Figure A.1: The top side contains a NXP TDA18271HDC2 tuner. Photograph courtesy of Steve Kerrison

Figure A.2: The bottom side contains an Empia em28174 controller chip and a Sony CXD2820R demodulator. Photograph courtesy of Steve Kerrison

37