Imaging Application Platform Using MB86S62 Application Processor

 Atsuhiro Suga  Hidetoshi Nakahara  Masato Hyodo  Shinichi Takimoto

Imaging solutions using application processors means providing customers with an application platform that includes software and evaluation boards based on Fujitsu’s extensive image- processing technologies. Customers who manufacture assembled products (set makers) can use this platform to develop and market multi-function products in a shorter period of time. In combination with Fujitsu’s Milbeaut image processor, this platform provides a high-perfor- mance image processing system that can provide customers with a competitive edge over other makers. This paper presents a specific example of this imaging application platform using Fujitsu’s MB86S62 application processor.

1. Introduction the development period and increasing the develop- Today’s smartphones and tablets are evolving at a ment costs compared to existing technologies. As a dramatic pace. Being capable of functions and applica- result, the development of LSI circuits using advanced tions that were inconceivable just a few years ago, they technologies increases risk and becomes a major fac- are now becoming an integral part of business and tor in reducing the customer’s return on investment everyday life. This epic development can be attributed (ROI) when developing individual products. In short, it to the use of application processors on the terminal makes new development projects all the more difficult. side to maintain sufficient processing speeds and to This state of affairs suggests a new business op- stress-free, routine Internet access made possible by portunity in providing new solutions for easing the the creation of a communication infrastructure cen- customer’s load (development costs, software devel- tered around Long Term Evolution (LTE).1) As a result, opment) in relation to 40-nm and beyond process a wide variety of compelling services are coming to be technologies. provided by interlinking terminal-side applications with Against this background, semiconductor manu- the cloud. facturers who develop application processors are, of Looking forward, we can expect advanced user course, targeting the smartphone and tablet markets, interfaces developed for smartphones and tablets to but they are also indicating a desire to participate be applied to a variety of products, which suggests that in the on-vehicle and digital-consumer-electronics image processing using application processors will be- markets. In contrast, customers who are makers of as- come increasingly important. sembled products (set makers) are beginning to move At the same time, a company would need cutting- in the direction of system integration using existing edge process technologies to fabricate such application application processors to reduce development costs processors for achieving novel products in a demanding and risks instead of developing new, proprietary LSI market and to enable it to differentiate itself from its circuits. This trend is providing Fujitsu—which excels in competitors. However, the development of LSI circuits the application-specific integrated circuit (ASIC) busi- using 40-nm and beyond process technologies means ness—an impetus for revolutionizing the way it supports an explosive increase in the difficulty of layout and set makers. the amount of labor for testing, thereby extending In particular, Fujitsu is focusing on applications

FUJITSU Sci. Tech. J., Vol. 49, No. 1, pp. 49–58 (January 2013) 49 A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

that can exploit the company’s extensive expertise over many years and its image processing technology in image processing technology and is providing its centered about its Milbeaut image processor. In addi- customers with an application platform that includes tion to excelling in power consumption, performance, software and evaluation boards. In this way, Fujitsu has and device area (and therefore lowering costs) by in- achieved a solution for providing application-processor- tegrating software and hardware based on advanced based development while enabling the customer to CPU/GPU devices, this platform provides a highly com- incorporate its proprietary logic to achieve original petitive multimedia framework featuring a sequence of functions in much the same way as can be done with image processing functions including sensing, analysis conventional ASICs. Furthermore, by combining this and processing, compression and decompression, and platform with Fujitsu’s Milbeaut image processor, cus- display and output. The platform provides customers tomers can use this application platform as an imaging with value by reducing costs and accelerating delivery platform in a manner different from what can be of- through a platform geared to imaging-related appli- fered by other application processor vendors. cation-specific standard products (ASSPs), by boosting This paper presents a specific example of using existing business and facilitating the creation of high- this imaging application platform in conjunction with value-added products, and by contributing to the Fujitsu’s MB86S62 application processor. creation of new businesses. The configuration of this platform is shown in 2. Imaging application platform using Figure 2. It consists of a system-on-a-chip (SoC) based MB86S62 on application processors, Linux2) and Android3) board The concept of an imaging application platform is support packages (BSPs), and a multimedia framework shown in Figure 1. comprised of Fujitsu image processing functions. This The idea here is to provide a user-friendly plat- platform makes for easy system development since it form that exploits Fujitsu’s ASIC technology developed allows the customer to create original applications on

Imaging application platform

Reduces costs and Provides competitive Provides APIs for easily accelerates delivery multimedia framework in an open implementing applications by using a platform geared to environment on target system imaging-related ASSPs Linux OS

Boosts existing business Has high driver-level efficiency and facilitates creation of high-value-added SoC for imaging application platform products CPU

GPU DSP Accelerator Connections Contributes to creation of new businesses High-efficiency architecture (power, performance, area, ease of use)

Fujitsu imaging technology

Analysis/ Compression/ Sensing processing decompression Display/output

Multimedia framework (comprehensive environment including video/audio media) GPU: Graphics processing unit DSP: Digital signal processor

Figure 1 Concept of imaging application platform. Figure 1 50 Concept of imaging application platform. FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

Android BSP

Set-maker development section Options Application Test application

Google supplied Expanded Application framework application framework ARM supplied OpenGL ES Expanded Standard library API library library

Linux BSP Hardware architecture

Multimedia framework FSL ISP section Imaging Computing subsystem High-bandwidth USB CSI SATA Audio Display Standard ISP Other subsystem connectivity HAL HAL HALs HAL HALs GMII/Ether CPU GPU PCIe ISP cluster cluster Cache coherent interconnect System control Clock Codec Reset I2C Generic kernel DSI Post DRAM Power-supply proc. interface UART control HDMI Platform specific (V4) Display GPIO Low-bandwidth UART connectivity Audio I2C High-bandwidth interconnect with QoS with interconnect High-bandwidth SPI Device drivers FSL ISP I2S section Standard Display GPU Other driver driver driver drivers FSL: Fujitsu Semiconductor ISP: Inter-chip connection DRAM HAL: Hardware abstraction layer QoS: Quality of service Figure 2 MB86S6Figure platform 2 confi guration. MB86S6 platform configuration. top of these Linux and Android BSPs. and a secondary cache of 256 KB. The following subsections describe the hardware 3) Powerful and extensive multimedia functions and software confi guration of this platform and associ- The MB86S62 features two camera input ated features. interfaces. The main interface supports up to 8 megapixels (Mpixels) using a MIPI (Mobile Industry 2.1 MB86S62 Processor Interface)4) camera serial interface (CSI) or an A block diagram of the MB86S62 application pro- 8-bit parallel interface. The sub-interface supports up cessor is shown in Figure 3, and its main specifi cations to 3 Mpixels using an 8-bit parallel interface. The video are listed in Table 1. In developing this platform, much codec supports a variety of video standards including attention was paid to achieving greatly reduced power MPEG-2,5) H.264,6) and VC-1.7) It supports, in particular, consumption and high-performance image processing. Full HD 1080p at 30 fps encoding/decoding and 720p The hardware measures taken to achieve these goals dual encoding. The audio codec supports various audio and key MB86S62 features are described below. standards such as MP38) and AAC and various interfaces 1) Low-power process including I2S9) and SPDIF.10) The use of an advanced 45-nm, low-power pro- 4) 3D graphics engine cess in this application processor has reduced power The MB86S62 integrates a 2D/3D graphics hard- consumption and lowered costs. ware accelerator supporting OpenVG1.111) and OpenGL 2) ARM Cortex™-A8 processor ES2.0 and featuring a dual core having a processing The MB86S62 incorporates an ARM core, a ability of 30 MPoly/s and 400 Mpixel/s. well-known embedded processor, which achieves a 5) Display functions supporting three-screen display maximum operating frequency of 1 GHz. This core in- and XGA tegrates a primary instruction/data (I/D) cache of 32 KB Featuring a 24-bit RGB interface and a MIPI display

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 51 A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

Memory controller mDDR, DDR, DDR2, SD, SDIO, SDHC, NAND, NOR, OneNAND MMC, SATAll

MPEG-2 TS MPEG2 TS Rx I/F X2 Memory controller Storage controller

Smart card USB 2.0 Full HD video Post processor USB Smart card I/F X2 Host & OTG

XGA LCD Display unit composite Camera CortexTM-A8 MIPI Parallel, MIPI interface (1 GHz) HDMI Tx HDMI 1.3a

HIP, External System Audio UART, SPI, peripheral Graphics 2D/3D Security control interface I2C, GPIO interface unit

I2S, SPDIF

Figure 3Figure 3 MB86S62 block diagram. MB86S62 block diagram.

Table 1 6) Low-power consumption through power-supply MB86S62 main specifications. management Item Specifications Mounted on the MB86S62 is a system control unit CPU ARM Cortex™-A8, max. 1 GHz designed to lower power consumption by changing the 1080p @30 fps encoding: power mode (normal, idle, stop, sleep, deep sleep) as MPEG-4, H.264 (720p dual encoding) needed, controlling the operating current by clock gat- Video 1080p @30 fps decoding: MPEG-2/4, H.264, VC-1, RV8.9.10, ing, and performing power gating on individual macro H.263 mode support, Divx streaming functions (leakage current can also be controlled). Graphics OpenGL2.0/OpenVG1.1 (30 MPoly/s, 400 Mpixel/s) The MB86S62 application processor is mounted XGA LCD I/F and dual LCD support Display in a 19-mm-square fine-pitch ball grid array (FBGA) (24-bit RGB or MIPI DSI/3D deinterlacer) package supporting external dynamic random ac- Camera Dual cameras (8 Mpixels and 3 Mpixels) cess memory (DRAM). We are now in the process of USB Host × 1, OTG × 1 PHY built-in MPEG-2 TS RX × 2 and SPI × 3, SmartCard I/F × 2 developing a system in package (SiP) version of this ap- Interfaces SD/MMC/SDIO/SDHC I/F plication processor. We will use chip stacking as shown SATAII/HDMI in Figure 4 to save space on the mounting board and to 19 mm × 19 mm Package 620-pin FBGA (0.65-mm pitch) further reduce power consumption.

2.2 Evaluation environment serial interface (DSI), the MB86S62 supports an XGA-size 2.2.1 Overview of developer tool kit using (1024 × 768) LCD interface and dual LCD display. It can MB86S62 also support a three-screen display by combining the The developer tool kit (DTK) is equipped with dual-LCD function with a High-Definition Multimedia MB86S62 peripheral devices and a variety of interfaces, Interface (HDMI)12) or a TV (NTSC or PAL) interface. making it easy to achieve desired functions.

52 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

Evaluation boards for development purposes development directly related to product specifi cations using the MB86S62 application processor are shown without having to be concerned for the most part about in Figure 5. Specifi cally, the DTK for developing soft- lower-level design. ware such as driver middleware is shown in Figure 5(a), and a tablet for developing applications is shown in 2.2.2 DTK basic functions Figure 5(b). The tablet can be connected to an exten- • LCD: 800 × 480 pixels sion board to construct a debug environment nearly • Double data rate (DDR) RAM capacity: 512 MB equivalent to that of the DTK. (system), 256 MB (media) The DTK includes BSPs for both Linux and Android • NAND flash memory: 8 GB (multi-level cell, MLC) systems given their popularity in the market. These • USB13) on-the-go (OTG) converter cable BSPs enable the customer to concentrate on upper-level • Non-crossing serial cable

Space saving Size: 12 mm square Height: 1.5 mm or less

Stacks 4 memory chips

Flip chip Achieves low power by reducing wiring load

Figure 4 ExampleFigure 4SiP confi guration. Example SiP configuration.

Top Bottom

(a) DTK for software development (b) Tablet for application development

Figure 5 EvaluationFigure 5 boards for development using MB86S52. Evaluation boards for development using MB86S52. FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 53 A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

• Stereo speakers 6) Switch functions • 5-V, 2-A adapter Switches for performing resets and selecting op- • 3.7-V single-cell battery (max: 4.2 V) eration modes are provided as follows: • Power switch 2.2.3 DTK function details • Reset switch 1) Power-supply functions • CPU reset switch A main power-supply switch and a light-emitting • Boot-mode setting switch diode (LED) power indicator are included. Two types of power-supply feeds are provided: 2.2.4 Software configuration • 5-V AC This platform envisions applications in the form • 3.7-V DC (single-cell battery) of mobile terminals (including tablets) and other 2) Evaluation functions digital devices having equivalent functions. It consid- The customer can perform product evaluation by ers Android and Linux to be the basic system OSes and connecting the main DTK board to a personal computer frameworks. via a debug board using any of the following means: The basic configuration of the Android system is • Ethernet shown in Figure 6. It features the following five layers. • JTAG 1) Application • RS-232C Android applications run in this layer. • USB 2) Application framework 3) External device functions This layer provides various services for Android The following functions are provided for connect- applications. ing to external devices: 3) Library/Android runtime • NAND flash memory This layer consists of the following basic library • Universal asynchronous receiver/transmitter groups. (UART1) • SQLite • Secure Digital (SD)14) Ch0/Ch1 • Libc • USB host: USB host connector • WebKit • Camera interface 0: parallel camera board • OpenGL ES (OpenGL for Embedded Systems) • Camera interface 1: parallel camera board • Core library • MIPI slave: MIPI camera board • Dalvik virtual machine (VM) • Micro SATA (Serial AT Attachment) interface15): 4) Hardware abstraction external 1.8-inch SATA hard-disk drive (HDD) This layer provides hardware abstraction to sepa- • Digital TV connector: Digital TV board rate actual hardware from the Android system. 4) Display/operation functions 5) Linux kernel The following display functions and operation The use of a Linux kernel means that device driv- keys are provided: ers developed specifically for Linux use can be used • Touch panel: 7-inch LED LCD as-is. • Home, menu, back, volume up/down The advantage of introducing the Android sys- 5) Input/output terminals tem into this application platform is that development The following input/output functions are provided: tools—such as the following—are already provided. • Microphone/line input • Android SDK (Software Development Kit) • Headphone output • Android NDK (Native Development Kit) • Stereo speakers (8 ohm) Either or both of these tools can be used depend- • CVBS (color, video, blanking, and sync) Out ing on the requirements and specifications of the target • S/PDIF (Sony/Philips Digital Interconnect Format) product. • HDMI This application platform enables the creation of an image-processing library based on a multimedia

54 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

Application

Home Browser Media Player Contacts

Application framework Active Window Resource Content Manager Manager Manager Providers

Library Android runtime Surface Media Manager Framework SQLite Libc Core library Audio Virtual machine (Dalvik VM) OpenGL ES Manager FreeType Hardware abstraction Graphics Audio Camera Bluetooth GPS Wi-fi Linux kernel Display Driver Camera Driver Bluetooth Driver Shared Memory Binder (IPC) Driver Driver Power USB Driver Key Driver Wifi Driver Audio Drivers Management

Figure 6 Basic confiFigure guration 6 of Android system. Basic configuration of Android system. library (OpenMAX/GStreamer)16) and 2D/3D graphics library (OpenVG/OpenGL ES), which constitute a gen- eral-purpose media framework for embedded devices.

These media libraries make extensive use of Camera hardware-assist functions, enabling the desired level MIPI interface of video performance to be achieved with low power consumption. Introducing the Android system in this way enables the customer to focus on application devel- opment based on standard application programming interfaces (APIs). Moreover, the use of the OpenGL ES/ Camera Milbeaut parallel interface image OpenMAX framework negates the need to create soft- processor ware assets based on APIs associated with SoC-unique Figure 7 technology. This provides the customer with shorter Figure 7 Application example using developer tool kit. development periods and reduced labor requirements. Application example using DTK.

3. Application example This section presents an example of using the DTK needs of the application product. introduced above to construct an integrated system The confi guration of this system is shown in using the Milbeaut image processor, which represents Figure 7. proprietary Fujitsu image-processing technology. Software support is easily provided by introducing This system provides two channels of parallel the Android system. Basically speaking, only device interfaces and one channel of a MIPI slave interface; drivers need to be prepared for the user. which of these three interfaces to use depends on the In this system, basic functions for handling a

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 55 A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

camera that captures images through a preview or measures listed below must be taken. shutter operation can be achieved by simply creating 1) Support of non-competitive areas an Android application since APIs are already supported • Device technology and CPU-core technology must in Android. In other words, the user need not be con- be of the same generation or one generation cerned about device details. ahead. Once such basic camera functions have been com- • Upgrades must be immediately made to reflect pleted, the user can devote valuable time to optimizing the latest versions of Linux and Android OSes, individual features that demonstrate key product speci- Khronos standard APIs, OpenCL, OpenCV17), and fications. For example, the user can now: other software. • Analyze performance 2) Support of competitive (company-differentiating) • Make picture-quality adjustments areas • Support added functions such as auto-focus • Device technology centered about low power con- In the above way, the DTK can be used as a de- sumption must be enhanced. velopment platform that makes it easy to construct an • Image processing technology must be enhanced. integrated system. The labor that previously had to be • Software assets for simplifying system integration allocated to developing an entire system can now be must be developed. concentrated in that portion of the system that empha- Basic, custom, and two-chip platforms for sizes those features—such as image/video quality—that achieving the above are shown in Figure 8. The plan differentiate the product-developing company from is to provide three types of platforms. One is a basic other companies. platform consisting of computing and imaging subsys- tems [Figure 8(a)]. A single chip with this platform 4. Future developments configuration can be provided as a one-chip solution The computing subsystem of the MB86S62 appli- with Linux and Android built-in. Next is a custom cation processor is basically the same as that of similar platform based on the above chip but embedding an products from other companies. To clearly differentiate application-unique section [Figure 8(b)]. The third is a product from the products of other companies, the a two-chip platform consisting of a platform chip and

Multimedia framework Middleware Multimedia framework Middleware Multimedia framework Middleware

Kernel Kernel Kernel

Drivers Drivers Drivers Drivers Drivers

Computing Imaging Computing Imaging Computing Imaging subsystem subsystem subsystem subsystem subsystem subsystem

Customer hardware specifications

DDR Inter-chip DDR Inter-chip DDR Inter-chip Peripherals Peripherals Peripherals I/F connection I/F connection I/F connection

(a) Basic platform (b) Custom platform

Inter-chip connection Platform-provided section Companion Application-unique added section ASIC

(c) Two-chip platform

FigureFigure 8 8 PlatformPlatform and productand product expansion. expansion.

56 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

2012 2013 2014 2015

MB86S7x In planning 20 nm Next CPU multi-core OpenGL ES, OpenVG, OpenCL Toward high-function applications OpenSL ES, OpenCV 8K4K resolution MB86S64 In development 28 nm Cortex™-A15 dual @2 GHz Cortex™-A7 dual @1 GHz OpenGL ES, OpenVG, OpenCL 4K2K resolution Toward low-power/mobile applications MB86S62 MB86S6x Development completed In planning 45 nm 28 nm Cortex™-A8 @1 GHz Cortex™-A7 quad @1 GHz H.264 Full HD Codec OpenGL ES, OpenVG, OpenCL OpenGL ES, OpenVG OpenSL ES, OpenCV Full HD support 4K2K resolution

4K2K resolution: 3840 × 2160 pixels Figure 9 8K4K resolution: 7680 × 4320 pixels Development roadmap. Figure 9 Development roadmap. a companion chip that can interface via a wideband 2) Linux. inter-chip connection [Figure 8(c)]. http://www.kernel.org/ Our plan is to set up a platform development en- 3) Android Developers. http://developer.android.com/index.html vironment that implements the above measures and to 4) MIPI Alliance. market by fiscal 2013 a product featuring 28-nm pro- www.mipi.org cess technology, a built-in CortexTM-A15 processor, and 5) MPEG LA. 4K-class video processing (Figure 9). http://www.mpegla.com/main/default.aspx 6) International Telecommunication Union Telecommunication 5. Conclusion Standardization Sector. http://www.itu.int/en/pages/default.aspx We introduced an imaging application platform 7) Society of Motion Picture and Television Engineers. using Fujitsu’s MB86S62 application processor. As a fu- http://www.smpte.org/ ture imaging solution, we envision a highly functional, 8) International Organization for Standardization. interactive, cloud-linked display system probably in http://www.iso.org/iso/home.html the form of a tablet that will find widespread use in 9) NXP Semiconductors. diverse scenarios in the home, office, and public fa- http://www.classic.nxp.com/acrobat_download2/ various/I2SBUS.pdf cilities. Looking forward, Fujitsu is committed to using 10) International Electrotechnical Commission. its advanced device technologies to develop similar http://www.iec.ch/ application platforms for 20-nm and beyond process 11) The Khronos Group: Connecting Software to Silicon. generations and to provide an expanded lineup of http://www.khronos.org/ image-processing solutions. 12) HDMI Licensing, LLC. http://www.hdmi.org/ 13) USB Implementers Forum, Inc. References http://www.usb.org/home 1) The 3rd Generation Partnership Project. 14) SD Association. http://www.3gpp.org/ https://www.sdcard.org/home/

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 57 A. Suga et al.: Imaging Application Platform Using MB86S62 Application Processor

15) Serial ATA International Organization. http://gstreamer.freedesktop.org/ http://www.serialata.org/ 17) OpenCV. 16) freedesktop.org. http://opencv.org/

Atsuhiro Suga Masato Hyodo Fujitsu Semiconductor Ltd. Fujitsu Semiconductor Ltd. Mr. Suga is engaged in the development Mr. Hyodo is engaged in the development of application processors for imaging of application processors for imaging solutions. solutions.

Hidetoshi Nakahara Shinichi Takimoto Fujitsu Semiconductor Ltd. Fujitsu Semiconductor Ltd. Mr. Nakahara is engaged in the devel- Mr. Takimoto is engaged in the devel- opment of application processors for opment of application processors for imaging solutions. imaging solutions.

58 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013)