Image Signal Processing LSI Chip for Mobile Phones

 Satoru Komatsu  Mitsumaro Kimura  Akira Okawa  Hideaki Miyashita

The camera function of mobile phones has progressed in the last few years, as typified by the rapid spread of high count, high resolution, and full high-definition video recording and playback features in smartphones. Although it is possible to implement camera functions such as rapid imaging and browsing from personal computers and mobile terminals into the applica- tion in a mobile phone, many customers prefer dedicated image signal processors (ISPs) as they offer image quality and performance comparable to those of compact high-end digital cameras. This paper describes the development and performance of the Milbeaut Mobile image processing LSI chip developed by for mobile phones.

1. Introduction satisfactory to users. The market for smart mobile phones continues to By 2008, the 3rd generation Milbeaut Mobile ISP expand due to their diverse applications and functions. incorporated such functions as still-im- The camera function in particular has been evolving age continuous automatic focus (CAF), face detection, steadily since it was introduced to mobile phones in and electronic motion compensation (Figure 1). The 2000 and is now a major mobile phone function, pro- latest generation ISP (the MBG046) incorporates nearly viding high-pixel count and high resolution as well as all camera functions. It also features full high-defi- full high-definition video recording and playback. nition (1920 × 1080 ) video, video CAF, and 3D Milbeaut Mobile is a series of image signal pro- video imaging as demanded by the market. cessors (ISPs) for use by the camera function in mobile The recent achievement of a further reduction in phones. It was developed by Fujitsu and has been on imaging device pixel size, a higher pixel count, and the market since 2003. Here, we present an overview faster imaging has caused problems with image quality of the Milbeaut Mobile series and describe the system and has increased ISP power consumption. architecture, features, functions, and processing per- formance of the MBG046, the latest Milbeaut Mobile 3. ISP architecture processor. The image processor module is the main func- tional component of the Milbeaut Mobile ISP (Figure 2). 2. Evolution in Milbeaut Mobile It sequentially processes the image data received in functionality the Bayer array signal from the , includ- The camera function is achieved by using a spe- ing noise removal and other corrective functions, to cialized lens and image sensor that must be small and produce a beautiful image and conversion into a for- thin for mounting on mobile phones. These constraints mat for display on the screen of a mobile terminal or initially prevented image quality from being as good as personal computer. that of digital cameras. An ISP must therefore be able The module can also access hardware image filters to raise the quality of images captured by a lens and and the CPU to perform additional processing (rotation image sensor under such severe constraints to a level and distortion correction, software image processing,

FUJITSU Sci. Tech. J., Vol. 49, No. 1, pp. 17–22 (January 2013) 17 S. Komatsu et al.: Milbeaut Image Signal Processing LSI Chip for Mobile Phones

Year —2008 2009 2010 2011 2012—

Market High pixel Still image 3D video High-speed imaging Video Low power demand count quality imaging consumption

High definition video Full high-definition video Still image Main Still-image CAF super-resolution zoom Video CAF 3D Zero shutter imaging Video Low power functions Face detection Shading correction Keystone correction super-resolution zoom consumption Electronic motion compensation Local WDR Panorama Stronger serial imaging Low-profile module

Milbeaut 6th 3rd 4th 5th generation Mobile generation generation generation series (MBG046)

WDR: Wide dynamic range Figure 1 Evolution in Milbeaut Mobile ISP functionality. Figure 1 Evolution in Milbeaut Mobile ISP Functionality.

MBG046

Image processor Codec processor External output module Pre- Color Resolution MIPI I/F Image sensor I/F processing processing Image JPEG MIPI CSI I/F conversion filtering Demosaic codec YUV I/F YUV I/F AE/AF/AWB CSC

Serial I/F SIO/SPI GPIO DMAC

2 2 CPU I C I/F I C EXT SDRAM PWM INT Motor UDC Peripheral macro module ARM module

Motor/flash External interrupt

AE: Automatic exposure GPIO: General-purpose input/output AF: Automatic focus I2C: Inter- ARM: High-speed processing INT: Internal using a platform made by ARM Ltd. MIPI: Mobile industry processor interface AWB: Automatic white balance PWM: Pulse width modulation CSC: Color-space conversion SIO: Serial input/output CSI: Camera serial interface SPI: Serial peripheral interface DMAC: Direct memory access controller UDC: USB device controller EXT: External YUV: Signal format consisting of luminance and color difference Figure 2 MBG046 block diagram. Figure 2 MBG046 block diagram.

18 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) S. Komatsu et al.: Milbeaut Image Signal Processing LSI Chip for Mobile Phones

etc.) on image data that is stored in SDRAM (memory simultaneously to match the properties of the image. device for image data). That enables flexible advanced 3) Fast image processing image processing such as face detection, high dynamic Products for mobile phones must be configured range (HDR) composite processing, still-image motion for connection to an application processor as the next compensation, object tracking, panoramic image gen- processing stage. Unless the Bayer array signal from eration, and scene detection. the sensor is converted and corrected and the result sent to the application processor as quickly as possible, 4. MBG046 ISP functions there may be a delay in response-sensitive cases such The latest generation Milbeaut as video. As shown in Figure 2, the Milbeaut Mobile is the MBG046. Its basic operation involves real-time achieves that speed by incorporating a function that conversion and correction of the Bayer array signal sends the image data processed by the image proces- from the sensor for output to the application proces- sor unit directly to the image output unit instead of the sor (not shown in Figure 2), which is connected to an usual scheme of performing the various interpolation external output module. The ISP is thus optimized for processes via SDRAM. That virtually eliminates delay mobile camera systems and enables the application due to the conversion and correction of the Bayer array processor to display the received image data on the signal (the delay is reduced to 1% or less of the transfer screen and store the data in the storage device without time for one image). modification. 4) Software configuration 1) High-speed image processing The software stack of the MBG046 is illustrated in As described in Section 3, the image processor Figure 3. The software processing that is essential to performs the basic image processing functions. It can the camera system is the hardware sequence control process 8-MB images at up to 27 frames per second and computation for the "3A module" (automatic white (fps) and high-definition images at more than 60 fps. balance adjustment, automatic exposure adjustment, The preprocessor, which is part of the image pro- and automatic focusing). The hardware controller cessor, accepts the Bayer array signal from the image performs timing control among different independent sensor and performs sequential conversion and correc- image processing hardware devices to implement one tion processing. It also collects statistical data on the sequence of the image processing flow. The 3A module image data. The statistical data is used in software calculates the appropriate exposure, white balance, computation to obtain an appropriate white balance and focal point from luminance, color-balance, and and exposure time. The calculated results are used to spatial-frequency information obtained from device reset the hardware to enable basic camera operation. drivers. By relegating those parts involving complex Thus, nearly all of the basic image processing functions processing but light computational load to software are implemented in hardware, resulting in high-speed and those parts that involve heavy-computational- processing and low power consumption. load image data processing to specialized hardware, 2) Strong noise reduction we were able to keep the computational load of the Because Milbeaut Mobile targets cameras built software light while maintaining image processing into mobile phones, it is most often used with lens performance. That makes it possible to assign CPU and image sensors that are less expensive than those resources to additional software image processing (mo- used in digital cameras, so image quality is a problem. tion compensation, face detection, etc.) and achieve a Dealing with is a particularly important system that is both flexible and capable of advanced function. processing. Therefore, we equipped the image processor with mechanisms to reduce luminance noise and color noise 5. Milbeaut Mobile 3D processing that are stronger than those used in digital cameras. functions To remove noise without loss of resolution, we incorpo- The configuration of a 3D video imaging system rated the latest technology into the MBG046, enabling with a Milbeaut Mobile ISP is shown in Figure 4. In application of different methods of noise removal a conventional camera system, 3D video imaging is

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 19 S. Komatsu et al.: Milbeaut Image Signal Processing LSI Chip for Mobile Phones

Host (application processor) I/F

3A (AWB/AE/AF) • White balance adjustment • Exposure adjustment Software image processing Device • Focusing • Motion compensation drivers • Face detection Hardware control • Dynamic range control • Image processing engine • Scene recognition sequence control • Frame memory management

OS:µITRON

AWB: Automatic white balance AE: Automatic exposure AF: Automatic focusing Figure 3 MBG046 software stack. Figure 3 MBG046 Software stack. Conventional 3D solution New 3D solution (MBG046)

1. Color processing 2. Geometric correction 1. Color processing 3. Parallax detection 3. Parallax detection 2. Geometric correction 4. Parallax correction 4. Parallax correction Sensor Milbeaut 5. Side-by-side generation 5. Side-by-side generation Mobile Sensor Bayer Application Milbeaut Application processor Bayer Mobile processor Sensor Sensor （MBG046） synchronization Image Image Sensor Milbeaut Bayer Mobile Bayer

1. Color processing 2. Geometric correction

Figure 4 Figure3D video imaging 4 system confi guration. Video imaging system configuration. implemented using two ISPs, each of which\\fronta2.lip.fujitsu.com\FSTJ_Jan_03\初校戻し processes 1) Geometric correction and parallax correction the image data input from one of a pair of image sen- 3D image generation requires two types of sors for the left and right eyes. The separate ISPs are correction. used for processing the left and right images, and The difference in the mounting position between a mechanism is required for sensor synchronization the two camera modules produces differences in image between them. The correction processing needed for height and angle between the images from the two 3D imaging is implemented at a later stage in the ap- image sensors that are input to the ISP. Correction of plication processor. Fujitsu Laboratories improved that those differences is called geometric correction. confi guration by developing a 3D correction algorithm The difference in the angle at which an object in that can do optimum 3D processing with a single ISP. actual 3D space is seen by the left and right eyes in

20 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) S. Komatsu et al.: Milbeaut Image Signal Processing LSI Chip for Mobile Phones

accordance with distance, which is called parallax, is image, and the degree of parallax correction that is the basis for human depth perception. Excessive par- required is determined. That parallax correction is allax may cause fatigue when viewing 3D, so parallax multiplied by the previously saved geometric correction correction is required. to create a rectangle conversion that is applied to the 2) 3D video processing flow original image. In this way, both corrections can be ac- The 3D video processing fl ow implemented in the complished at one time. This more effi cient use of in MBG046 is shown in Figure 5. First, it is necessary to the processing bandwidth makes it possible to imple- prepare correction data for each individual difference ment full high-defi nition 3D video imaging at 30 fps. to be used in the geometric correction. Because the 3) 3D display format conversion camera modules are mounted in fi xed positions, the After correction by rectangle conversion, the correction processing for that difference is done once, signal must be converted to side-by-side or another and the result is stored in non-volatile memory for use standard 3D display format for output to a 3D display. in later imaging. That correction value is obtained by The MBG046 has two systems for this, one for image imaging a fl at-screen test chart and then calculating an sensor input and one for internal processing. That en- error value from height and rotation deviations. ables integrated processing, which extends the color Next, the geometric correction value is used to processing performed by an ordinary ISP to 3D process- reshape (square) the left and right images, leaving ing on a single chip, thus achieving high-quality 3D parallax as the only difference between the images. video imaging. This approach also can simplify image If excessive parallax is detected, a value for correcting sensor synchronization and reduce the load on the ap- the parallax to within an appropriate range is calcu- plication processor. lated. That value is used as the basis for the rectangle conversion. This two-pass rectangle conversion would 6. Conclusion consume processing bandwidth if performed on the We have described the evolution, architecture, original image, so single-pass processing is used. First, and functions of Fujitsu's latest Milbeaut Mobile image the geometric correction is performed on a compressed processor for mobile phones. We can expect continued

Binocular camera Left-eye image Right-eye image MBG046

Installation Right-eye camera Image error detection processing Pre-measured individual installation error Left-eye camera Geometric Geometric correction correction value Parallax detection Parallax correction value 3D TV Parallax correction Calculation of parallax correction value appropriate for human eye

Video imaging: Full high-definition 30 fps

Figure 5 MBG046Figure 3D video 5 imaging process. MBG046 3D video imaging process. FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 21 S. Komatsu et al.: Milbeaut Image Signal Processing LSI Chip for Mobile Phones

steady improvement in the performance of mobile that will accompany faster network channels. Fujitsu terminals, which will lead to increasing demand for im- will continue developing image signal processors that aging processors to cope with further development of have optimum performance and functions, which will the camera function toward lower power consumption, enable us to offer a reliable product line that keeps up faster still imaging, a lower camera module profile, with those trends. and the higher capacity and frame rates for video data

Satoru Komatsu Akira Okawa Fujitsu Semiconductor Ltd. Fujitsu Microelectronics Solutions Ltd. Mr. Komatsu is engaged in development Mr. Okawa is engaged in development of of the Milbeaut LSI for mobile phones. firmware for the Milbeaut LSI for mobile phones.

Mitsumaro Kimura Hideaki Miyashita Fujitsu Microelectronics Solutions Ltd. Fujitsu Microelectronics Solutions Ltd. Mr. Kimura is engaged in development of Mr. Miyashita is engaged in development firmware for the Milbeaut LSI for mobile of firmware for the Milbeaut LSI for mobile phones. phones.

22 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013)