YUV422 to YUV420 Color Space Conversion on FPGA

Research and Applications: Embedded System Volume 3 Issue 2 DOI: [To be assigned]

Abhilash R. Namboodiri1, Sujata D. Badiger2*, Rajagopal Narayanasamy3 1,2Department of Electronics and Communication Engineering, RV College of Engineering, Bengaluru, Karnataka, India 3Senior Member of Technical Staff, Cypress Semiconductors, RV College of Engineering, Bengaluru, Karnataka, India *Corresponding Author E-Mail Id: [email protected]

ABSTRACT YUV is a popular color encoding system similar to the popular RGB variant but is less data intensive. YUV stores only the difference among the chrominance values and absence of data bytes can be represented as a grayer image resulting in reduced data consumption. YUV422 (YCbCr) forms the standard video format used in various broadcasting technologies and standards. However, at high resolutions and higher frame rates it needs to be converted to YUV420 format which is similar but not as data intensive. This conversion is realized in Verilog and implemented on an FPGA which converts the video data for real-time applications.

Keywords: Color Space Conversion, YCbCr, YUV Encoder, YUV420, MIPI, FPGA Color conversion

INTRODUCTION describe of brightness and color samples in YUV is a color encoding system typically a hypothetical region that is A pixels wide, used as part of a color image pipeline.[17] and 2 pixels high. The ratios are described It is a raster graphics extension and stores by: a bitmap of the resulting image. YUV splits color into Y, U and V values; storing  A: Horizontal reference for sampling; luminance (brightness) as Y values, and translates to width of the region. In chrominance (color) as U and V. It example: 4 encodes a color image or video taking  B: Chrominance samples (Cr, Cb) human perception into account; thus, present first row of A pixels or chrominance components are marked with horizontal sampling a reduced bandwidth. This allows  C: Number of changes of chrominance transmission errors and compression samples (Cr, Cb) between first and which can be masked more efficiently by second row of J pixels, which the human perception than using a translates to vertical sampling B must standard RGB based representation. [8,9] be either zero or equal to A (except in rare irregular cases like 4:4:1 and YUV is implemented via chroma 4:2:1). subsampling where images are encoded with lesser chroma, U representing YUV422 formats (UYVY and YUY2) take Chroma-blue (Cb) and V representing 16 bits per pixel and are packed Chroma-red (Cr), information for its sequentially, at high resolutions and frame corresponding luma (Y) information and is rates it becomes easier to process them at expressed as a ratio in three different parts YUV420 formats which take only 12 bits A:B:C (e.g. 4:2:2), [14]these values of data per pixel which allow them to be

HBRP Publication Page 1-7 2020. All Rights Reserved Page 1 Research and Applications: Embedded System Volume 3 Issue 2 DOI: [To be assigned] easily processed and streamed in real for both encoder and decoder. YUV420 to time.[21-23] YUV422 conversion involves interpolation, commonly accepted and followed interpolation technique is the cubic convolution interpolation [6], developed by Robert Keys, which is seen to drastically outperform linear and nearest neighbor interpolations. Cubic convolution can also be performed easily by processor without any addition of numerical methods to approximate. Cubic convolution interpolation function is found to converge uniformly to the function being interpolated as the sampling increment tends to zero, or it approximates better when sample size taken is smaller.[19]

Fig. 1: YUV422 A:4 B:2 C:2. YUV color space is widely used for image processing with multiple techniques to deal with widely available in literature. Y neither corresponds completely to luminance and U and V do not represent the actual color or chroma content, however, a rough correspondence between the YUV signals and visual attributes does exist. This modular feature makes YUV color space more convenient for image processing than various other color spaces, [15,18] suggesting the possibility that color enhancement algorithms could be developed using the YUV color space having equivalent performances with other popular uniform color spaces, like CIECAM02.[4,7] Fig. 2: YUV420 A:4 B:2 C:0. Raw video is usually encoded and YUV is the preferred format for compressed by a standard video encoder broadcasting and televisions all around the and later decoded at the time of playback, world [5]. Another advantage adopting for efficient storage and transmission. An YUV standards is compression, taking interframe method is described in [16] advantage of human perception to lower where first decoded YUV frame is resolution without adversely impacting transferred completely and then converted subjective quality. Chroma subsampling to RGB using a color space converter. This consists in several color pixels sharing the is followed by only Y component same chroma value leading to lowering of transmission in the second frame. Cb and spatial resolution of chroma channels Cr components are re-used from the vertically, horizontally or both. Chroma previous frame making it computationally subsampling is useful for both memory efficient. and throughput as it allows higher bitrate

SYSTEM ARCHITECTURE

Fig. 3: System Block Diagram.

Objective of the system is to build a low Controller (Cypress FX2LP) cost USB2 UVC Camera. Composition of 2. FPGA (Lattice Crosslink) the system includes: configuration flow via I2C  Cypress FX2LP board, a peripheral transactions from USB Controller controller connected to Host (PC) and can relay data to and from host device To configure FPGA, Verilog code initially connected to a USB connector. written on host must first be sent to  I2C EEPROM to store any USB2.0 controller before it is then configuration data. transferred to the FPGA.[11-13]] This  Bridge board consisting of: transaction of the entire resulting o Lattice CrossLink: FPGA functioning Bitstream occurs via Control Endpoint in as MIPI parallel bridge connected to USB 2.0 to recursively transmit the entire FX2LP in 16-bit parallel data bitstream. Host Application automatically interfaced with FX2LP’s GPIO pins. detects recently enumerated USB Device o Image Sensor module to relay video and allows choice of the file to be data from the image sensor to MIPI transferred. It also updates status bridge (CrossLink) periodically after every block of 4096 o Miscellaneous components including bytes. This recursively occurs till it Voltage regulators, USB Serial/I2C reaches the last block and sends the debugger remaining of the bytes which may or may not be equal to 4096 bytes. The application EXPERIMENTAL DETAILS AND copies data from the file accessed by a file CONFIGURATION FLOW pointer onto a temporary buffer on the host Configuration of FPGA convertor logic which is then accessed by the control pipe involves the following: and loaded onto control packets finally 1. Bitstream transfer via Control decoded and stored in Endpoint 0 endpoint from Host to USB peripheral Buffer.[2,9,10,20]

Fig. 4: USB Control Transfer.

Once the FPGA logic is sent to USB This is captured as end of config and Controller it is then transferred to FPGA switches the device to user state. Assertion according to its specific configuration of the internal DONE causes a Wake-up sequence. When the entire configuration state machine to run and CrossLink enters data in the form of a bitstream is received, User Mode immediately when the Wake- FPGA asserts an internal DONE status. up sequence has completed.

Fig. 5: I2C Signal Trace.

Our aim was conversion of the packed sensor is arranged in packed format with a YUV422 format to a planar YUV420 macro-pixel defined by two luma (Y) and format. This involves horizontal respective chroma (U & V) bytes. This is subsampling of Chroma values (Cb and converted to YUV420 planar format as in Cr), which is achieved by down sampling Figure 7 with all luma elements sent at chroma for every alternate packed set of once and then utilizing a frame buffer to two pixels. The UV samples are stored in a store respective chroma values separate UV subspace and eventually sent alternatively and append them in the end in after all the luma components are sent as a planar format with different Luma and referenced by Figure 6 and Figure 7. Chroma planes. Incoming packed format from the camera

Fig. 6: YUV422 Packed Format.

Fig. 7: YUV420 Planar Format.

RESULTS AND DISCUSSION format. This was set for a simulation Color Conversion occurred from packed design and will subsequently interface YUV422 format to a planar YUV420 with a Camera/UVC Sensor Interface.

Fig. 8: Waveforms of YUV420 Color Bar Generation after Conversion.

This was supplemented with YUV420 firmware and Verilog code written. Color Bar Generation to test out the design

Fig. 9: YUV Color Bar Generated to Simulate Sensor Data.

CONCLUSIONS demand for high resolution video services Advent of video encoding, high resolution in the areas of TV broadcasting, personal displays and faster network technologies, media has increased multifold. Interfacing

HDMI along with the popular USB convolution. IEEE Transactions on standard becomes an increasing need to Image Processing, 15(7), 1857-1870. cater to growing demand in our 8. Digital interfaces for studio signals interconnected world. Conversion of Color with 1920 × 1080 image formats, format assumes paramount importance as Recommendation ITU-R BT.1120-9, higher resolutions and even faster frame December 2017 rates cannot be accommodated even by the 9. Shah, K. How to Implement an Image large USB3.0 (SuperSpeed) bandwidth. Sensor Interface Using EZ-USB FX3 Conversion to YUV420 from YUV422 in a USB Video Class (UVC) leads to minimal loss in perception can be Framework. Cypress Semiconductors, applied to USB Video applications, App Note: 75779 implemented here on a low-cost FPGA 10. Universal Serial Bus Device Class and USB2.0 peripheral controller. Definition for Video Devices, Revision 1.5, USB-IF, August 2012 REFERENCES 11. Stewart, G., Renshaw, D., & Riley, M. 1. Bae, S. H., Kim, J., Kim, M., Cho, S., (2007, April). A Low-Cost, FPGA & Choi, J. S. (2013). Assessments of Based, Video Streaming Server. In subjective video quality on HEVC- 2007 3rd Southern Conference on encoded. IEEE Standard for System Programmable Logic (pp. 187-190). Verilog--Unified Hardware Design IEEE. video for beyond-HDTV broadcasting 12. Denolf, K., Chirila-Rus, A., Turney, services. IEEE Transactions on R., Schumacher, P., & Vissers, K. Broadcasting, 59(2), 209-222. (2005, August). Memory efficient 2. Wei-Ming, Y. (2010, November). A design of an MPEG-4 video encoder study of USB 3 in perspective aspect. for FPGAs. In International In International Conference on Conference on Field Programmable Computational Collective Intelligence Logic and Applications, 2005. (pp. (pp. 425-432). Springer, Berlin, 391-396). IEEE. Heidelberg. 13. Chang, Y. C., Chao, W. M., & Chen, 3. Bus, U. S. 3.2 Specification, 1st L. G. (2004, October). Platform-based edn.(September 2017). MPEG-4 video encoder SOC design. 4. Levkowitz, H., & Herman, G. T. In IEEE Workshop on Signal (1993). GLHS: a generalized Processing Systems, 2004. SIPS 2004. lightness, hue, and saturation color (pp. 251-256). IEEE. model. CVGIP: Graphical Models 14. Lin, T., Zhang, P., Wang, S., Zhou, and Image Processing, 55(4), 271- K., & Chen, X. (2012). Mixed chroma 285. sampling-rate high efficiency video 5. Series, B. T. (2012). Parameter values coding for full-chroma screen content. for ultra-high definition television IEEE Transactions on Circuits and systems for production and Systems for Video Technology, 23(1), international programme exchange. 173-185. 6. Keys, R. (1981). Cubic convolution 15. Lee, S., Kwak, Y., Kim, Y. J., Park, interpolation for digital image S., & Kim, J. (2012). Contrast- processing. IEEE transactions on preserved chroma enhancement acoustics, speech, and signal technique using YCbCr color space. processing, 29(6), 1153-1160. IEEE Transactions on Consumer 7. Shi, J., & Reichenbach, S. E. (2006). Electronics, 58(2), 641-645. Image interpolation by two- 16. Kamat, S. P. (2010). Low bandwidth dimensional parametric cubic YCbCr data processing technique for

video applications in handheld Transactions on Consumer devices. IEEE Transactions on Electronics, 54(2), 695-702. Consumer Electronics, 56(3), 1770- 20. Vatsa, R. HDMI to USB 3.0 Capture 1774. Card Reference Design. M. Tech 17. Kwak, Y. (2010, January). Design Major Project Report, Dept. Principles of wide gamut YCC color Electronics & Communication encoding space for effective signal Engineering, Nirma University, sub-sampling. In Color Imaging XV: Ahmedabad, India. December 2019. Displaying, Processing, Hardcopy, 21. DDR3 Demo for the ECP5™ and and Applications (Vol. 7528, p. ECP5-5G™ Versa Development 75280O). International Society for Boards User Guide, Lattice Optics and Photonics. Semiconductor, UG97 Version 1.2, 18. Tkalcic, M., & Tasic, J. F. (2003). October 2016 Colour spaces: perceptual, historical 22. Double Data Rate (DDR3) SDRAM and applicational background (Vol. 1, Controller IP Core User Guide, pp. 304-308). IEEE. Lattice Semiconductor, IPUG80 19. Mancuso, R., Smorfa, S., & Olivieri, Version 1.9, October 2016 M. (2008). A novel high-quality 23. IEEE Standard for System Verilog-- YUV-based image coding technique Unified Hardware Design, for efficient image storage in portable Specification, and Verification electronic appliances. IEEE Language, IEEE 1800-2012, 2013.