INFORMATION SHEET Reducing Sony Block Camera Bandwidth with Bayer Filtering Problem High-performance implementations of GigE Vision® can achieve video throughput of up to 980 Mbps. While this throughput is more than adequate for a wide variety of video modes available on Sony FCB-EH and FCB-EV block cameras, some video formats cannot be transmitted uncompressed. In addition, some video formats don’t allow the simultaneous transmission of two or more image streams through a GigE link. Data Rate (Mbps)1 Format Width Height Monochrome YUV4112 YUV4223 (pixels) (pixels) (8 bits per pixel) (12 bits per pixel) (16 bits per pixel) 720p @ 30 fps 1280 720 221 332 442 720p @ 60 fps 1280 720 442 664 885 1080p @ 30 fps 1920 1080 498 746 995 Table 1: Approximate data rates for common video formats One option is to use image compression. While image compression techniques such as MPEG-4, H.264, H.265, JPEG (sometimes referred to as MJPEG or “Motion JPEG”), and JPEG 2000 are mature, well-accepted, and widely adopted, they are not appropriate for applications where low latency or high fidelity of video images is of paramount concern. While these compression techniques achieve good image quality and excellent compression ratios, they are “overkill” for applications where the desired compression ratio is 2:1 or 3:1. Furthermore, these algorithms are computationally expensive, adding to the overall system cost for both cameras (compression) and displays (decompression). Bayer Filter A Bayer filter allows only a single color (either red, green, or blue) to be transmitted for any given pixel on an image sensor. Invented by Bryce Bayer in 1974 while working for Eastman Kodak®, the filter is used today in most single-chip image sensors for personal and industrial still cameras and video cameras. The filter uses twice as many green elements as red or blue, as the human eye is most sensitive to green. This filtering slightly decreases the quality of reds and blues; however, these losses are generally imperceptible to the human eye. 1 Represents image data only, and does not take into consideration the overhead of Ethernet, IP, User Datagram Protocol (UDP), and GigE Vision Streaming Protocol (GVSP), which is less than one percent when utilizing Ethernet “jumbo” frames of at least 8000 bytes in length 2 Corresponds to the YCbCr709_411_8_CbYYCrYY enumeration entry of the PixelFormat feature of the SB-GigE External Frame Grabber 3 Corresponds to the YCbCr709_422_8_CbYCrY enumeration entry of the PixelFormat feature of the SB-GigE External Frame Grabber For more information, visit www.pleora.com Incoming Light Filter Layer Sensort Array Resulting Pattern Figure 1: Overview of Bayer filter (photo courtesy of Wikipedia) Because of this filtering, images output from image sensors employing a Bayer filter must be modified before they are displayed to the user. This is achieved by employing one of many demosaicing algorithms that interpolate the set of complete red, green, and blue values for each pixel. There are many demosaicing algorithms, some which prioritize the quality of the demosaiced image over the speed of demosaicing, and vice versa. Solution In the iPORT™ SB-GigE External Frame Grabber, Pleora employs a Bayer filtering technique on images received from Sony block cameras to meet the following requirements: 1. Accommodate uncompressed video streams with data rates greater than 980 Mbps on a GigE link; and 2. Accommodate multiple (“n”) uncompressed video streams with individual data rates greater than 980/n Mbps on a GigE link. The use of this filtering mode is enabled by the user by selecting the BayerGR8 pixel format. Before converting image data to Ethernet frames, the image is converted from its native video format (e.g. YUV422 at 16 bits per pixel for Sony block cameras) to Bayer (8-bits per pixel) format. Conversion is done while individual pixels are being received, not after the entire image has been received, adding negligible latency to the video stream (significantly less than 1 millisecond). Most importantly, this reduces the video stream data rate by a factor of two. Format Width (pixels) Height (pixels) Data Rate (Mbps) 720p @ 30 fps 1280 720 221 720p @ 60 fps 1280 720 442 1080p @ 30 fps 1920 720 498 Table 2: Approximate data rates for 8-bit Bayer pixel formats Once transmitted, these images are received by Pleora’s eBUS SDK. The SDK has supported the conversion of 8-bit Bayer filtered images to displayable RGB images for many years, as Bayer pixel formats are widely deployed in cameras for industrial vision and defense applications with both human and computer vision system analysis. Users of the eBUS SDK can choose between two interpolation methods: 1. 2x2 matrix – Approximately 3 milliseconds (ms) when processing an XGA image; 2. 3x3 matrix – Approximately 9 ms with the same image as above, but results in a “smoother” output as it takes into consideration more neighboring pixels when interpolating the red, green, and blue values for the current pixel. The above benchmarks were achieved on an Intel® quad core Xeon® processor, running at 3.4 GHz, and using Microsoft Windows 8, 64-bit edition. Because a Bayer filter uses twice as many green elements as red or blue to mimic the physiology of the human eye, no perceivable information is lost in the output image. Figure 2: Original image (left), which is subsequently converted to 8-bit Bayer, then back to RGB (right). Figure 3: Image subtraction and use of Sobel filter for edge detection highlights that minimal information has been lost. Conclusion Bayer filtering is a well-accepted method of reducing the data rate of color image sensors. It reduces bandwidth by a factor of two when YUV422 is the original pixel format (as it is with Sony block cameras), and produces output images with little to no noticeable artifacts. The algorithm adds minimal latency to the system, providing an excellent low overhead, low latency, and high-quality substitute for well-known compression techniques suitable in industrial and defense applications. As a result, Bayer filtering allows for color images to be transmitted over a GigE link at resolutions and rates otherwise not possible with native pixel formats (e.g. YUV422), including common video formats such as 1080p at 30 fps. Pleora Technologies Inc. Tel: +1.613.270.0625 © 2014 Pleora Technologies Inc. vDisplay, iPORT, eBUS-PureGEV, and AutoGEV are trademarks of Pleora Technologies Inc. Information in this document is provided in connection with 340 Terry Fox Dr, Suite 300 Fax: +1.613.270.1425 Pleora Technologies products. No license, express or implied, by estoppels or otherwise, to Kanata, Ontario Email: [email protected] any intellectual property rights is granted by this document. Pleora may make changes to specifications and product descriptions at any time, without notice. Other names and brands K2K 3A2 www.pleora.com may be claimed as the property of others..