University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Faculty Publications from the Department of Electrical & Computer Engineering, Department Electrical and Computer Engineering of 2011 Energy-efficientor F eground Object Detection on Embedded Smart Cameras by Hardware-level Operations Mauricio Casares University of Nebraska-Lincoln, [email protected] Paolo Santinelli University of Modena, [email protected] Senem Velipasalar University of Nebraska-Lincoln, [email protected] Andrea Prati University of Modena, [email protected] Rita Cucchiara University of Modena and Reggio Emilia Follow this and additional works at: https://digitalcommons.unl.edu/electricalengineeringfacpub Part of the Electrical and Computer Engineering Commons Casares, Mauricio; Santinelli, Paolo; Velipasalar, Senem; Prati, Andrea; and Cucchiara, Rita, "Energy- efficientor F eground Object Detection on Embedded Smart Cameras by Hardware-level Operations" (2011). Faculty Publications from the Department of Electrical and Computer Engineering. 202. https://digitalcommons.unl.edu/electricalengineeringfacpub/202 This Article is brought to you for free and open access by the Electrical & Computer Engineering, Department of at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Faculty Publications from the Department of Electrical and Computer Engineering by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 doi: 10.1109/CVPRW.2011.5981838 Energy-efficient Foreground Object Detection on Embedded Smart Cameras by Hardware-level Operations Mauricio Casares1, Paolo Santinelli2, Senem Velipasalar1, Andrea Prati2 and Rita Cucchiara2 1University of Nebraska-Lincoln, Dept. of Electrical Engineering [email protected], [email protected] 2University of Modena and Reggio Emilia, Modena Italy [email protected], [email protected] Abstract sources, such as computational power, memory and energy. Since battery-life is limited, and video processing tasks Embedded smart cameras have limited processing consume considerable amount of energy, it is essential to power, memory and energy. In this paper, we introduce have lightweight algorithms, and methodologies to increase two methodologies to increase the energy-efficiency and the the energy-efficiency of each camera. With the advances battery-life of an embedded smart camera by hardware- in hardware technology, embedded devices are becoming level operations when performing foreground object detec- more sophisticated. Embedded smart cameras are being tion. We use the CITRIC platform as our embedded smart equipped with general purpose processing units that allow camera. We first perform down-sampling at hardware level implementing sophisticated vision algorithms on these plat- on the micro-controller of the image sensor rather than per- forms. forming software-level down-sampling at the main micro- Fleck et al. [4] present a network of smart cameras for processor of the camera board. In addition, we crop an im- tracking people. They use commercial IP-based cameras, age frame at hardware level by using the HREF and VSYNC which consist of a CCD image sensor, a Xilinx FPGA and signals at the micro-controller of the image sensor to per- a Motorola PowerPC. Cameras communicate via Ethernet form foreground object detection only in the cropped search connections. Quaritsch et al. [7] employ smart cameras region instead of the whole image. Thus, the amount of data with multiple DSP processors for data processing. Bram- that is moved from the image sensor to the main memory berger et al. [1] presented a smart camera architecture reach- at each frame, is greatly reduced. Thanks to reduced data ing a processing power of 9600 MIPS with onboard mem- transfer, better use of the memory resources and not occu- ory of 784 MB. While this high-end platform provides suf- pying the main microprocessor with image down-sampling ficient capabilities for image processing, it requires an aver- and cropping tasks, we obtain significant savings in energy age power consumption of 35 Watts. consumption and battery-life. Experimental results show Wired or IP-based cameras have powerful processing ca- that hardware-level down-sampling and cropping, and per- pabilities and relatively high bandwidth for communication. 54.14% forming detection in cropped regions provide de- However, they have high power consumption and are larger 121.25% crease in energy consumption, and increase in in size. Many embedded vision platforms have been devel- battery-life compared to performing software-level down- oped more recently. The Cyclops [9] and MeshEye [5] plat- sampling and processing whole frames. forms have 7.3-MHz and 55-MHz processors, respectively. Thus, in both of these platforms the processing power is 1. Introduction very limited. The camera mote introduced by Kleihorst et al. [6] has an 84MHz XETAL-II SIMD processor, and Wireless embedded smart cameras are stand-alone units uses a higher resolution. The CMUcam2 [11] is a low- that combine sensing, processing and communication on cost embedded camera with 75MHz RISC processor and a single embedded platform. Rather than transferring all 384KB SRAM. Due to the limited memory and processing the data to a back-end server, they can process images, power, only low-level image processing can be performed. and extract data locally. Even though battery-powered em- Panoptes platform [3] hosts a 206-MHz processor, but has bedded smart cameras provide a lot of flexibility in terms high energy consumption. SensEye [8] is a multi-tier net- of camera quantities and placement, they have limited re- work of heterogeneous wireless nodes and cameras. Rinner 150 et al. [10] present a comparison of various smart camera sensor, a microprocessor, external memories and other sup- platforms. Kerhet et al. [14] employ a hybrid architecture porting circuits. The microprocessor PXA270 is a fixed- FPGA-MCU, where the processing load is distributed to in- point processor with a maximum speed of 624 MHz and crease the efficiency of the camera mote. Chen et al.[2] 256 KB of internal SRAM. The PXA270 is connected to a introduced the CITRIC camera mote that provides more 64 MB of SDRAM and 16MB of NOR FLASH. Attached computing power and tighter integration of physical com- to the camera board is a TelosB mote from Crossbow Tech- ponents while still consuming relatively little power. An nology with a maximum data rate of 250 Kbps. Omnivision sensor OV9655 is employed to capture frames. The down-sampling of the images is performed by software using the CITRIC API libraries. Casares et al. [12] introduced an algorithm for resource- efficient foreground object detection, and presented the sav- ings in processing time and energy consumption on CITRIC cameras. They obtained the savings in software-level. In this paper, our goal is to increase the energy-efficiency and the battery-life further by hardware-level operations when performing object detection. We present two methods to Figure 1. CITRIC camera: the wireless embedded smart camera achieve this: (i) rather than performing down-sampling and platform employed in the experiments. image cropping at the main microprocessor on the camera board, we perform these operations at the micro-controller 2.1. The Image Sensor of the OV9655; (ii) we crop an image frame at hardware The image sensor on the CITRIC camera is a OmniVi- level by using the HREF and VSYNC signals at the micro- sion OV9655 [13], which is a low voltage SXGA CMOS controller of the image sensor to perform foreground ob- image sensor with an image micro-controller on board. It ject detection only in the cropped search region instead of supports image sizes SXGA (1280 × 1024), VGA (640 × the whole image. Performing these functions at hardware 480), CIF (352 × 288), and any size scaling down from CIF level provides multiple advantages including savings in pro- to 40 × 30, and provides 8-bit/10-bit data formats [2]. cessing time and significant decrease in energy consump- tion. The amount of data, which is moved from the im- age sensor to the main memory at each frame, is greatly reduced. This, in turn, leads to significant savings in en- ergy consumption thanks to the better use of the mem- ory controller and the memory resources and not occupy- ing the main microprocessor with performing image down- sampling and cropping at software-level. We compare performing down-sampling and cropping by software using the CITRIC API libraries, and by hard- ware using the micro-controller of the image sensor. We present the experimental results showing the savings in processing time and energy consumption, and the gain in Figure 2. Interconnection of OV9655 and the Intel Quick Capture the battery-life when performing down-sampling and crop- Interface on ARM PXA270. ping operations at hardware-level, and processing only the cropped region of an image frame. Due to lack of com- The image sensor offers the full functionality of a cam- patibility between the device driver containing the existing era and image micro-controller on a single chip. There is kernel and the actual camera sensor, the results and savings a complete control over image quality. Formatting, output are provided based on a single frame. Currently, a new ker- data transfer and all required image processing functions nel patch is under evaluation to capture successive cropped are also programmable. The architecture and hardware in- frames with adaptive window sizes. terfaces of the OmniVision OV9655 are depicted in Fig. 2. The Serial Camera Control Bus (SCCB) interface is used to 2. The Embedded Smart Camera Platform program the sensor behavior by setting all the control reg- isters in the device. It is an Inter-Integrated Circuit (I2C) The wireless embedded smart camera platform em- compatible hardware interface. The Digital Video Port pro- ployed in our experiments is a CITRIC mote [2]. It consists vides a connection between the sensor and the CITRIC cam- of a camera board and a wireless mote, and is shown in Fig- era main processor PXA270.