Using Xcode with Opencv Instructor - Simon Lucey

Home , C (programming language)

Using Xcode with OpenCV Instructor - Simon Lucey

16-623 - Advanced Computer Vision Apps Today

• Insights into Mobile Computer Vision. • Extremely Brief Intro to Objective C • Using OpenCV in Xcode.

Balancing Power versus Perception

4 Algorithm

Software

Architecture

SOC Hardware Correlation Filters with Limited Boundaries

Hamed Kiani Galoogahi Terence Sim Simon Lucey Istituto Italiano di Tecnologia National University of Singapore Carnegie Mellon University Genova, Italy Singapore Pittsburgh, USA Algorithm [email protected] [email protected] [email protected]

Abstract

Correlation filters take advantage of specific properties in the Fourier domain allowing them to be estimated efficiently: (NDlog D) in the frequency domain, versus (D3 +O ND2) spatially where D is signal length, and ON is the number of signals. Recent extensions to cor- (a) (b) relation filters, such as MOSSE, have reignited interest of their use in the vision community due to their robustness and attractive computational properties. In this paper we Software demonstrate, however, that this computational efficiency 1 comes at a cost. Specifically, we demonstrate that only D proportion of shifted examples are unaffected by boundary effects which has a dramatic effect on detection/tracking performance. In this paper, we propose a novel approach (c) (d) to correlation filter estimation that: (i) takes advantage of inherent computational redundancies in the frequency do- Figure 1. (a) Defines the example of fixed spatial support within main, (ii) dramatically reduces boundary effects, and (iii) the image from which the peak correlation output should occur. (b) The desired output response, based on (a), of the correlation is able to implicitlyAx exploit all possible patches densely ex- = b filter when applied to the entire image. (c) A subset of patch ex- tracted from training examples during learning process. Im- amples used in a canonical correlation filter where green denotes pressive object tracking and detection results are presented a non-zero correlation output, and red denotes a zero correlation in terms of both accuracy and computational efficiency. output in direct accordance with (b). (d) A subset of patch examples used in our proposed correlation filter. Note that our proposed approach uses all possible patches stemming from different 1. Introduction parts of the image, whereas the canonical correlation filter simply Architecture employs circular shifted versions of the same single patch. The Correlation between two signals is a standard approach central dilemma in this paper is how to perform (d) efficiently in to feature detection/matching. Correlation touches nearly D−1 the Fourier domain. The two last patches of (d) show that T every facet of computer vision from pattern detection to ob- patches near the image border are affected by circular shift in our ject tracking. Correlation is rarely performed naively in the method which can be greatly diminished by choosing D<vi- have been proposed [15, 18, 20, 19], the approach’s central sion world through the recent work of Bolme et al. [5] on tenet is to learn a filter, that when correlated with a set of Minimum Output Sum of Squared Error (MOSSE) correla- training signals, gives a desired response, e.g. Figure 1 (b). tion filters for object detection and tracking. Bolme et al.’s SOC Hardware Like correlation, one of the central advantages of the ap- work was able to circumvent some of the classical problems Algorithm

Software

Architecture

Hardware SIMD Vector Extensions Algorithm

+ x 4-way Software

What is it? SIMD (Single Instruction, Multiple Data) Extension of the ISA DataArchitecture types and instructions for the parallel computation on short (length 2, 4, 8, …) vectors of integers or floats Names: MMX, SSE, SSE2, …

Why do they exist? SOCUseful: HardwareMany applications have the necessary fine-grain parallelism Then: speedup by a factor close to vector length Doable: Relative easy to design; chip designers have enough transistors to play with

SSE: Intel x86 Processors Streaming SIMD extension x86-16 8086 AVX: Advanced vector extensions 286 x86-32 386 register 486 width Pentium MMX Pentium MMX 64 bit (only int) SSE Pentium III time SSE2 Pentium 4

SSE3 Pentium 4E 128 bit x86-64 / em64t Pentium 4F

Core 2 Duo SSE4 Penryn Core i7 (Nehalem) AVX Sandy Bridge 256 bit AVX2 Haswell

© Markus Püschel How to write fast numerical code Computer Science Spring 2015 MOBILE GP-GPU: CAPABILITY AND LIMITATIONS Algorithm We implemented Fast Fourier Transform (FFT), a kernel Software computation of many image and signal processing algorithms, on a mobile GPU with the intent of testing the applicability of utilizing mobile GPU for higher level tasks. Its performance and power Architecture consumption were then compared to a mobile CPU and a desktop (a) GPU. Our experiments were performed on an Nvidia Tegra SoC [3] SOC Hardware with the following specifications: a 1GHz dual-core ARM Cortex- A9 CPU, 1GB of RAM, an Nvidia ultra-low-power GeForce GPU running at 333MHz, and 512MB of Flash memory. FFT on GPU A 1D FFT of sample N is defined as

, where . In our study, we examined an implementation of the Cooley-Tukey FFT algorithm, based on the approach presented in [15], using OpenGL ES 2.0 API and the shader language. The processing flow of the Cooley-Tukey method is depicted in Figure 2. A total of stages are required to complete the computation for N samples. (b) Samples in each stage form pairs between two groups, as shown in the dotted box of Figure 2. The computation of this group is Figure 1. (a) A high-level block diagram of a mobile SoC system. expressed as: and . The coefficients The GPU shares the system bus with the CPU and other computing hardwares. (b) The processing flow and block diagram of an ultra- of each sample (i.e. ) is pre-computed and stored as a texture low-power GeForce GPU in the Tegra SOC [3]. The inclusion of for the shader program to fetch. It should be noticed that the sign of caches and performing the depth culling before the pixel processing the coefficient is also included in the texture in order to avoid the help reduce the traffic to system memory and reduce the overall conditional computation (e.g. branches) within the shader program. power dissipation. The transformation of a 2D image is done by applying 1D FFT to Programming API rows and columns consecutively. For each stage, a quad is rendered covering the entire 2D data array. Each stage requires a OpenGL ES 2.0 [8] is the primary graphics programming texture of size to store the pre-computed coefficients. To interface for handheld and embedded devices with a programmable reduce the memory bandwidth between system memory and GPU GPU: a programmable vertex shader and fragment shader for per memory, the coefficients for the real part and the imaginary part are vertex position and per pixel calculation respectively. Other stages in stored in the two channels of the texture. Another texture storing the the rendering pipeline, such as rasterization and culling, remain as fetch indices is required. Although it could be combined with fixed functions. OpenGL ES 2.0 is a subset of the widespread coefficient texture, such implementation is wasteful because indices adopted OpenGL standard used in desktop systems and game require a lower resolution texture than that required for coefficients. consoles. This subset removes some redundancy from the OpenGL Therefore, we allocate a texture with a lower resolution for fetch API. For example, if multiple methods can perform the same indices to reduce the memory bandwidth. The iterative processing is operation, the most useful method is adopted and other redundant implemented by multiple rendering passes, with a floating point methods are removed. An example is that only vertex arrays are used framebuffer object (FBO) (Chapter 12 in [11]) to store the to specify geometry in OpenGL ES 2.0, whereas in OpenGL an intermediate rendering results. application can also use the immediate mode and the display lists in addition to the vertex array to specify geometry. There are also new features introduced to address specific constraints of handheld devices. For example, to reduce the power consumption and increase the performance of the shaders, precision qualifiers were introduced to the shading language: lowp (10-bit fixed point format in the range [-2,2), with a precision of 1/256), mediump (16-bit floating point values in the range [-65520, 65520]), and highp (32-bit floating point variables) [11]. To utilize a mobile GPU as a general-purpose accelerator, programmers have to map the algorithms to the graphics operations, and write the shader programs to configure the vertex and fragment Figure 2. Processing flow of a 8-sample 1D FFT shaders. However, the graphics APIs expose little controllability to Comparison With Mobile CPU the low-level hardware, and hence makes it less flexible to use the GPU for general-purpose computing. For example, the graphics The measured execution time and measured power consumption APIs in the current versions of OpenGL ES do not have the “scatter” of computing 2D complex FFT of various sizes on the Tegra CPU operation (i.e. write to an arbitrary memory location), or thread-level and GPU are shown in the first and second rows respectively of synchronization. The commonly used high-level APIs for a desktop Table 1. The listed execution time is the average time of computing environment, such as CUDA [10] and OpenCL [9], are not supported FFT and IFFT 50 times. For FFT, the GPU is 3x faster in the embedded platform yet. and consumes 8% more power than the CPU (1 second vs. 3.1 seconds, and 4.1 watts vs. 3.7 watts). The slightly higher power when using GPU is because the CPU is not idle when the GPU is Algorithm Optimize Optimize Software

Architecture

SOC Hardware MATLAB OpenCV Some Insights for Mobile CV

• Very difficult to write the fastest code. • When you are prototyping an idea you should not worry about this, but you have to be aware of where bottle necks can occur. • This is what you will learn in this course. • Highest performance in general is non-portable. • If you want to get the most out of your system it is good to go deep. • However, options like OpenCV are good when you need to build something quickly that works. • To build good computer vision apps you need to know them algorithmically. • Simply knowing how to write fast code is not enough. • You need to also understand computer vision algorithmically. • OpenCV can be dangerous here.

Some insights taken from Markus Püschel’s lectures on “How to Write fast Numerical Code”. Today

• Insights into Mobile Computer Vision. • Extremely Brief Intro to Objective C. • Using OpenCV in Xcode. Objective C

• Developed in the early 80s, selected by NeXT as its main language from which OS X and iOS are derived. • Designed as an object-oriented extension to the C language, is based on message passing. • Objective C is a thin layer on top of C. • source code have .m filenames, • header/interface files have a .h extension, and • Objective-C++ files are denoted with a .mm file extension. • We will be using Objective-C++ files in most of our work in this course. • Mainly used to interface with the UI. Objective C versus SWIFT

• Goal in this course is not for you to become experts in either Objective C or SWIFT. • However, you must have some degree of competency in order to build and run an iOS App.

vs. SWIFT

• Essentially “Objective C without the C”. • SWIFT does not expose pointers and other unsafe accessors.

• Not going to be using it much in this course, most iOS computer vision tutorials and books still in Objective C. Drawbacks to SWIFT

• SWIFT is a language undergoing substantial amounts of change still evolving.

• Most vision algorithms are derived from C (e.g. C++) so Objective C is a closer fit.

• Objective C can have C & C++ integrated directly. Develop Your App in the Workspace Window

Perform your core development tasks in the Xcode workspace window, your primary interface for creating and managing projects. A project is the main unit of development in Xcode. It includes all the elements needed to build your app, framework, plug-in, or other software product. It also maintains the relationships between those elements. For more detail on projects, see A Project Is a Repository of Files and Resources for Building Apps (page 31).

The workspace window automatically adapts itself to the task at hand, and you can further configure the window to fit your work style. You can opXcodeen as many wor Layoutkspace windows as you need. The components of the workspace window are shown in the following figure.

(Taken from “Xcode Overview”. ) The workspace window always includes the editor area. When you select a file in your project, its contents appear in the editor area, where Xcode opens the file in an appropriate editor. For example, in the figure above, the editor area contains AdventureScene.swift, a swift code file that is selected in the Navigator area on the left of the workspace window.

2014-10-20 | Copyright © 2014 Apple Inc. All Rights Reserved. 19 Objective-C Cheat Sheet and Quick Reference Class Header (.h) Creating an Object What is a Property? ! ! #import "AnyHeaderFile.h" ClassName * myObject = 1) Automatically defines a private instance variable: [[ClassName alloc] init]; type _propertyName; @interface ClassName : SuperClass 2) Automatically creates a getter and setter: // define public properties Calling a Method // define public methods ! - (type)propertyName; [myObject doIt]; - (void)setPropertyName:(type)name; [myObject doItWithA:a]; @end Using _propertyName uses the private instance [myObject doItWithA:a b:b]; variable directly. Using self.propertyName uses

(TakenClass from Ray Wenderlinch’s Objective C CheetImplementation Sheet) (.m) the getter/setter. ! Declaring Variables #import "YourClassName.h" Custom Initializer Example type myVariable; !

@interface ClassName () - (id)initWithParam:(type)param { // define private properties Variable types if ((self = [super init])) { // define private methods int 1, 2, 500, 10000 _propertyName = param; @end float 1.5, 3.14, 578.234 } double return self; @implementation ClassName { BOOL YES, NO } // define private instance variables ClassName * NSString *, NSArray *, etc. ! } id Can hold reference to any object ! NSString Quick Examples // implement methods ! Defining Properties NSString *personOne = @"Ray"; @end ! NSString *personTwo = @"Shawn"; @property (attribute1, attribute2) NSString *combinedString = type propertyName; [NSString stringWithFormat: Defining Methods @"%@: Hello, %@!", ! strong Adds reference to keep object alive personOne, personTwo]; - (type)doIt; weak Object can disappear, become nil NSLog(@"%@", combinedString); - (type)doItWithA:(type)a; assign Normal assign, no reference NSString *tipString = @"24.99"; - (type)doItWithA:(type)a copy Make copy on assign float tipFloat = [tipString floatValue]; b:(type)b; nonatomic Make not threadsafe, increase perf

readwrite Create getter&setter (default) NSArray Quick Examples readonly Create just getter Implementing Methods ! ! ! NSMutableArray *array = - (type)doItWithA:(type)a Using Properties [@[person1, person2] mutableCopy]; b:(type)b { [array addObject:@"Waldo"]; // Do something with a and b... [myObject setPropertyName:a]; NSLog(@"%d items!", [array count]); return retVal; for (NSString *person in array) { } myObject.propertyName = a; // alt NSLog(@"Person: %@", person); a = [myObject propertyName]; } NSString *waldo = array[2]; a = myObject.propertyName; // alt !Source: raywenderlich.com. Visit for more iOS resources and tutorials! Version 1.5. Copyright 2013 Ray Wenderlich. All rights reserved. Objective-C Cheat Sheet and Quick Reference Class Header (.h) Creating an Object What is a Property? ! ! #import "AnyHeaderFile.h" ClassName * myObject = 1) Automatically defines a private instance variable: [[ClassName alloc] init]; type _propertyName; @interface ClassName : SuperClass 2) Automatically creates a getter and setter: // define public properties Calling a Method // define public methods ! - (type)propertyName; [myObject doIt]; - (void)setPropertyName:(type)name; [myObject doItWithA:a]; @end Using _propertyName uses the private instance [myObject doItWithA:a b:b]; variable directly. Using self.propertyName uses

Class Implementation (.m) the getter/setter. ! Declaring Variables #import "YourClassName.h" Custom Initializer Example type myVariable; !

@interface ClassName () - (id)initWithParam:(type)param { // define private properties Variable types if ((self = [super init])) { // define private methods int 1, 2, 500, 10000 _propertyName = param; @end float 1.5, 3.14, 578.234 } double return self; @implementation ClassName { BOOL YES, NO } // define private instance variables ClassName * NSString *, NSArray *, etc. ! } id Can hold reference to any object ! NSString Quick Examples // implement methods ! Defining Properties NSString *personOne = @"Ray"; @end ! NSString *personTwo = @"Shawn"; (Taken from Ray Wenderlinch’s Objective C Cheet Sheet) @property (attribute1, attribute2) NSString *combinedString = type propertyName; [NSString stringWithFormat: Defining Methods @"%@: Hello, %@!", ! strong Adds reference to keep object alive personOne, personTwo]; - (type)doIt; weak Object can disappear, become nil NSLog(@"%@", combinedString); - (type)doItWithA:(type)a; assign Normal assign, no reference NSString *tipString = @"24.99"; - (type)doItWithA:(type)a copy Make copy on assign float tipFloat = [tipString floatValue]; b:(type)b; nonatomic Make not threadsafe, increase perf

Class Implementation (.m) the getter/setter. ! Declaring Variables #import "YourClassName.h" Custom Initializer Example type myVariable; !

readwrite Create getter&setter (default) NSArray Quick Examples readonly Create just getter Implementing Methods ! ! ! NSMutableArray *array = - (type)doItWithA:(type)a Using Properties [@[person1, person2] mutableCopy]; b:(type)b { [array addObject:@"Waldo"]; // Do something with a and b... [myObject setPropertyName:a]; NSLog(@"%d items!", [array count]); return retVal; for (NSString *person in array) { } myObject.propertyName = a; // alt NSLog(@"Person: %@", person); a = [myObject propertyName]; } (Taken from Ray Wenderlinch’s Objective C Cheet Sheet) NSString *waldo = array[2]; a = myObject.propertyName; // alt !Source: raywenderlich.com. Visit for more iOS resources and tutorials! Version 1.5. Copyright 2013 Ray Wenderlich. All rights reserved. Objective-C Cheat Sheet and Quick Reference Class Header (.h) Creating an Object What is a Property? ! ! #import "AnyHeaderFile.h" ClassName * myObject = 1) Automatically defines a private instance variable: [[ClassName alloc] init]; type _propertyName; @interface ClassName : SuperClass 2) Automatically creates a getter and setter: // define public properties Calling a Method // define public methods ! - (type)propertyName; [myObject doIt]; - (void)setPropertyName:(type)name; [myObject doItWithA:a]; @end Using _propertyName uses the private instance [myObject doItWithA:a b:b]; variable directly. Using self.propertyName uses

Class Implementation (.m) the getter/setter. ! (Taken fromDeclaring Ray Wenderlinch’s Objective C Cheet Sheet) Variables #import "YourClassName.h" Custom Initializer Example type myVariable; !

• On your browser please go to the address, https://github.com/slucey-cs-cmu-edu/Intro_iOS_Lena

• Or again, you can type from the command line.

$ git clone https://github.com/slucey-cs-cmu-edu/Intro_iOS_Lena.git (Taken from ViewController.mm file within the Detect_Lena project described on the previous slide. ) UIImage Class

• High-level way in Objective C for working with images. • Relatively simple to use, you need to use UIImageToMat function convert between OpenCV’s Mat class. • Be careful as function entail’s a memory copy, so frequent calls will effect performance. • Be mindful of how color channels are ordered, • 8 bits per color channel. • Channel ordering in OpenCV’s Mat class is BGR. • UIImage’s ordering is RGB. Today

• Insights into Mobile Computer Vision. • Extremely Brief Intro to Objective C. • Using OpenCV in Xcode. Caution!

• Danger with OpenCV is that it allows you to do a lot with very little understanding for what is going on. • It is also assumed that you know C++ going forward. OpenCV as a Framework??

• Frameworks are intended to simplify the process of handling dependencies. • They encapsulate headers, and binary files, so you do not need to handle them manually. • Framework is just a specially structured folder. • All dependencies are handled automatically. Using OpenCV in Xcode

• On your browser please go to the address, https://github.com/slucey-cs-cmu-edu/Intro_iOS_OpenCV

• Or again, you can type from the command line.

$ git clone https://github.com/slucey-cs-cmu-edu/Intro_iOS_OpenCV.git OpenCV then and now…. History

Willow 2.4.5 NVIDIA Garage 2.2 2.4 2.3 Intel 2.1 1.1 2.0 1.0 Itseez

• Professionally maintained by Itseez • GSoC, corporate contributions

• TakenContributors from OpenCV 3.0 latest news and the fromroadmap. all around the world FunctionalityWhat can OpenCVoverview do?

Image Processing

Transformations Edges, Robust Segmentation Filters contours features Video, Stereo, 3D

Calibration Pose Optical Flow Detection and Depth estimation recognition

Taken from OpenCV 3.0 latest news and the roadmap. OpenCV 3.0

Migration is relatively smooth from 2.4 • Mostly cleanings – Refined C++ API – Use cv::Algorithm everywhere • API changes – C API will be marked as deprecated – Old Python API will be deprecated – Monstrous modules will be split into micromodules – Extra modules