CS 332 Programming Language Concepts

Lecture 12 – Alternative Language Examples (Digital Media Pipeline Processing with Halide)

April 7, 2017  Sam Siewert Reminders Exercise #4, Turned in, Grading in Progress

Exercise #5

Exercise #6 is your Final Report

Exam #2, Last Day of Class, April 27, Review Tues Before

Final Exam – Your Presentation – Team or Individual Analysis or Mini-Alternate Programming Language – Session #1 (12:30-2:30PM 4/29) – Session #2 (7:30-9:30PM 4/29)

 Sam Siewert 2 Major Concepts – Post Production Digital Video Frame Processing Pipeline – Movie! 1. Acquire, Decode, or Read Frame Files 2. Apply Transforms – Simple (Brightness), Convolution (Edge Enhancement), Color Edit, … 3. Add CGI – Generate Computer Generated Imagery with Rendering 4. Composite Frame with Alpha Blending or Green Screen Human Characters into CGI Background 5. Encode Frames into MPEG

Linux Tools Used Today – ffmpeg a.k.a avconv, OpenCV, Pixie (NC- Renderman), Unity 3D, Python Scripting to Coordinate file-to-tool-to- file RenderMan NC Renderman, (Scene Unity, Other CG Description)

MPEG Raw Composite ffmpeg Frames OpenCV Frames ffmpeg

 Sam Siewert 3 Major Concepts – Video Analytics Computer Vision Pipeline – Find Bad Guy! 1. Acquire, Decode, or Read Frame Files 2. Apply Transforms – Simple (Brightness), Convolution (Edge Enhancement), False Color, Segment, 3-D Correspondence, SIFT, AdaBoost 3. Match to Recognition Database (Facial) 4. Present to User

Linux Tools Used Today – ffmpeg a.k.a avconv, OpenCV, Python Scripting to Coordinate file-to-tool-to-file

MPEG ffmpeg Raw Frames OpenCV Multi- MPEG ffmpeg core

 Sam Siewert 4 Observations APIs Support Building Tools Many Tools Lead to File to File Processing Pipelines File to File is Much Slower than Buffer to Buffer However, File to File Scales with Clusters! Buffer to Buffer Scales with Clusters With MPI or Shared Memory Multi-Core Could Scripting Coordination of File to File Be Replaced with a Language? Could the Language Optimize Use of Buffers (Locality) and Parallel and Vector Processing Hardware Features? Frees Programmer to Explore Options Halide – Potential Option

 Sam Siewert 5 Brighten, Contrast Transform {P} = clamp[ {Q}*alpha + beta] – Very Simple – Scale Each Pixel – Brightness Increase/Decrease – Bias Each Pixel – Contrast Increase/Decrease – Make Sure Result Does Not Exceed Saturation Compare C++ OpenCV API and Halide C++ Extension

 Sam Siewert 6 C Code – Is It Ugly? Over-specified?

#define PIXIDX ((i*col*chan)+(j*chan)+k) #define SAT (255) void main(int argc, char *argv[]) { char header[512]; unsigned char img[640*480*3], newimg[640*480*3]; int bufflen, hdrlen; unsigned row=0, col=0, chan=0, pix; int i, j, k; double alpha=1.25; unsigned char beta=25;

header[0]='\0'; readppm(img, &bufflen, header, &hdrlen, &row, &col, &chan, argv[1]);

for(i=0; i < row; i++) for(j=0; j < col; j++) for(k=0; k < chan; k++) { newimg[PIXIDX] = (pix=(unsigned)((img[PIXIDX])*alpha)+beta) > SAT ? SAT : pix; }

writeppm(newimg, bufflen, header, hdrlen, "brighter.ppm"); }

 Sam Siewert 7 OpenCV C++ API – Brighten/Contrast #include #include #include using namespace cv; using namespace std; double alpha=1.0; int beta=10; /* contrast and brightness control */ int main( int argc, char** argv ) { Mat image = imread( argv[1] ); // read in image file Mat new_image = Mat::zeros( image.size(), image.type() ); std::cout<<"* Enter alpha brighten factor [1.0-3.0]: ";std::cin>>alpha; std::cout<<"* Enter beta contrast increase value [0-100]: "; std::cin>>beta; Loop through Rows, // Do the operation new_image(i,j) = alpha*image(i,j) + beta for( int y = 0; y < image.rows; y++ ) Columns, and Color { Channels to Apply for( int x = 0; x < image.cols; x++ ) { Dereference Pixel at X,Y for( int c = 0; c < 3; c++ ) for each Color new_image.at(y,x)[c] = saturate_cast( alpha*( image.at(y,x)[c] ) + beta ); } }

namedWindow("Original Image", 1); namedWindow("New Image", 1); imshow("Original Image", image); imshow("New Image", new_image); waitKey(); return 0; }  Sam Siewert 8 Halide – Brighten/Contrast

// Adapted from Halide tutorial lesson 2. #include using Halide::Image; #include "../apps/support/image_io.h" int main(int argc, char **argv) { Halide::Image input = load(argv[1]); Halide::Func brighter; Halide::Var x, y, c; Halide::Expr value = input(x, y, c); No loop specification, just transform function, so Halide can optimize locality, vector value = Halide::cast(value); processing and multi-core features. value = value * 1.5f; value = Halide::min(value, 255.0f); value = Halide::cast(value);

brighter(x, y, c) = value;

Halide::Image output = brighter.realize(input.width(), input.height(), input.channels());

save(output, "brighter.png");

return 0; }

 Sam Siewert 9 Compiling Halide Code

Extension to g++

E.g. build line from Halide/tutorial:

g++ brighten.cpp -I ../include -L ../bin -lHalide -lpthread -ldl -lpng -o brighten

 Sam Siewert 10 Halide Overview http://halide-lang.org/ – Decouples Pipeline Specification (Buffer-to-Buffer) and Optimization – Built on LLVM and Clang – Extensions to C/C++ and Embedding – Extension to C++ with Use of Operator Overloading

Watch Video Overview on Web

Alternatives 1. APIs – OpenCV, OpenNI (for C++, Python, Java) 2. Interactive VHLLs – MATLAB CV Toolbox, Mathematica 3. IDL – Interactive Data Language

Discuss – API vs. Specialize Language?

 Sam Siewert 11 Assignment #5 & 6 Help Data Structures and Algorithms You Already Know – Pick One You Implemented and Tested in an Imperative Procedural Language (Graphs, Trees, Lists, Numerical) – Re-implement in Functional or Declarative and Compare

Pick New Application Domain (Image Processing, Cryptography, Simulation, Mathematics, AI) – Choose Interesting but Simple Algorithm (e.g. Image Blur, Sharpen, Brightness & Contrast, Gamma Correction, …) – Implement in Both Imperative Procedural and Alternative PL (Functional, Declarative, Specialized Scripting (R, Halide), or Anything other than OO Imperative)

Present Comparison to Class in Person, and/or Capture and Record in PowerPoint [Slides Due Same Day as Exam]

 Sam Siewert 12 Sensor Fusion

E.g. Algorithm Exploration with Images

 Sam Siewert

13 Sensor Fusion - Basic Concepts Visible Image Includes 3 Wavelengths – Red, 650 nanometers – Green, 510 nanometers – Blue, 475 nanometers

Add Thermal LWIR Imaging – Thermal Image Intensity –False Color? – Match Resolution and Overlay with Visible? http://landsat.usgs.gov/gallery.php - Rodeo Chediski Fires, AZ Many Applications for LWIR Multi- Spectral – Cold Spots and Hot Spots - Ice, Fire Hazards – Vegetation, Soil Moisture – Animals and People – E.g. SEEK Imager, 206x156, 7200 to 13000 nanometers

USGS Landsat Images  Sam Siewert 14 Research vs. Development Vision, Objectives, Goals NSF (OMB) – Acquisition of Knowledge "Research" is defined as a systematic study – Requirements Less Emphasized, To Be directed toward fuller scientific knowledge or Determined understanding of the subject studied. Research is classified as either basic or applied, according – Proof-of-Concept and Prototypes to the objectives of the investigator. Emphasized – Repeatability of Results, Both Expected "Development" is the systematic use of and Unexpected knowledge and understanding gained from research directed toward the production of useful materials, devices, systems, or Experimental Research methods, including design and development – Emphasis on Validation & Verification of prototypes and processes. – Design of Experiments (PL Compare) DoD – Data Analysis

Theoretical Research – Models – Mathematical, Logical, Statistical, Probability, Simulation – Analysis of Problems and Domains

 Sam Siewert https://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1194/MR1194.appb.pdf 15 Simple View of Research [R&D] Acquisition of Knowledge – Theory or Experiment Defined by Existing Knowledge and Expectations

Development Innovation / Adv. Tech Applied Basic • Well-known (standard) • Recombination of well-known • Expectations • Nobody knows • Specified application • Specified application • Misconceptions • No application • Assumptions • Specific Problem or Class

Spectrum

 Sam Siewert 16 Learning Objectives Beyond Imperative Procedural Problem – Proliferation of PLs and Selection – “Which is Best for My Problem?”, “What Defines Best?” Hypothesis for Alternative PL Selection Collaboration (Alternate PL, Primary PL) –Declarative Functional – Lisp, Haskell, Scheme Dataflow – Verilog, VHDL, Halide Logic - Prolog Template – XSLT, XML – Imperative Scripted (Interpreted) – SQL, R, MATLAB, Python Procedural – C, Fortran Strong Typed OO – Java, Smalltalk, Ada 83 or 95 – Other (Propose to Instructor) Literature Review of Alternate PL and Applications Application Design and Prototype A-PL, P-PL Evaluation and Experiment Design Presentation of Comparative Results

 Sam Siewert 17 Scaffolding Assignments #1-#4 Practice – Imperative Procedural – Alternative Paradigm Compare

Assignment #5 – Propose PL Comparison or Custom PL – Explore Design – Follow Interest in Alternative Approach – Choose One: 1. Design the Alternative for Proof-of-Concept Interpreter [Compiler] 2. Choose Imperative Procedural PL [OO or Modular/Structured] and Alternate Paradigm [Functional, Declarative Logic, Dataflow, Other?]

Assignment #6 – Propose PL Comparison or Custom PL – Side by Side Application Comparison Implementations – Report on Results

Final Oral Exam – Present Results to Class

 Sam Siewert 18