Introduction to Optimization Theory and Applications

Introduction to Optimization Theory and Applications: Course Overview and Linear Programming Overview WIFI: SSID Student USERname ucsc-guest 1 Password EnrollNow! James G. Shanahan 1Independent Consultant

EMAIL: James_DOT_Shanahan_AT_gmail_DOT_com

TIM 206 (30155) Introduction to Optimization Theory and Applications Thursday, January 10, 2013 Lecture 01

TIM 206206 (30155 (30155 ) )Introduction Introduction to Optimizationto Optimization Theory Theory and Applications, and Applications, Winter 2013Winter © 2013 James© 2013 G. James Shanahan G. Shanahan 1 Course Info

• http://courses.soe.ucsc.edu/courses/tim206

• https://courses.soe.ucsc.edu/courses/tim206/ Winter13/01 – Schedule – Exam during

Final Exam (closed book): Week 11 of the Quarter

Performance Evaluation: Homework 30% Midterm 20% (Week 6 of the Quarter) Class participation 20% Final Exam 30% (Week 11 of the Quarter)

• Background – Industry/Academia – Major – Programming experience • Expectations from taking TIM206

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 5 Brief Bio James G. Shanahan • 25 years in the fields of data mining and operations research – Principal and Founder, Boutique Data Consultancy • Clients include: Adobe, Digg, Adobe, SearchMe, AT&T, SkyGrid, TapJoy – Affiliated with University of California Santa Cruz (UCSC, ISM250,251,209) – Chief Scientist, Turn Inc. (A CPX ad network, DSP) – Principal Scientist, Clairvoyance Corp (CMU spinoff; sister lab to JRC) – Co-founder of Document Souls (an anticipatory information system) – Research Scientist, Xerox Research – Research Engineer, Mitsubishi Group – PhD in machine learning (1998), University of Bristol, UK; B.Sc. Comp. Science (1989), Uni. of Limerick, Ireland • Areas of expertise include: – Local search, web search, online advertising, machine learning, ranking, user modeling, statistics, social networks

• The Authors retains all rights, including copyrights and distribution rights. • No publication or further distribution in full or in part permitted without explicit written permission from the author • Living vicariously!

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 8 Click here https://courses.soe.ucsc.edu/courses/tim206/Winter13/01 Course Description • Essentially, every problem in business, computer science and engineering can be formulated as the optimization of some function under some set of constraints. – This universal reduction automatically suggests that such optimization tasks are intractable. Fortunately, most real world problems have special structure, such as convexity, locality, decomposability or submodularity. – These properties allow us to formulate optimization problems that can often be solved efficiently. – As such operations research is a quantitative discipline that deals with the application of advanced analytical methods to help make better decisions. – The terms management science and decision science are sometimes used as more modern-sounding synonyms. Employing techniques from other mathematical sciences, such as mathematical modeling, statistical analysis, and mathematical optimization, operations research arrives at optimal or near-optimal solutions to complex decision-making problems.

• Techniques from operations research are leveraged within many industries, in both online and offline modes, and are responsible for both batch and real-time decision-making involving trillions of dollars ($10^12) annually worldwide. For example: – Finance: How Wall Street manages stock portfolios? – Digital Advertising: How publishers optimize advertiser revenue online? – Logistics: How Fedex ships billions of packages annually across 220 countries? – Social networks: How to suggest people you may know? – Web search: How to rank web pages with out queries? – Healthcare: How do hospitals run? From organ allocation to schedule generation? Cancer treatment TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 10 Some applications of operations research

Other Areas • Digital Advertising • Web search

• More concretely, this class serves as a first graduate course in optimization with an emphasis on theory, intuition, and problem solving in management and engineering. • This class will focus on problem formulation, software technologies and analytical methods for optimization serving as an introduction to a wide variety of optimization problems and techniques including: – linear programming, nonlinear programming, dynamic programming, network flows, integer programming, heuristic approaches, Markov chains, game theory, and decision analysis.

Date Week Lecture Topics 1/10/12 1 Introduction and Linear Programming Intro, intro R part 1 1/17/12 2 Simplex Method and Sensitivity and Duality, intro R part 2 Other LP Methods, LP Applications, Transportation, Assignment and 1/24/12 3 Network Optimization Problems 1/31/12 4 Nonlinear Unconstrained, Gradient descent etc. 2/7/12 5 Nonlinear Constrained, convex optimization, machine learning 2/14/12 6 Exam, (and Dynamic Programming) Integer Programming, Branch and Bound, Multiple choice Knapsack Problem (MCKP), [Genetic Algorithms] Optimal Bid Determination in 2/21/12 7 Sponsored Search as an MCKP 2/28/12 8 Metaheuristics and Markov Chains 3/7/12 9 Game Theory, Decision Analysis and Queueing Theory 3/14/12 10 Inventory Theory and Simulation Wk of 3/18 11 Final Exam: Closed book final Exam will be during this week

• Describe core optimization techniques – e.g., linear programming, nonlinear programming, convex optimization, dynamic programming, genetic algorithms, Markov processes and more – Discuss the pros and cons of these approaches – Explain how to use these approaches and solve problems in the fields of digital advertising and marketing, healthcare, logistics and planning, airline industry, ecommerce • Identify real-world problems that could be solved using optimization techniques • Solve real-world optimization problems in the above fields using solvers, and other optimization software (and possibly implement a couple of the above algorithms from scratch; this is not a requirement though) • Optional: gain a proficiency optimization problem solving in Excel and in R (R is optional but a great skill to develop!).

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 14 TIM 206 Background • This course is introductory and will develop the fundamental operations research models and techniques in the context of real world applications and examples. – We will develop the techniques systematically and sequentially, but will move back-and-forth the different real world applications and domain areas. – The subsequent courses, such as ISM 250 and 251, will expand on the techniques and domain areas such as web mining, computational advertising and online marketing, social networks and relational learning, reinforcement learning, constrained optimization, and also explore significant projects, including possibly with industry. • Prerequisites: Students should have some background in probability, statistics, calculus, and linear algebra, though most of the concepts from these areas, which are core to optimization, will be reviewed for completeness during the course. TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 15 TIM 206 is timely!

• TIM 206 (Survey class) – Precursor to TIM 250 (Stochastic optimization) which goes deeper into NLP, MDPs, Convex optimization, Machine learning as optimization) and TIM 251 (scalable, distributed decision support systems)

• …. with applications in digital advertising and online marketing, healthcare, airlines, financial • Timely: – Growing flood of online data, Budding industries (e.g., digital advertising, healthcare, financial) – Computational power is available (PC, Cloud computing, Hadoop) – Progress in algorithms and theory and applications

• The old days were about asking, ‘What is the biggest, smallest, and average?’ ” says Michael Olson, CEO of startup Cloudera. “Today it’s, ‘What do you like? Who do you know?’ It’s answering these complex questions.” • Personalization through optimization – Healthcare; finance; marketing and advertising

• In the old days: – A retailer such as Macy’s (M) that once pored over last season’s sales information could shift to looking instantly at how an e-mail coupon for women’s shoes played out in different regions.

Hadoop, Java, Python, R Hacking Skills Statistics, Optimization Theory, Social Network Analytics, Geo- Informational Math, OR Science Digital Advertising & Domain Statistics Marketing, Expertise Probability Econometrics, Web Search, Theory Healthcare, Finance

• It’s not going too far to say that data analytics and optimization has even gotten hip. – The San Francisco offices of startup Splunk have all the of- the-moment accoutrements you’d find at Twitter or Zynga. • The engineers work in what amounts to a giant living room with pinball machines, foosball tables, and Hello Kitty-themed cubes. • Weekday parties often break out—during a recent visit, it was Mexican fiesta. Employees were wearing sombreros and fake moustaches while a dude near the tequila bar played the bongos.

DEMAND SUPPLY Advertiser wishes to reach consumers

Ads Consumers • Sponsored Search A A • Contextual P d d • Display v u e A • Classified r g • Email b t e • Social i n l s c • Mobile i e y s r h TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 24 e r Sponsored Search, Display Volumes

DEMAND SUPPLY Advertiser wishes to reach consumers • Sponsored Search • 208 Billion in US (2009) Ads • Display Consumers • 4 Trillion in US (2009) A A [JP Morgan 2010] P d d v u e A r g b t e i n l s c i e y s r h TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 25 e r From Mad Men To Wall Street and beyond! BannerBanner Click+Data Personal Social 2007

Data

Personalization

Advertisers still in broadcast mode

1st Generation 2nd Generation 3rd Generation

North

1 Paid Ad East 2 Paid Ads 3 (SEM)

Organic results 4 (SEO) 5

Ads are clearly distinguishable from the actual search results and they rotate

Click

Impressions à Clicks à Transactions

CPC ß RPC ß Revenue ctr = clicks / impressions rpc = revenue /clicks cost = clicks * cpc profit = revenue - cost cpa = cost / conversions margin = profit / cost

From To

Adv. Adv. Sports Adv. Adm PredictedSupply PageViews Sports, Sports, 18-25, San Francisco 18-25

AudienceZone 1 (e.g., Sports, Volume11 Volumeij Volumeij Volumeij 35 18-25, San Francisco) Revenue11 Revenueij Revenueij Revenueij AudienceZone 2 … … … … 50

AudienceZone 3 Volumeij Volumeij Volumeij Volumeij 15 …….. Revenueij Revenueij Revenueij Revenueij Demand Contracted 45 20 30 5 PageViews

Use Linear Programming to generate suggested Ad Sales schedule; This schedule shows how much volume is available in each audience zone and how it should be allocated to advertiser targeting criteria to maximize publisher revenue. E.g., This schedule shows how much of Audience zone 1 should be sold to advertisers who want to target sport people; this optimized volume, Volume11 (based on a global picture) will generate a corresponding Revenue11

Social Network

WebSearch

The optimal portfolios plotted along the curve have the highest expected return possible for the given amount of risk. TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 33 LP Healthcare Example: Design of External Beam Radiation Therapy • MARY has just been diagnosed as having a cancer at a fairly advanced stage. Specifically, she has a large malignant tumor in the bladder area (a “ whole bladder lesion” ). • The goal of the design is to select the combination of beams to be used, and the intensity of each one, to generate the best possible dose distribution. • Cross section of Mary’s tumor (viewed from above) – as well as nearby critical tissues to avoid and – the radiation beams being used. – normally dozens of possible beams must be considered • Location of her tumor is in a tricky spot.

• Because of the need to carefully balance all these factors, the design of radiation therapy is a very delicate process. • The goal of the design – is to select the combination of beams to be used, and – the intensity of each one, to generate the best possible dose distribution. – (The dose strength at any point in the body is measured in units called kilorads. ) • Once the treatment design has been developed, it is administered in many installments, spread over several weeks.

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 35 F(Beam intensity) = Absorption IsoDose Map • The contour lines represent the dose strength as a percentage of the dose strength at the entry point. A fine grid then is placed over the isodose map. By summing the radiation absorbed in the squares containing each type of tissue, the average dose that is absorbed by the tumor, healthy anatomy, and critical tissues can be calculated. (measured in Kilorads)

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 36 F(Beam intensity) = Absorption IsoDose Map • For any proposed beam of given intensity, the analysis of what the resulting radiation absorption by various parts of the body would be requires a complicated process. • In brief, based on careful anatomical analysis, the energy distribution within the twodimensional cross section of the tissue can be plotted on an isodose map, where the contour lines represent the dose strength as a percentage of the dose strength at the entry point. A fine grid then is placed over the isodose map. By summing the radiation absorbed in the squares containing each type of tissue, the average dose that is absorbed by the tumor, healthy anatomy, and critical tissues can be calculated. With more than one beam (administered sequentially), the radiation absorption is additive (cross product terms).

• ..

• IRhythm makes a type of oversize, plastic band-aid called the Zio Patch that helps doctors detect cardiac problems before they become fatal. • Patients affix the Zio Patch to their chests for two weeks to measure their heart activity. The patients then mail the devices back to IRhythm’s offices, where a technician feeds the information into Amazon’s cloud computing service. • Patients typically wear rivals’ much chunkier devices for just a couple of days and remove them when they sleep or shower—which happen to be when heart abnormalities often manifest. The upside of the waterproof Zio Patch is the length of time that people wear it—but 14 days is a whole lot of data.

• Personal devices (with GPS’ and accelerometers) – Earphones; Nike+ (measures and records the distance and pace of a walk or run); asthma inhaler with built-in GPS tracking

• Personal/social services – Mint, Twitter, diets, health, exercise, FaceBook

• These data streams create a huge privacy problem

•

Records events using multiple wearable sensors Provides access to these data at multiple levels of granularity and abstraction, using access mechanism http://www.imrc.kist.re.kr/ wiki/LifeLog basedTIM 206 on (30155 the ) Introductionepisodic to Optimization memory Theory of and human Applications, beings. Winter 2013 © 2013 James G. Shanahan 42 Backend Technology

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 43 3rdi Art Project • A New York University arts, Professor Bilal • A surgically-implanted camera (12/15/2010) – 3rdi Project, has already generated international media attention and anticipation. On Dec. 15 images from the "third eye" in the back of Bilal's head -- a surgically-impanted camera -- will be unveiled in Doha, Qatar as part of the Told/Untold/Retold exhibition that inaugurates the new Arab Museum of Modern Art near Education City, Doha’s intellectual hub. • Transmits one image per minute to a website (www.3rdi.me), displayed a Doha gallery – with the inaugural images to be displayed in a custom- designed room in the Doha gallery. Bilal’s piece will be part of the museum’s new permanent collection, 20 years in the making, including more than 6,000 works by Arab artists from North Africa to the Gulf, from the 1920s to the present day.

• Smartphones 50% share in mid2011 (US) • Tablet computers – Large Format Benefit – Enhanced mobile apps – Total media tablets device market – 28MM in 2011 (ABI, 2010; Barclays Capital, 2010) • IPTV – Play IPTV digital content originating from the iTunes Store, Netflix, YouTube, Flickr, MobileMe or any Mac OS X or Windows computer running iTunes onto an enhanced-definition or high-definition widescreen television – Still early days but • Theatre

Data

Personalization

Advertisers still in broadcast mode

1st Generation 2nd Generation 3rd Generation

• Supervised ML – Constrained and unconstrained optimisation

• Unsupervised ML – Constrained and unconstrained optimisation

• Reinforcement Learning – Markov processes

• An incremental algorithm for SVD was developed by Simon Funk based on a paper by Genevieve Gorrell, in which the SVD is approximated numerically via gradient descent • An approximation rank r is chosen beforehand such that there are r feature vectors for movies and users • The ∑ matrix is left blended into the feature vectors, so only U and V remain • Initial values for these feature vectors are set arbitrarily, i.e. 0:1 • Gradient descent is used to minimize the function for each feature vector along users and movies

Fill in empty cells using U and V factorization

Netflix: 500,000 users, 17,000 movies = 8.5Billion ratings; only 100Million provided 20 million free parameter for Rank-40 SVD Decomposition : 5*10^5 *40 +(40 *17000))

• Alternatively, instead of using a simple linear factor model (or bounded version as above), the inner product between user-specific and item-specific feature vectors is mapped through a nonlinear logistic function: • g(x) = 1/(1+exp(−x)), which bounds the range of the predictions into [0, 1].

Some simple math using a mountain of data can get you 80% of the way!

http://www.businessweek.com/magazine/data-analytics- crunching-the-future-09082011.html

* McKinsey Global Institute 2011

• Course will focus on getting students familiar with core principles in operations researcg • Grounding these principles in both – (1) examples taken primarily form online advertising (a $70 Billion industry), healthcare, Airline industry, etc. – And in (2) example projects and code in R/Excel/IOR Solver. • Each class will be composed of theory, geometry/ code, and problems, thereby informing and inspiring students on how to apply theory to practice.

• Problem solving • Data analysis • Coding up algorithms • Real-world datasets • Evaluations and metrics • Collaboration • Presentation • Teamwork

Final Exam (closed book): Week 11 of the Quarter

Performance Evaluation: Homework 30% Midterm 20% (Week 6 of the Quarter) Class participation 20% Final Exam 30% (Week 11 of the Quarter)

• Background – Industry/Academia – Major – Programming experience • Expectations from taking TIM206

• Socratic Method (both inspiration and information) – participation strongly encouraged (please state your name and affiliation) • Highly interactive and adaptable – Questions welcome!! • Lectures emphasize intuition, rigor and detail – Build on lectures – Background reading will provide more rigor & detail • Action Items – Read suggested books first (and then papers), read/write Wikipedia, watch/make YouTube videos, take other courses, participate in competitions, do internships, network – Prototype, simulate, publish, participate – Classic (core) versus trendy (applications)

• ..

• Introduction and some motivating problems • Linear Algebra Basics Review • Fundamental theorem of LP • Matrix-view and the fundamental insight • Duality • Interior point Algorithm • Transportation Problem • Applying linear programming to online advertising • Summary

• (HL) Hillier and Lieberman: Introduction to Operations Research, McGraw-Hill, 9th Edition. – This book is available for purchase from the UCSC bookstore (in eBook form at a negotiated reduced price of $80 compared with the print version prices of $250; please use ISBN 9781121774254 for purchasing) and comes with very useful software. – There should be copies available in the Library also. • https://highered.mcgraw- hill.com/sites/ 0073376299/ student_view0/

• http://cran.r-project.org/web/views/Optimization.html

• NLP SUMT – R sumt(…) • http://rss.acs.unt.edu/Rdoc/library/clue/html/sumt.html • https://r-forge.r-project.org/scm/viewvc.php/pkg/R/sumt.R? view=markup&root=maxlik

– SUMT examples and good write up • http://www.me.utexas.edu/~jensen/ORMM/supplements/units/nlp_methods/const_opt.pdf • The methods that we describe presently, attempt to approximate a constrained optimization problem with an unconstrained one and then apply standard search techniques to obtain solutions. The approximation is accomplished in the case of penalty methods by adding a term to the objective function that prescribes a high cost for violation of the constraints. In the case of barrier methods, a term is added that favors points in the interior of the feasible region over those near the boundary. For a problem with n variables and m constraints, both approaches work directly in the n-dimensional space of the variables. The discussion that follows emphasizes penalty methods recognizing that barrier methods embody the same principles.

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 70 Course References • R – Practical Regression and Anova using R, http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf ,by JJ Faraway (please download PDF) – John Fox (2010), Sage, An R and S-PLUS Companion to Applied Regression (second edition) • Chapter 1 - Getting Started With R (PDF available) • Chapter 6 - Diagnosing Problems in Linear and Generalized Linear Models – The R book, by Michael J. Crawley, Wiley 2009

## Setup up coordinate system (with x==y aspect ratio): plot(c(-2,3), c(-1,5), type = "n", xlab="x", ylab="y", asp = 1) ## the x- and y-axis, and an integer grid abline(h=0, v=0, col = "gray60") text(1,0, "abline( h = 0 )", col = "gray60", adj = c(0, -.1)) abline(h = -1:5, v = -2:3, col = "lightgray", lty=3) abline(a=1, b=2, col = 2) text(1,3, "abline( 1, 2 )", col=2, adj=c(-.1,-.1))

• OR Courseware is a program called OR Tutor – personal tutor to help you learn the algorithms. It consists of many demonstration examples that display and explain the algorithms in action • OR Courseware includes many interactive routines for executing the algorithms interactively in a convenient spreadsheet format. – The computer does all the routine calculations while you focus on learning and executing the logic of the algorithm. • Microsoft Excel: formulate small OR models in a spreadsheet format. The Excel Solver then is used to solve the models. – Your OR Courseware includes a separate Excel file for nearly every chapter in this book. Each time a chapter presents an example that can be solved using Excel, the complete spreadsheet formulation and solution is given in that chapter’s Excel file. For many of the models in the book, an Excel template is available also. • R is an open source programming language and software environment for statistical computing and graphics. • CPLEX is an elite state-of-the-art software package that is widely used for solving large and challenging OR problems TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 73 On Windows 8 (via Parallels on Mac): Press Command Key for a couple of Seconds to bring up Windows VM

• ..

http://www.java.com/en/ download/index.jsp for IOR Tutor

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 76 TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 77 IORTutorial (interactive) and OR Tutor

1. Define the problem of interest Gather relevant data. 2. Formulate a mathematical model to represent the problem. 3. Build model: 1. Develop a computer-based procedure for deriving solutions to the problem from the model. 4. Debug/evaluate the model and refine it as needed. 5. Build supporting infrastructure for using such a model in production (accessing data, logging data, make decisions, etc.) 6. Deploy (and ABTest).

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 82 6 Steps to OR Modeling in Practice 6 Deploy System in the wild (and AB test) 5 Build supporting infrastructure for models 4 Debug/Evaluate models 3 Derive solutions 2 Formulate solution/model 1 Understand the domain and Systems Modeling is inherently Define problems interactive and iterative

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 83 6 Steps to Data Modeling in Practice 6 Deploy System in the wild (and AB test) 5 Interpret and Evaluate discovered knowledge 4 Modeling: Extract Patterns/Models 3 Feature Engineering 2 Collect requirements, and Data 1 Understand domain, Collect Systems Modeling is inherently requirements & Data interactive and iterative

• In contrast to textbook examples, most practical problems encountered by OR teams are initially described to them in a vague, imprecise way. • Therefore, the first order of business is to study the relevant system and develop a well-defined statement of the problem to be considered. • This includes determining such things as the appropriate objectives, constraints on what can be done, interrelationships between the area to be studied and other areas of the organization, possible alternative courses of action, time limits for making a decision, and so on. • This process of problem definition is a crucial one because it greatly affects how relevant the conclusions of the study will be. It is difficult to extract a “right” answer from the “wrong” problem!

• Beer flow problem (Taps versus Point of Sale) – Problem – Data

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 85 Step 2: Formulate a mathematical model • Formulate a mathematical model to represent the problem. • Mathematical models are idealized representations, that are expressed in terms of mathematical symbols and expressions. • Thus, if there are n related quantifiable decisions to be made, Decision they are represented as decision variables (say, x1 , x2 , . . . , variables xn ) whose respective values are to be determined. • The appropriate measure of performance (e.g., profit) is then expressed as a mathematical function of these decision Objective variables Function – P = 3x1 + 2x2 + 5xn – This function is called the objective function. • Restrictions on variables can be captured as constraints

– 2x1 + x2 + 5xn ≤ 10 • Parameters are the constants – The constants (namely, the coefficients and right-hand sides) in the constraints Parameters and the objective function are called the parameters of the model.

• Use Linear Programming as an example – Define problem – Gather data LP(c, a, b) à x – Formulate model x: Decision variables – Solve c: Objective coefficients a: b: RHS resource constraints Parameters Decision Maximize Z = 3X + 5X 1 2 Variables Subject to: SOLVE X1 = 2, X2 = 6 COMPUTE 1X1 ≤ 4 2X2 ≤ 12 3X1 + 2X2 ≤ 18 TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 87 Determining the parameter values • Determining the appropriate values to assign to the parameters of the model (one value per parameter) is both a critical and a challenging part of the model-building process. • In contrast to textbook problems where the numbers are given to you, determining parameter values for real problems requires gathering relevant data. • Gathering accurate data frequently is difficult. Therefore, the value assigned to a parameter often is, of necessity, only a rough estimate. • Sensitivity analysis helps simulate many possible parameters settings – Because of the uncertainty about the true value of the parameter, it is important to analyze how the solution derived from the model would change (if at all) if the value assigned to the parameter were changed to other plausible values. • E.g., Maximize profit (what if sale price fluctuates; cost of raw materials fluctuates) TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 88 Step 3: Deriving Solutions from the Model

Plug • Apply a computer-based procedure for deriving and solutions to the problem from the mathematical chug model. • Sometimes, in fact, it is a relatively simple step, in which one of the standard algorithms (systematic solution procedures) of OR is applied on a computer by using one of a number of readily available software packages.

10-20% of • For experienced OR practitioners, finding a modeling solution is the fun part, whereas the real work effort comes in the preceding and following steps, including the postoptimality analysis • Course: How to find solutions to various types of math models TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 89 Step 3:From Optimizing to Satisficing

• Course: How to find solutions to various types of math models • Herb Simon coined the term satisficing as a combination of the words satisfactory and optimizing • Satisficing (approximate solutions) – Use Heuristic procedures to find suboptimal solutions – tendency of managers to seek a solution that is “good enough” for the problem at hand. Goals may be set to establish minimum satisfactory levels of performance in various areas, based perhaps on past levels of performance or on what the competition is achieving. If a solution is found that enables all these goals to be met, it is likely to be adopted without further ado. Such is the nature of satisficing. TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 90 Postoptimality Analysis: Sensitive Parameters Step 3: • Analysis done after finding an optimal solution – This analysis also is sometimes referred to as what-if analysis because it involves addressing some questions about what would happen to the optimal solution if different assumptions are made about future conditions. – These questions often are raised by the managers who will be making the ultimate decisions rather than by the OR team • Postoptimality analysis involves conducting sensitivity analysis to determine which parameters of the model are most critical (the “sensitive parameters”) in determining the solution. • The value assigned to a parameter commonly is just an estimate of some quantity (e.g., unit profit) whose exact value will become known only after the solution has been implemented. – Therefore, after the sensitive parameters are identified, special attention is given to estimating each one more closely, or at least its range of likely values. One then seeks a solution that remains a particularly good one for all the various combinations of likely values of the sensitive parameters.

• Developing a large mathematical model is analogous in some ways to developing a large computer program. When the first version of the computer program is completed, it inevitably contains many bugs. • Post-optimality analysis is also key here • Reexamining the definition of the problem and comparing it with the model may help to reveal mistakes. (imagine an exam question) – It is also useful to make sure that all the mathematical expressions are dimensionally consistent in the units used. Additional insight into the validity of the model can sometimes be obtained by varying the values of the parameters and/or the decision variables and checking to see whether the output from the model behaves in a plausible manner. • Retrospective test over held out data (just like ML!) TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 92 Step 5: Build Supporting Infrastructure

• Produce documentation • Model and analysis reports • Solution procedure

• Build supporting infrastructure for using such a model in production (accessing data, logging data, make decisions, etc.)

• Run system in a production setting (e.g., in a data center)

• AB Test new system with incumbent system

• Refine model, repeat step 2-6 if necessary (live performance is not satisfactory)

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 94 6 Steps to OR Modeling in Practice 6 Deploy System in the wild (and AB test) 5 Build supporting infrastructure for models 4 Debug/Evaluate models 3 Derive solutions 2 Formulate solution/model 1 Understand the domain and Systems Modeling is inherently Define problems interactive and iterative

Maximize x1 + 2x2 Subject to x1 + 0 ≤ 2 0 + x2 ≤ 4

What is a good a good upper bound for this LP problem (Primal)?

• Hint: use the fundamental insight and weak and strong duality property (cx ≤ by)

• Solve using the IOR Tutor and provide primal and dual solutions and comment how the dual is solved using LP (i.e., input the dual and solve as an LP). What steps does the LP go thru; describe TIM 206 (30155in ) Introductiondetail to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 97 Solve LP graphical (along with its Dual)

• ..

• What if I said try x = (2, 4) and that I gave a feasible dual of (1, 2) (y1, y2) 2 decision variables in the dual (recall we have just 2 constraints)

• What can we say about these two solutions? • Hint: strong duality property

• What if I said try x = (2, 4) and that I gave a feasible dual of (1, 2) (y1, y2) 2 decision variables in the dual (recall we have just 2 constraints)

• What can we say about these two solutions? • Hint: strong duality property

• What if yb – cx is small……. Good approximation

• Linear programming (LP, or linear optimization) is a mathematical method allocating limited resources among competing activities in a best possible (i.e., optimal ) manner – More precisely, this problem involves selecting the level of certain activities that compete for scarce resources that are necessary to perform those activities. The choice of activity levels then dictates how much of each resource will be consumed by each activity. • Applications – Allocation of production facilities to products; – Allocation of national resources to domestic needs; – Portfolio selection; selection of shipping patterns; agricultural planning; design of radiation therapy; and so on

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 101 Goal LP (and simplex algo.) • The goal of the simplex method is to find the how to use the available resources in the most profitable feasible way

• LP(xij, bi, cj) à xj

• Given xij, bi, cj and determine xj as output

• Sensitivity analysis : vary the known constants,

xij, bi, cj and see how the optimal solution to the

decision variables xj is changed (Z value changes or suddenly becomes feasible)? – Is this new problem that we have to solve from scratch? – Or can we pivot from the existing solution – Or do nothing (as certain conditions are met) TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 102 Linear Programming == Linear Planning

• Linear programming uses a mathematical model to describe the problem of concern. • The adjective linear means that all the mathematical functions in this model are required to be linear functions. • The word programming does not refer here to computer programming; rather, it is essentially a synonym for planning. • Thus, linear programming involves the planning of activities to obtain an optimal result. – a result that reaches the specified goal best (according to the mathematical model) among all feasible alternatives

• Remarkably efficient solution procedures exist (polynomial time) to solve linear programming problems…. Millions of decision variables • Thousands of successful applications • Workhorse of many industries

• Devote three to four lectures to LP (3 now and one later in the quarter (guest lecture))

• Introduction and background material • Properties of LPs • Simplex method • Sensitivity and Duality • Alternative Methods for solving

• The founders of the subject are Leonid Kantorovich, 1939 a Russian mathematician who developed linear programming problems in 1939, George B. Dantzig, 1947 who published the simplex method in 1947, John von Neumann, who developed the theory of the duality in the same year. • The linear programming problem was first shown to be solvable in polynomial time by Leonid Khachiyan in 1979, but a larger theoretical 1984 and practical breakthrough in the field came in 1984 when Narendra Karmarkar introduced a new interior point method for solving linear programming problems.

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 107 What is Linear Programming • Linear programming (LP) = – Linear Algebra + inequalities + optimization (minimize or maximize) • LP is a technique for optimization of a linear objective function, subject to linear equality and linear inequality constraints. – Informally, linear programming determines the way to achieve the best outcome (such as maximum profit or lowest cost) in a given mathematical model and given some list of requirements represented as linear equations. – More formally, given a polytope (for example, a polygon or a polyhedron), and a real-valued affine function

– defined on this polytope, a linear programming method will find a point in the polytope where this function has the smallest (or largest) value. TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 108 Prototypical Problem

• ..

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 109 Data Gathering à Parameter Estimates • Number of hours of production time available per week in each plant for these new products. – (Most of the time in these plants already is committed to current products, so the available capacity for the new products is quite limited.) • Number of hours of production time used in each plant for each batch produced of each new product. • Profit per batch produced of each new product – (Profit per batch produced was chosen as an appropriate measure after the team concluded that the incremental profit from each additional batch produced would be roughly constant regardless of the total number of batches produced. Because no substantial costs will be incurred to initiate the production and marketing of these new products, the total profit from each one is approximately this profit per batch produced times the number of batches produced.) TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 110 Product Production: 2 new products • Product 1: An 8-foot glass door with aluminum framing • Product 2: A 4 X 6 foot double-hung wood-framed window

Plant1: Aluminum Framing Plant3: Glass cut and Plant2: assembly Wood framing

Wyndor (Window or Door) Glass problem

Each batch of product 1 produced per week uses 1 hour of production • .. time per week in Plant 1, whereas only 4 hours per week are available.

x1 = number of batches of product 1 produced per week x2 = number of batches of product 2 produced per week Z = total profit per week (in thousands of dollars) from producing these two products Thus, x1 and x2 are the decision variables for the model. Using the bottom row of Table 3.1, we obtain Z = 3x1 + 5x2

• ..

The number of hours used per week in Plant 3 (shared plant) by choosing x1 and x2 as the new product rates would be 3x1 + 2x2. This number needs to be <= the plant available capacity of 18.

• Treat the Wyndor (Window or Door) Glass problem as a linear programming problem

• Now solve using – Graphical approach(this lecture) – Simplex algorithm via tableau approach (next lecture) – Simplex algorithm via matrices (next lecture) – Interior point algorithm (see H&L textbook)

Note that the nonnegativity restrictions x1 ≥ 0 and x2 ≥ 0 require (x1 , x2 ) to lie on the positive side of the axes (including actually on either axis), i.e., in the first quadrant. Next, observe that the restriction x1 ≤ 4 means that (x1 , x2 ) cannot lie to the right of the line x1 ≤ 4.

The demo called Graphical Method in your OR Tutor provides a more detailed example of constructing a feasible region.

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 116 Step 2: Select Point in Feasible Region that maximizes the objective function • To discover how to perform this step efficiently, begin by trial and error. Try (draw line), for example, – Z = 10 + 3x1 + 5x2 – to see if there are in the permissible region any values of (x1 , x2 ) that yield a value of Z as large as 10 – Shoot for big values of Z (20, 36)

Figure reveals that a segment of the line 3x1+ 5x2 + 10 lies within the region, so that the maximum permissible value of Z must be at least 10.

(profit) contour lines of the objective function in the Slope-Intercept form. Slope is -3/5 (i.e., each unit increase of x1 changes x2 by -3/5)

Slope is fixed so all lines will be parallel; only the intercept changes.

So do trial-and-error exploration of profits, such that the profit contour line that contain at least one point in the feasible region

E.g., Contour line: using a point and the slope. Given an intercept of 2, implies a Z value of 10 and that the point (0, 2) is on the contour line. So use the point (2,0) and the slope (-3/5) to plot the line (using abline() in R)

Slope Y-Intercept

• Do it on paper and then in R – Plot feasible region • Plot Contour Lines of the the Obj Function – So do trial-and-error exploration of profits, such that the profit contour line contain at least one point in the feasible region and locate the optimal – Plot (profit) contour lines of the objective function using the Slope-Intercept form. Slope is -3/5 (i.e., each unit increase of x1 changes x2 by -3/5). – [RULER] Set your ruler to the right slope and move it Then move the ruler with fixed slope through the feasible region in the direction of improving Z (for maximization problems).

• Optimal solution is x1 = 2, x2 = 6, with Z = 36. – This solution indicates that the Wyndor Glass Co. should produce products 1 and 2 at the rate of 2 batches per week and 6 batches per week, respectively, with a resulting total profit of $36,000 per week. No other mix of the two products would be so profitable—according to the model. • Are we done?

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 120 Optimal Solution for Wyndor ? Are we done? • Optimal solution is x1 = 2, x2 = 6, with Z = 36. – This solution indicates that the Wyndor Glass Co. should produce products 1 and 2 at the rate of 2 batches per week and 6 batches per week, respectively, with a resulting total profit of $36,000 per week. No other mix of the two products would be so profitable—according to the model. • Are we done? – In well-conducted OR studies do not simply find one solution for the initial model formulated and then stop. – All six phases are important, including thorough testing of the model and postoptimality analysis – In full recognition of these practical realities, the OR team now is ready to evaluate the validity of the model more critically and to perform sensitivity analysis on the effect of the estimates being different because of inaccurate estimation, changes of circumstances, etc. TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 121 Continuing the Learning Process with Your OR Courseware

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 122 On Windows 8 (via Parallels on Mac): Press Command Key for a couple of Seconds to bring up Windows VM

• ..

http://www.java.com/en/ download/index.jsp for IOR Tutor

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 125 TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 126 (1) IORTutorial (interactive) and (2) OR Tutor

(1)

(2)

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 128 TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 129 • ..

• ..

Boot up Explorer in Parallels and then type in:

• Solve a variety of OR models using – The Excel Solver – LINGO/LINDO – MPL/CPLEX – R (optim) • http://www.r-bloggers.com/linear-programming-in-r-an- lpsolveapi-example/ • http://astrostatistics.psu.edu/su07/R/html/boot/html/ simplex.html – Install.packages(“boot”) – library("boot")

install.packages(“boot”) library("boot") # This example is taken from Exercise 7.5 of Gill, Murray, # and Wright (1991). enj <- c(200, 6000, 3000, -200) fat <- c(800, 6000, 1000, 400) vitx <- c(50, 3, 150, 100) vity <- c(10, 10, 75, 100) vitz <- c(150, 35, 75, 5) simplex(a = enj, A1 = fat, b1 = 13800, A2 = rbind(vitx, vity, vitz), b2 = c(600, 300, 550), maxi = TRUE)

• Read Hillier and Lieberman, pages 1-30 for lecture 1 • For lecture 2, read the remainder of chapter 3 and Chapter 4

• Take the IOR Tutor and OR Tutor for a test drive

TIM 206 (30155 ) Introduction to Optimization Theory and Applications, Winter 2013 © 2013 James G. Shanahan 139 Guidelines for Homework • Please provide code, graphs and comments in a Word or PDF report. Don’t forget to put your name, email and date of submission on each report. Please follow the Springer LNCS style (templates for Word and Latex are available at – http://www.springer.com/computer/lncs?SGWID=0-164-6-793341-0 – I.e., pretend you are writing a conference paper (at in format) • Please provide R code in a separate file .R file and embed the code also in your answers along with the graphs and tables. Please comment your code so that I or anybody else can understand it and please cross reference code with problem numbers and descriptions. Please label each figure and table appropriately. • Please name files as follows: TIM206-2013-HWK-Week01- StudentLastName.R, .doc, .pdf etc.. • Please create a separate driver function for each exercise or exercise part (and comment!) • If you have questions please raise them in class or via email or during office hours if requested • Homework is due on Wednesday, of the following week by 7PM. • Please submit your homework by email to: [email protected] and [email protected] with the subject “TIM 206 Winter 2013 Homework 1” • Have fun!

• Exercises in H&L Book (9th Edition) • 2.1.1, 2.1.3 • 3.1.5; in this problem use R to plot the feasible region and contour lines (NOTE: use figure 3.3 as a role model for the graph, labels for the axes, the contour lines, and also for the figure caption!) • 3.1.6 (using R) • 3.1.7 (using R and the IOR Tutorial)

• Do forget to send me a 200 word bio!