Machine Learning Logistics Book by Ted Dunning & Ellen Friedman © 2018 (O’Reilly)

Beyond the Algorithm: What Makes Machine Learning Work?

Ellen Friedman, PhD 11 June 2018 Berlin Buzzwords #bbuzz

Ellen Friedman, PhD Principal Technologist, MapR Technologies Committer Apache Drill & Apache Mahout projects O’Reilly author

Email [email protected] [email protected]

Twitter @Ellen_Friedman #bbuzz

Had never tried machine learning, but he caught the bug…

Rhode Island Red Buff Orpington Jay

Image from Wikipedia & used under Creative Image from Wikipedia & used under Creative Image from Wikipedia & used under Creative Commons https://en.wikipedia.org/wiki/ Commons https://upload.wikimedia.org/wikipedia/ Commons https://en.wikipedia.org/wiki/ File:Rhode_Island_Red_cock,_cropped.jpg commons/7/74/ Aphelocoma#/media/ Barred_Plymouth_Rock_Rooster_001.jpg File:WesternScrubJay2.jpg

Chicken Chicken Not a Chicken

Chicken Chicken Predator

Run the Deploy Train model model model

Update model

How long does it take for Scrub Jay to peck an egg? • < 30 seconds

• Oops…

• Not sure yet. That’s a problem…

Right: This what a Buff Orpington really looks like.

But…this error in domain knowledge (wrong name) did not matter for SLAs.

Image from Wikipedia & used under Creative Image from Wikipedia used under Creative Commons Commons https://upload.wikimedia.org/wikipedia/ https://en.wikipedia.org/wiki/Orpington_chicken#/media/ commons/7/74/ File:Coq_orpington_fauve.JPG Barred_Plymouth_Rock_Rooster_001.jpg

• In some cases, building or training a model is simple

• Domain knowledge matters (really)

• Pay attention to SLAs

• For real business value, have a plan of action in response to machine learning insights (producing a report ≠ taking an action)

• Software engineers have a role in machine learning (!)

• Use clicks as input data: recommender gives poor performance – Model is testing the wrong preferences: how well people liked titles

• Use first 30 seconds of viewing as input data: recommender performance is good – Model now tests how well people liked the videos, not just the titles

GET /personal/comparison-table.jsp?iODg2OQ=51a90 HTTP/1.1GET /photo.jpg HTTP/1.1 Host: www.sometarget.com Host: lh4.googleusercontent.com User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0 Accept-Encoding: deflate Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Charset: UTF-8 Accept-Language: en-US,en;q=0.5 Accept-Language: fr Accept-Encoding: gzip, deflate, br Cache-Control: no-cache Referer: https://www.google.com Pragma: no-cache Connection: keep-alive Connection: Keep-Alive If-None-Match: "v9” Cache-Control: max-age=0

Attacker request Real request

GET photo.jpg HTTP/1.1 GET cc/borken.json HTTP/1.1 Host: lh4.googleusercontent host: c.qrs.my User-agent: Mozilla/5.0 (Ma user-agent: Mozilla/4.0 (co Accept: image/png,image/* accept: application/json, t Accept-language: en-US,en accept-language: en-US,en Accept-encoding: gzip, defl accept-encoding: gzip, defl Referer: https://www.google referer: none Connection: keep-alive connection: keep-alive If-none-match: "v9” if-none-match: "v9” Cache-control: max-age=0 cache-control: max-age=0

Real request Attacker request

• Detected anomaly in headers for the attackers vs normal (real) requests

• Pattern of behavior for attackers was allowable for headers and it was not predictable: but it was different

All other mages © E. Friedman

Image courtesy Mtell used with permission ©WesAbrams

© 2018 Ellen Friedman 23 Simple But Valuable: Accounting Audit Targeting Big industrial company with a lot of machinery • Tracking actions & parts – label as repairs, contracted services, delivery of supplies, etc. • Which are to be taxed, expensed or counted as revenues – Mislabelling can cost millions of dollars • Use machine learning to target potential mislabelling for audit review

• Relatively simple models (pattern matching; exception detection) deliver a huge business value.

https://mapr.com/blog/tensorflow-mxnet-caffe-h2o-which-ml-best/

It’s the logistics

Patient Facilities EMR management

With the right messaging tool at the heart of stream-1st Real-time analytics architecture you support other Medical test classes of use cases (B & C) results A

Insurance Medical tests audit

P C

Kafka / P C MapR Streams

P C

Transport Processing

Good stream transport is persistent, performant & pervasive!

• Many models to manage

• Model-to-model evaluation needs to be convenient & accurate

• Respond as the world changes: Deploy to production with agile roll out & roll

• There’s a need for good data engineering (not just data science)

Decoy Archive

m1 Features / Raw Input proﬁles Scores Rendezvous Results m2

Rendezvous Architecture described in:

- “Rendezvous Architecture” by Ted Dunning & Ellen Friedman, chapter in Encyclopedia of Big Data Technologies. Sherif Sakr and Albert Zomaya, editors. Springer International Publishing, © 2018 in press.

Decoy Archive

m1 Features / Raw Input proﬁles Scores Rendezvous Results m2

Free copy online courtesy of MapR: https://mapr.com/ebook/machine-learning- logistics/

• Looks like a model, but it just archives inputs

• Safe in a good streaming environment, less safe without good isolation

Decoy Archive

Input Model 2 Scores

Model 3

• Acts as a baseline reference for reasonable performance

• Can be compared to performance of newly deployed models