Closing the Loop on Big Data Data

Closing the Loop on Big Data data

NSF Big Data Workshop March 21, 2013 Rob Nowak www.ece.wisc.edu/~nowak BIG DATA: A Marriage of Machine and Man

Human Judgements labeling, annotation, comparisons

questions /queries

model “training space data” BIG DATA: A Marriage of Machine and Man

Human Judgements labeling, annotation, comparisons

questions /queries

“training model space data” active learning Machine Learning (Passive) Raw unlabeled data

passive learner expert/oracle provides labels Machine Learning (Passive) Raw unlabeled data

Labeled data

passive learner expert/oracle provides labels Machine Learning (Passive) Raw unlabeled data

Labeled data

passive learner expert/oracle rule for predicting Y from X provides labels Active Learning Raw unlabeled data

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

X2 identical or very similar to X1... no need to ask for label

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

X2 identical or very similar to X1... no need to ask for label

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

X2 identical or very similar to X1... no need to ask for label

active learner expert/oracle provides labels Active Learning Raw unlabeled data

machine requests labels for selected data

X2 identical or very similar to X1... no need to ask for label

active learner expert/oracle provides labels

rule for predicting Y from X Computer Algorithms Rely Increasingly on Human Helpers - NYTimes.com 3/11/13 10:28 AM

March 10, 2013 Algorithms Get a Human Hand in Steering Web By STEVE LOHR Trading stocks, targeting ads, steering political campaigns, arranging dates, besting people on “Jeopardy” and even choosing bra sizes: computer algorithms are doing all this work and more.

But increasingly, behind the curtain there is a decidedly retro helper — a human being.

Although algorithms are growing ever more powerful, fast and precise, the computers themselves are literal-minded, and context and nuance often elude them. Capable as these machines are, they are not always up to deciphering the ambiguity of human language and the mystery of reasoning. Yet these days they are being asked to be more humanlike in what they figure out.

“For all their brilliance, computers can be thick as a brick,” said Tom M. Mitchell, a computer scientist at Carnegie Mellon University.

And so, while programming experts still write the step-by-step instructions of computer code, additional people are needed to make more subtle contributions as the work the computers do has become more involved. People evaluate, edit or correct an algorithm’s work. Or they assemble online databases of knowledge and check and verify them — creating, essentially, a crib sheet the computer can call on for a quick answer. Humans can interpret and tweak information in ways that are understandable to both computers and other humans.

Question-answering technologies like Apple’s Siri and I.B.M.’s Watson rely particularly on the emerging machine-man collaboration. Algorithms alone are not enough.

Twitter uses a far-flung army of contract workers, whom it calls judges, to interpret the meaning and context of search terms that suddenly spike in frequency on the service.

For example, when Mitt Romney talked of cutting government money for public broadcasting in a presidential debate last fall and mentioned Big Bird, messages with that phrase surged. Human judges recognized instantly that “Big Bird,” in that context and at that moment, was mainly a political comment, not a reference to “Sesame Street,” and that politics-related messages should pop up when someone searched for “Big Bird.” People can understand such references more accurately and quickly than software can, and their judgments are fed http://www.nytimes.com/2013/03/11/technology/computer-algorithms-rely-increasingly-on-human-helpers.html?hpw&pagewanted=print Page 1 of 4 “You need judgement, and to be able to intuitively recognize the smaller sets of data that are most important,” Mr. Taylor said. “To do that, you need some level of human involvement.” Big Data & Big Computing

Amount of data being collected

Time Big Data & Big Computing

Amount of data being collected

Amount of computational resources

Time Big Data & Big Computing

Amount of data being collected

Amount of computational resources

Cost of computational resources

Time Fast Computers and Slow People

Amount of data being collected

Time Fast Computers and Slow People

Amount of data being collected

Human response time

Time Fast Computers and Slow People

Amount of data being collected

Human response time

Human average hourly wage

Time Binary Classification sports

Republicans in Close Race Vie for Senate Runoff Vote - NYTimes.com 7/31/12 12:27 PM

July 30, 2012 A Runoff Is Down to the Wire in Texas By ERIK ECKHOLM HOUSTON — It may be the armadillo days of summer in Texas, but a runoff vote on Tuesday for the Republican Senate nomination has jolted the party establishment here and around the country as a magnetic Tea-Party conservative with no elective experience gains momentum against the chosen candidate of Gov. Rick Perry.

Only a few months ago the longtime lieutenant governor, David Dewhurst, 66, seemed all but certain to win the nomination, which in Republican-dominated Texas is tantamount to winning the seat being vacated in November by Senator Kay Bailey Hutchison.

A successful businessman with Romney-esque wealth, Mr. Dewhurst has been allied with Mr. Perry — himself a favorite of the Tea Party and religious conservatives — as Texas burnished its reputation as a low-tax, small-government state with rare growth in jobs. Beyond Mr. Perry, who calls him “a great conservative leader,” Mr. Dewhurst has been endorsed by many party leaders, big business groups and the farm bureau.

But Mr. Dewhurst won only 45 percent of the votes in the May primary, forcing him into a runoff against the No. 2 finisher, Ted Cruz, a Harvard-educated lawyer and former state solicitor general who trailed him by 11 points.

Now, after a bitter and costly campaign, Mr. Cruz, 41, appears to have pulled even or even nudged ahead, political analysts said, and his campaign has already signaled a profound shift in Texas politics as frustrated conservative activists rattle the party leadership.

“If Cruz wins, it will be a huge upset and a further indication that the grass roots is a very powerful force in Texas,” said Tom Pauken, a commissioner of the Texas Workplace Commission and a former state Republican chairman who has stayed neutral in the race. In one telling sign of growing divisions, Governor Perry was booed at the state Republican convention in June when he introduced his “friend” David Dewhurst.

A victory for Mr. Cruz would follow other triumphs for insurgent conservatives in this year’s primaries, including the toppling of Senator Richard G. Lugar of Indiana by Richard Mourdock and the defeat of a veteran Republican by Deb Fischer for the Senate nomination in Nebraska.

http://www.nytimes.com/2012/07/31/us/politics/republicans-in-close-race-vie-for-senate-runoff-vote.html?hp&pagewanted=print Page 1 of 3 Republicans in Close Race Vie for Senatenon-sports Runoff Vote - NYTimes.com 7/31/12 12:27 PM

http://www.nytimes.com/2012/07/31/us/politics/republicans-in-close-race-vie-for-senate-runoff-vote.html?hp&pagewanted=print Page 1 of 3 unlabeled documents

expert/oracle provides labels to machine learner Tong and Koller (2001) Active Learning

Learning Problem: Consider a binary prediction problem involving a collection of “classifiers.” Each classifier maps points in the “feature-space” (e.g., Rd)to binary labels. The features and labels are governed by an unknown distribution P . The goal is to select the classifier that minimizes the probability of misclassification using as few training examples as possible. feature 2 feature

feature 1 Active Learning

best linear classifier

feature 1 Active Learning

best linear classifier

feature 1 Active Learning

which unlabeled point is likely to be most informative ?

for which point do we have the greatest uncertainty about its label ? feature 2 feature

best linear classifier

feature 1 Active Learning feature 2 feature

best linear classifier

feature 1 prob. of prob. error

# questions Active Learning

Non-adaptive strategy: Label a random sample feature 2 feature

best linear classifier

feature 1 prob. of prob. error

# questions Active Learning

Non-adaptive strategy: Label a random sample

Adaptive strategy: Label a sample near best decision boundary based on labels seen so far feature 2 feature

best linear classifier

feature 1 prob. of prob. error

# questions Active Learning

Non-adaptive strategy: Label a random sample

Adaptive strategy: Label a sample near best decision boundary based on labels seen so far feature 2 feature

best linear classifier

feature 1

active learning finds optimal classifier with much less human assistance! prob. of prob. error

# questions Active Learning in the Age of Big Data

How do we get the same amount of information from humans with fewer questions? Ask better questions. (a.k.a. active learning, adaptive querying, learning with feedback)

passive Same answer with fewer questions! active error

# questions

Does adaptivity always help? Not always, but it does help in many Big Data methods: - Classification (SVMs) - Graphical Modeling - Clustering - Function Estimation / Optimization - Ranking and Low-Dimensional Embedding - Sparsity and Compressed Sensing

Can active learning be scaled up to work with “BIGDATA”? Challenges

1. Infrastructure: Aggregate information from many sources from many places

2. Interaction: Humans are relatively slow, but delays in interaction must be imperceptible

3. Optimization: Selecting most informative queries must be computationally tractable Challenges

1. Infrastructure: Aggregate information from many sources from many places Linux + Apache MySQL Python

Server in the cloud setup: 10 Minutes, $0-$100/month http://www.robotmedia.net/2011/04/how-to-create-an-amazon-ec2-instance-with-apache-php-and-mysql-lamp/

2. Interaction: Humans are relatively slow, but delays in interaction must be imperceptible

3. Optimization: Selecting most informative queries must be computationally tractable Challenges

1. Infrastructure: Aggregate information from many sources from many places Linux + Apache MySQL Python

Server in the cloud setup: 10 Minutes, $0-$100/month http://www.robotmedia.net/2011/04/how-to-create-an-amazon-ec2-instance-with-apache-php-and-mysql-lamp/

2. Interaction: Humans are relatively slow, but delays in interaction must be imperceptible

http server + database + algorithm logic: an afternoon, $0 https://docs.djangoproject.com/en/1.5/intro/tutorial01/

3. Optimization: Selecting most informative queries must be computationally tractable Challenges

1. Infrastructure: Aggregate information from many sources from many places Linux + Apache MySQL Python

Server in the cloud setup: 10 Minutes, $0-$100/month http://www.robotmedia.net/2011/04/how-to-create-an-amazon-ec2-instance-with-apache-php-and-mysql-lamp/

2. Interaction: Humans are relatively slow, but delays in interaction must be imperceptible

http server + database + algorithm logic: an afternoon, $0 https://docs.djangoproject.com/en/1.5/intro/tutorial01/

3. Optimization: Selecting most informative queries must be computationally tractable

BIG Mathematical Challenge BIG DATA:Optimization A Marriage ofChallenge Machine and Man

Human Judgements labeling, annotation, comparisons

questions /queries Optimizing interaction between human experts and machine can be posed as a BIG dynamic programming problem...

“training model space data” active learning

Tuesday, March 19, 13 BIG DATA:Optimization A Marriage ofChallenge Machine and Man

Human Judgements labeling, annotation, comparisons

questions /queries Optimizing interaction between human experts and machine can be posed as a BIG dynamic programming problem...

Unfortunately this problem is computationally intractable except in toy-like settings

“training model space data” active learning

Tuesday, March 19, 13 BIG DATA:Optimization A Marriage ofChallenge Machine and Man

Human Judgements labeling, annotation, comparisons

questions /queries Optimizing interaction between human experts and machine can be posed as a BIG dynamic programming problem...

Unfortunately this problem is computationally intractable except in toy-like settings

“training model space data” active learning

Tuesday, March 19, 13 A BIG research question is how to develop scalable and provably stable approaches to active learning Learning from Comparative Judgements

L. L. Thurstone

Is model A better than B?

answers = bits

model data space space Learning from Comparative Judgements

Humans are much more reliable and L. L. Thurstone consistent at making comparative judgements, than at giving numerical ratings or evaluations Bijmolt and Wedel (1995) Stewart, Brown, and Chater (2005) Is model A better than B?

answers = bits

model data space space Learning from Comparative Judgements

answers = bits

data model space space active learning Ranking and Embedding

Use human judgements to understand relationships among N objects Ranking and Embedding

Use human judgements to understand relationships among N objects

sonar signals Ranking and Embedding

Use human judgements to understand relationships among N objects

Republicans in Close Race Vie for Senate Runoff Vote - NYTimes.com 7/31/12 12:27 PM

A victory for Mr. Cruz would follow other triumphs for insurgent conservatives in this year’s primaries, including the toppling of Senator Richard G. Lugar of Indiana by Richard Mourdock sonar signals documentsand the defeat of a veteran Republican by Deb Fischer for the Senate nomination in Nebraska.

http://www.nytimes.com/2012/07/31/us/politics/republicans-in-close-race-vie-for-senate-runoff-vote.html?hp&pagewanted=print Page 1 of 3 Ranking and Embedding

Use human judgements to understand relationships among N objects

Republicans in Close Race Vie for Senate Runoff Vote - NYTimes.com 7/31/12 12:27 PM

A victory for Mr. Cruz would follow other triumphs for insurgent conservatives in this year’s primaries, including the toppling of Senator Richard G. Lugar of Indiana by Richard Mourdock sonar signals documentsand the defeat of a veteran Republican by Deb Fischer for the Senate nomination in Nebraska. beers

http://www.nytimes.com/2012/07/31/us/politics/republicans-in-close-race-vie-for-senate-runoff-vote.html?hp&pagewanted=print Page 1 of 3 Ranking and Embedding

Use human judgements to understand relationships among N objects

Republicans in Close Race Vie for Senate Runoff Vote - NYTimes.com 7/31/12 12:27 PM

A victory for Mr. Cruz would follow other triumphs for insurgent conservatives in this year’s primaries, including the toppling of Senator Richard G. Lugar of Indiana by Richard Mourdock sonar signals documentsand the defeat of a veteran Republican by Deb Fischer for the Senate nomination in Nebraska. beers

http://www.nytimes.com/2012/07/31/us/politics/republicans-in-close-race-vie-for-senate-runoff-vote.html?hp&pagewanted=print Page 1 of 3

Can human perceptions about these objects be represented in a low-dimensional space?

How many questions/comparisons are needed to determine how a new object is related to others? Active Ranking Consider N objects embedded in a d-dimensional space. Goal: rank objects according to similarity to a new or “ideal” object W.

A B w

F E

D G Active Ranking Consider N objects embedded in a d-dimensional space. Goal: rank objects according to similarity to a new or “ideal” object W.

A B w your “ideal” beer in “beer space” (e.g, hoppiness, lightness, maltiness,...)

F E

D G Active Ranking Consider N objects embedded in a d-dimensional space. Goal: rank objects according to similarity to a new or “ideal” object W.

A B w your “ideal” beer in “beer space” (e.g, hoppiness, lightness, maltiness,...)

C Is A better than B ? is A W < B W ? , k k k k F E

D G Ranking According to Distance

C < A < B < E < G < D < F A B w

F E

D G Ranking According to Distance

A B

C E < B < F < G < C < A < D

w F E

D G Ranking According to Distance

A B

F E D < G < C < E < A < B < F

D w G Ranking According to Distance

A Goal: Determine ranking by asking comparisons like “Do you prefer A or B?” B

F E D < G < C < E < A < B < F

D w G Ranking According to Distance

A Goal: Determine ranking by asking comparisons like “Do you prefer A or B?” B

... now there are at most n2d rankings (instead of n!), and so in principle no more than 2d log n bits of information are needed.

F E D < G < C < E < A < B < F

D w G Ranking and Geometry suppose we have ranked 4 beers ranking implies that W lies in shaded region Ranking and Geometry suppose we have ranked 4 beers ranking implies that W lies in shaded region

next beer (chosen at random) Ranking and Geometry suppose we have ranked 4 beers ranking implies that W lies in shaded region

Testing for intersection is a fast

linear program, and only n log n next beer tests are needed in total. (chosen at random) Ranking and Geometry suppose we have ranked 4 beers ranking implies that W lies in shaded region

Testing for intersection is a fast

linear program, and only n log n next beer tests are needed in total. (chosen at random)

Key Observation: most queries will not be ambiguous, therefore the expected total number of queries made by lazy binary search is about d log n Ranking and Geometry suppose we have ranked 4 beers ranking implies that W lies in shaded region

Testing for intersection is a fast

linear program, and only n log n next beer tests are needed in total. (chosen at random)

Key Observation: most queries will not be ambiguous, therefore the expected total number of queries made by lazy binary search is about d log n

Meta-Principle: Machine randomly selects potential queries and quickly assesses potential information gain

Beer Space

http://www.ratebeer.com/beer/two-hearted-ale/ Beer Space

http://www.ratebeer.com/beer/two-hearted-ale/

IPA Pilsner Pale ale Blond Light lager Amber Brown ale Doppelbock Belgian light Porter Wit Belgian dark Lambic Wheat Stout caring joviality Love Learning an Embeddingenvy From Comparisons Joy zest Surprise sentimentality Anger 0.6 “Is word A more like adoration liking Sadness distress word B or C?” tenseness Fear homesicknesssympathy hope longing ecstasy compassion remorse jubilation melancholypride affectionglee cheerfulness 0.4 apprehension desire revulsion enjoymentdelight uneasiness thrilleagernessamazementfondness anxiety astonishment relief excitement surprise regret unhappinessannoyance 0.2 tendernesssatisfaction depression gladness joy defeat jollinesselation hysteria contentmentpleasureenthusiasm worry arousal ferocity lust exhilarationlove fright bliss wrath exasperation 0 scorn nervousness dread loathing infatuationamusementoptimism agonyanguishhumiliation guilt frustration grief dislike misery gloom despair zeal −0.2 grumpinessneglect grouchiness passion hurt euphoria rejection sorrow enthrallment attraction bitterness happiness aggravation isolation dejectionalienation triumph −0.4 disappointmentalarm lonelinessinsult hatesuffering rapture horror vengefulness insecurity displeasuretorment contempt spite resentment woe jealousyanger hostility dismay fear fury sadness rage −0.6 panicagitation pity terror disgust outrage hopelessness mortification glumness shock embarrassment

shame −0.8

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 caring joviality Love Learning an Embeddingenvy From Comparisons Joy zest Surprise sentimentality Anger 0.6 “Is word A more like adoration liking Sadness distress word B or C?” tenseness Fear homesicknesssympathy hope longing ecstasy compassion remorse jubilation melancholypride affectionglee cheerfulness 0.4 apprehension N = 400desire words revulsion enjoymentdelight uneasiness thrilleagernessamazementfondness anxiety astonishment relief excitement 3 surprise regret N unhappinessannoyance 0.2 30 milliontendernesssatisfaction comparisons depression 2 gladness joy defeat ⇡ jollinesselation hysteria contentmentpleasureenthusiasm worry arousal ferocity lust exhilarationlove fright bliss wrath exasperation 0 scorn nervousness dread loathing infatuationamusementoptimism agonyanguishhumiliation guilt frustration grief dislike misery gloom despair zeal −0.2 grumpinessneglect grouchiness passion hurt euphoria rejection sorrow enthrallment attraction bitterness happiness aggravation isolation dejectionalienation triumph −0.4 disappointmentalarm lonelinessinsult hatesuffering rapture horror vengefulness insecurity displeasuretorment contempt spite resentment woe jealousyanger hostility dismay fear fury sadness rage −0.6 panicagitation pity terror disgust outrage hopelessness mortification glumness shock embarrassment

shame −0.8

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 caring joviality Love Learning an Embeddingenvy From Comparisons Joy zest Surprise sentimentality Anger 0.6 “Is word A more like adoration liking Sadness distress word B or C?” tenseness Fear homesicknesssympathy hope longing ecstasy compassion remorse jubilation melancholypride affectionglee cheerfulness 0.4 apprehension N = 400desire words revulsion enjoymentdelight uneasiness thrilleagernessamazementfondness anxiety astonishment relief excitement 3 surprise regret N unhappinessannoyance 0.2 30 milliontendernesssatisfaction comparisons depression 2 gladness joy defeat ⇡ jollinesselation hysteria contentmentpleasureenthusiasm worry arousal ferocity lust exhilarationlove embedding is equivalentfright tobliss ﬁnding wrath exasperation 0 N rankings (one wrt each object) scorn nervousness dread loathing infatuationamusementoptimism agonyanguishhumiliation guilt frustration grief dislike misery gloom despair zeal −0.2 grumpinessneglect grouchiness passion hurt euphoria rejection sorrow enthrallment attraction bitterness happiness aggravation isolation dejectionalienation triumph −0.4 disappointmentalarm lonelinessinsult hatesuffering rapture horror vengefulness insecurity displeasuretorment contempt spite resentment woe jealousyanger hostility dismay fear fury sadness rage −0.6 panicagitation pity terror disgust outrage hopelessness mortification glumness shock embarrassment

shame −0.8

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 caring joviality Love Learning an Embeddingenvy From Comparisons Joy zest Surprise sentimentality Anger 0.6 “Is word A more like adoration liking Sadness distress word B or C?” tenseness Fear homesicknesssympathy hope longing ecstasy compassion remorse jubilation melancholypride affectionglee cheerfulness 0.4 apprehension N = 400desire words revulsion enjoymentdelight uneasiness thrilleagernessamazementfondness anxiety astonishment relief excitement 3 surprise regret N unhappinessannoyance 0.2 30 milliontendernesssatisfaction comparisons depression 2 gladness joy defeat ⇡ jollinesselation hysteria contentmentpleasureenthusiasm worry arousal ferocity lust exhilarationlove embedding is equivalentfright tobliss ﬁnding wrath exasperation 0 N rankings (one wrt each object) scorn nervousness dread loathing infatuationamusementoptimism agonyanguishhumiliation guilt frustration grief dislike misery gloom despair zeal −0.2 grumpinessneglect grouchiness passion hurt should be possible with abouteuphoria rejection sorrow Ndlog N 7000enthrallment comparisonsattraction bitterness happiness aggravation isolation ⇡ dejectionalienation triumph −0.4 disappointmentalarm lonelinessinsult hatesuffering rapture horror vengefulness insecurity displeasuretorment contempt spite resentment woe jealousyanger hostility dismay fear fury sadness rage −0.6 panicagitation pity terror disgust outrage hopelessness mortification glumness shock embarrassment

shame −0.8

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 Learning an Embedding From Comparisons

longing

aggravation hope

user1

t=4,answer t=1,query

server Linux + Apache MySQL t=2,query t=5,answer t=3,query t=6,answer Python

user2 user3 The Scientific Process in a Laboratory

experiments

data scientist The Scientific Process at Large

Lab A The Scientific Process at Large

Lab B

Lab A The Scientific Process at Large

Lab B

Lab A Lab C The Scientific Process at Large

Lab B

Lab A Lab C ... The Scientific Process at Large

Lab B

Lab A Lab C ...

We need new mathematical theory and methods to optimize this! Big Data Challenge: Human-in-the-Loop

Amount of data being collected

Amount of computational resources

Time Big Data Challenge: Human-in-the-Loop

Amount of data being collected

Amount of computational resources

Human response time

Time Big Data Challenge: Human-in-the-Loop

Amount of data being collected

Amount of computational resources

Human response time

Human average hourly wage

Time