Modeling Annotators a Generative Approach to Annotators Are Useful
Total Page：16
File Type：pdf, Size：1020Kb
Zaidan & Eisner – Modeling ModelingAnnotators Annotators: Zaidan & Eisner – Modeling Annotators A Generative Approach to Annotators are useful. Learning from Annotator Rationales They provide us with data sets that we use everyday in the context of statistical learning. Omar F. Zaidan Jason Eisner Annotators are smart! ☺ The Center for Language We achieve good performance using such data in and Speech Processing many NLP tasks. Johns Hopkins University Annotators are underutilized… EMNLP 2008 – Honolulu, HI A lot of thought goes into annotation, but little of that is th Saturday October 25 , 2008 captured. {ozaidan |jason} @cs.jhu.edu Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators Annotators are Underutilized? Annotators are Underutilized! م اب ,Hey annotator, Armageddon “Hey annotator“ ها ا ار ه آر . ا، إاج is this review This disaster flick is a disaster alright. Directed by is this review Tony Scott (Top Gun), it's the story of an asteroid the ت ( ج ب ) ، وي آ positive or size of Texas caught on a collision course with Earth. positive or و س ام ة ار . negative?” After a great opening, in which an American negative?” ا را، ء أ، إ إ spaceship, plus NYC, are completely destroyed by a رك، ا اآ و . comet shower, NASA detects said asteroid and go ن ا ا ( وس و ) ، :Annotator says: into a frenzy. They hire the world's best oil driller Annotator says (Bruce Willis), and send him and his crew up into و و إ اء ح ا ه. .space to fix our global problem ه ارة واآ و و ي The action scenes are over the top and too ludicrous آت، إ وت أ، و for words. So much so, I had to sigh and hit my head و ات . و أ ن رؤ ب with my notebook a couple of times. Also, to see a رن آا ه اه . ا ا wonderful actor like Billy Bob Thornton in a film like (y = –1) this is a waste of his talents. The only real reason for (y = –1) ها ا ه و اق د اآ making this film was to somehow out-perform Deep . ا، م اب . ??Useful! Impact. Bottom line is, Armageddon is a failure. Useful 1 Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators Annotators are Underutilized! Annotators are Underutilized! Hey annotator, Armageddon“ م اب ,Hey annotator“ is this review This disaster flick is a disaster alright. Directed by ها ا ار ه آر . ا، إاج is this review Tony Scott (Top Gun), it's the story of an asteroid the .positive or size of Texas caught on a collision course with Earth ت ( ج ب ) ، وي آ positive or و س ام ة ار . negative?” negative?” After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a ا را، ء أ، إ إ comet shower, NASA detects said asteroid and go رك، ا اآ و . Annotator says: into a frenzy. They hire the world's best oil driller ن ا ا ( وس و ) ، :Annotator says (Bruce Willis), and send him and his crew up into .space to fix our global problem و و إ اء ح ا ه. The action scenes are over the top and too ludicrous ه ارة واآ و و ي for words. So much so, I had to sigh and hit my head آت، إ وت أ، و with my notebook a couple of times. Also, to see a و ات . و أ ن رؤ ب wonderful actor like Billy Bob Thornton in a film like رن آا ه اه . ا ا (y = –1) (y = –1) this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep ها ا ه و اق د اآ .Useful! Impact. Bottom line is, Armageddon is a failure . ا، م اب . !Useful Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators Why would Rationales be Useful? Related Work • “Annotators are underutilized . Let’s ask them to do more than just annotate class.” • Let’s ask annotator to identify relevant features . – Haghighi & Klein (2006), Raghavan et al. (2006), Rationales Class labels Druck et al. (2008). • But: – Features could be hard to describe. Apple shares up 3% apple apple- noun – Features could be hard for annotators to understand. Annotator – We might want different features in the future. 2 Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators Saving Private Ryan War became a reality to me after seeing Saving Private Ryan. Steve Spielberg goes beyond reality with his latest production. Keep the kids home as the R rating is for Reality. Tom Hanks is stunning as Capt John Miller, set out in France during WW II to . Non-annotated . rescue and return home a soldier, Private Ryan (Matt Damon) who lost three brothers in the war. documents Spielberg takes us inside the heads of these individuals as they face death during the horrific battle scenes. Private Ryan is not for everyone, but I felt the time was right for a movie like this to be made. The movie reminds us of the sacrifices made by our fighting men and women. For this I thank them and for Steve Spielberg for making a movie that I will never forget. And I’m sure the Academy . will not forget Tom Hanks come April, as another well deserved Oscar with be in Tom's possession. Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators The Postman <x,y,r> Question: after the disaster that was Waterworld, Class and also what the fuck were the execs who gave Costner the money to make another movie thinking?? “rationales ” In this 3 hour advertisement for his new hair weave, Costner plays a nameless drifter who dons a long . dead postal employee's uniform (I shit you not) and Annotated gradually turns a nuked-out USA into an idealized hippy-dippy society. (The main accomplishment of documents this brave new world is in re-inventing polyester.) t?? When he's not pointing the camera directly at ha himself, director Costner does have a nice visual w w sense, but by the time the second hour rolled no around, I was reduced to sitting on my hands to … keep from clawing out my own eyes. Mark this one OK "return to sender". 3 Zaidan & Eisner – Modeling Annotators Zaidan & Eisner – Modeling Annotators Linear Classifier Linear Classifier • In our experiments, we use a linear classifier. • At test time, when presented with document x: r r • Classifier is represented by a weight vector . +1 , θ ⋅ f (x) > 0 Choose y* = r r −1 , θ ⋅ f (x) < 0 • Rationales will play a role when learning . • Where:r • Improvements solely due to a better-learned . ⋅ –fr )( : feature vector (18k binary unigram features). (vs. no rationales) –θ : corresponding weights (i.e. 18k weights). • At test time: Positive weights favor y = +1 – No change to decision rule. Negative weights favor y = –1 – No new features. – No need for rationales. Zaidan & Eisner – Modeling Annotators x Zaidan & Eisner – Modeling Annotators Trees Lounge is the directoral debut from one of my favorite actors, Steve Buscemi. He gave memorable performences in in The Soup, Fargo, and Reservoir Dogs. Now ! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better he tries his hand at writing, directing and acting all in the same flick. The movie starts brother buscemi but capable career chloe cream debbie debut decides did dies out awfully slow with Tommy (Buscemi) hanging around a local bar the "trees directing director dogs drive even excellent expectation fargo favorite finding flick lounge" and him pestering his brother. It's obvious he a loser. But as he says "it's for from gave girl good hand hanging having he him his i ice i'm in interest is it it's better I'm a loser and know I am, then being a loser and not thinking I am." Well put. job know liked local loser lounge love memorable michael movie mr my named not The story starts to take off when his uncle dies, and Tommy, not having a job, decides now obvious of off old one out pick put reach reservoir same says seen sevigny slow to drive an ice cream truck. Well, the movie starts to pick up with him finding a love soup starts steve story take the then think thinking this though to tommy trees tries interest in a 17 year old girl named Debbie (Chloe Sevigny) and ... I liked this movie truck uncle up well when with writing year you you've alot even though it did not reach my expectation. After you've seen him in Fargo and Reservoir Dogs, you know he is capable of a better performence. I think his brother, abc abc abc abc abc favor y = +1 Michael, did an excellent job for his debut performence. Mr. Buscemi is off to a good -0.2 -0.1 0.0 +0.1 +0.2 favor y = –1 career as a director! ! " () , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe cream debbie . f (x): 0/1 vector; 18k elements fand = 1, freach = 1, fterrific = 0, … ! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl . brother buscemi but capable career chloe cream debbie debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick good hand hanging having he him his i ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie .