PROCESSING NATURAL LANGUAGE DAY 1: INTRODUCTION TO NLP Mark Granroth-Wilding 1 2 COURSE OUTLINE PROCESSING NATURAL LANGUAGE Mark Leo Lidia Granroth-Wilding Lepp¨anen Pivovarova Getting information out of language • Day Topic 1 Introduction to NLP Example 2 NLU pipeline and toolkits Who made the first electric guitar? 3 Finite state methods; statistical NLP Week 1 4 Syntax and parsing Query: string of characters 5 Evaluation • How to represent meaning in suitable form to find answer? 6 NLG and dialogue • 7 Vector space models and lexical semantics How to get from text to that? • 8 Information extraction; advanced stat. NLP What sort of processes are needed? Week 2 • 9 Ascension: no lectures What does system need to know about language? 10 Semantics and pragmatics; the future • 3 4 DAY OUTLINE ASSIGNMENTS Afternoons: practical programming assignments • Daily introduction Each day: • TAs available to help 9:15 – 12:00 Lectures D123 • Includes: 12:00 – 13:15 Lunch • Python programming 13:15 – 14:00 Introduction • ∼ BK107 Use of NLP tools 14:00 – 16:00 Practical assignments • Implementation of algorithms, statistical models, etc from • lectures Analysis of system output/behaviour • Consideration of method uses, limitations, . • 5 6 TEACHING ASSISTANTS ASSIGNMENTS Assistance with assignments will be provided by: Submitted same day: i.e. not homework • Submit using Moodle • Important part of learning (not optional) • Mark Leo Lidia Marked in 1-2 days • No detailed personal feedback • Collective feedback in lectures: common problems • Ask specific questions on Moodle forum • Eliel Khalid Elaine 7 8 ASSESSMENT COURSE MATERIALS Requirements to pass the course • Speak to me if Attend all lectures • problematic Attend all practicals (at least start) • Attempt assignments Course homepage: https://g-w.fi/nlp2019 • Pass 2/3 assignments • Grading available shortly after course Lecture slides • • Further reading recommendations: end of lectures We don’t expect state-of-the art, amazing systems! • • Assignment instructions & data We do expect you to • • try everything Moodle • • show understanding of lecture content • 9 10 MOODLE MOODLE FORUM Feel free to post • Discusss assignments / lecture content • Help other students • 11 12 MOODLE GLOSSARY READING MATERIAL Provided at end of each lecture • Not expected to read everything in evenings! • Further explanations of material • More details • Lots of new terminology Further reading to delve deeper • • We’ll add explanations during course • Add new entries (dummy content) to suggest addition • 13 14 READING MATERIAL READING MATERIAL Foundations of Statistical NLP Manning & Sch¨utze, 1999. Good reference for statistical topics Main course textbook: Speech and Language Processing Jurafsky & Martin, 2nd ed. NLP with Python, ‘The NLTK book’ Bird, Klein & Loper. New draft: https://web.stanford.edu/ jurafsky/slp3/ https://www.nltk.org/book/ Natural Language Processing Eisenstein. References to online draft where possible: J&M3 https://tinyurl.com/eisenstein-nlp Print version, 2nd edition: J&M2 Linguistic Fundamentals of NLP Bender, 2013. http://tinyurl.com/bender-nlp 15 16 PRE-COURSE QUESTIONNAIRE WHY NLP? https://presemo.helsinki.fi/nlp2019 Why do we need computers to understand Quick questionnaire • (or generate) human language? Your familiarity with topics • Not a test! AnonymousSame link: • ask questions during lecture People expect interactive agents to Fine if all answers are 1! We’ll learn about everything • • communicate in NL E.g. dialogue systems Rating Meaning • 1 Never heard of it Huge knowledge encoded in language • 2 Name familiar Hard to find: requires NLP • Automatic processing central to AI: 3 Basic familiarity (not much detail) • 4 Studied/read about before knowledge acquisition bottleneck Information extraction (more later) 5 Studied in detail • 17 20 WHY EVEN SIMPLE NLP IS HARD WHY NLP? Example Search • What is the forecast mean daytime temperature for Kumpula corpora, libraries, medical datasets, .... • tomorrow? Computational models of human processing • Tools for studying: Simple: answer in a database! • • language (corpus linguistics) • No reasoning/computation: just query sociology • • history... • Query Analysing human behaviour • SELECT day_mean FROM daily_forecast And much more! WHERE station = ’Helsinki Kumpula’ • AND date = ’2019-05-21’; 21 22 WHY EVEN SIMPLE NLP IS HARD WHY EVEN SIMPLE NLP IS HARD What is the forecast mean daytime SELECT day_mean FROM daily_forecast What is the forecast mean daytime SELECT day_mean FROM daily_forecast WHERE station = ’Helsinki Kumpula’ temperature for Kumpula tomorrow? temperature for Kumpula tomorrow? WHERE station = ’Helsinki Kumpula’ AND date = ’2019-05-21’; AND date = ’2019-05-21’; What is the forecast mean salary for What temperature is predicted in Kumpula tomorrow? ? Kumpula during the day tomorrow? What is the forecast mean salary for How hot will it be in Arabia tomorrow? CEOs tomorrow? ? Many ways to say the same thing Similar utterances mean very different things • • 23 24 WHY EVEN SIMPLE NLP IS HARD EXERCISE In small groups SELECT day_mean FROM daily_forecast: WHERE station = ’Helsinki Kumpula’: Look at sentences below What is the mean temperature in AND date = ’2019-05-21’; • Kumpula? Assume you: • SELECT day_mean FROM weekly_forecast are a computer • WHERE station = ’Helsinki Kumpula’ have database of logical/factual world knowledge • AND week = ’w22’; have lots of rules/statistics about English • What steps are involved in: • SELECT MEAN(day_temp) analysing this textual input? FROM weather_history • extracting & encoding relevant information? Ambiguity WHERE station = ’Helsinki Kumpula’ • • AND year = ’2019’; answering the question? Many forms • • Every level/step of analysis A robotic co-pilot developed under • ...? What agency has created a DARPA’s ALIAS programme has The big challenge of NLP computer that can pilot a plane? • already flown a light aircraft. 25 26 NATURAL LANGUAGE PROCESSING MACHINE TRANSLATION Language NLU Knowledge Text Knowledge representation representation Speech NLG Natural Language Understanding (NLU) NLU NLG Natural Language Generation (NLG) Language 1 Language 2 Mostly different models/algorithms text text • Some sharing possible • This course: mostly NLU • Not the standard approach NLG: day 6 • 27 28 MACHINE TRANSLATION STEPS OF NLU Intralingua Task: given sentence, get some representation computer can use for question answering NLU Phrases, syntax, NLG semantics, . John loves Mary Language 1 Direct translation Language 2 1. Divide into words (by spaces) text text 2. Identify John and Mary as names Potentially tricky 100k in English... MT pyramid 3. Recognise main relation loves ∼ Variety of approaches translate at different levels 4. Identify John as agent, Mary as patient Syntactic rules Large field: no more detail here. Plenty of courses available! 29 30 STEPS OF NLU STEPS OF NLU The number of moped-related crimes rose from 827 in 2012 to more than 23,000 last year. The list of people unhappy with this decision is decidedly longer and more comprehensive than the list that support the move. Extra difficulties: More difficult to segment words Reference to earlier context • • Varied vocabulary Repeated references • • More complex syntax Actual meaning requires inference! • • More complex meaning structure And then... (more later today) • Vagueness/ambiguity in meaning Ambiguity Disfluency • • • Noise Multiple languages • • 31 32 A BRIEF HISTORY OF NLP LESS NATURAL LANGUAGE? Why not just write a more natural query language? • 1600s Discussion of machine translation (MT), theoretical! Query Give me the daily mean from the forecast data 1930s Early proposals for MT using dictionaries for the station called ‘Helsinki Kumpula’ for 21.5.2019. 1950 Alan Turing: proposed ‘Turing Test’, depends on NLP 1954 Georgetown-IBM experiment: Still have to learn specialized language to interact • simple closed-domain MT, some grammatical rules Not good for non-expert users • 1957 Noam Chomsky: Syntactic Structures Natural interaction requires natural language • formal grammars, NLP becomes computable! Not just interaction • Extraction of information from (existing) text • 33 36 A BRIEF HISTORY OF NLP A BRIEF HISTORY OF NLP 1960s-70s Algorithms for parsing, semantic reasoning 1990s Statistical methods Formal representations of syntax, semantics, logic Statistical models for sub-tasks Hand-written rules 1987 Probabilistic n-gram language models 1964 ELIZA: simple dialogue system 1996 MT: IBM statistical, word-based models 1970 SHRDLU: narrow-domain system with NL commands 1997 Parsing: 1970 Augmented Transition Networks Probabilistic context-free grammars (PCFGS) automata for parsing text 1998 Distributional semantics (simeq word embeddings) 1980s More sophisticated parsing, semantics, reasoning ∼ Applications! 1999 Probabilistic (unsupervised) topic models 37 38 A BRIEF HISTORY OF NLP WORD FREQUENCIES Type Tokens 2010s More computing power, more data, more statistics , 11 341 the 5 792 Deep learning, neural networks, Bayesian models, ... I 5 087 2013 word2vec: word embeddings from lots of data and 4 708 ... 2014 RNNs for MT unhappy 5 2015 RNNs for NLG resolve 5 murderers 5 And so on... ... overwhelm 1 lamented 1 insufficient 1 39 41 WORD FREQUENCIES ZIPF’S LAW Most frequent: 4 , 10 10000 Log-log scale 103 Inverse log-log distribution of frequencies 8000 • Power law Next: • 6000 60002 Almost any linguistic phenomenon the 10 • Zipf’s law / Zipfian distribution 4000 • ‘Long tail’ A few things are very common 101 • Many things are