NLTK: The Natural Language Toolkit
Steven Bird Edward Loper Department of Computer Science Department of Computer and Software Engineering and Information Science University of Melbourne University of Pennsylvania Victoria 3010, Australia Philadelphia PA 19104-6389, USA [email protected] [email protected]
Abstract an interpreted language, Python facilitates interac- tive exploration. As an object-oriented language, The Natural Language Toolkit is a suite of program mod- Python permits data and methods to be encapsulated ules, data sets, tutorials and exercises, covering symbolic and re-used easily. Python comes with an extensive and statistical natural language processing. NLTK is standard library, including tools for graphical pro- written in Python and distributed under the GPL open gramming and numerical processing. The recently source license. Over the past three years, NLTK has added generator syntax makes it easy to create inter- become popular in teaching and research. We describe active implementations of algorithms (Loper, 2004; the toolkit and report on its current state of development. Rossum, 2003a; Rossum, 2003b). 1 Introduction The Natural Language Toolkit (NLTK) was developed in conjunction with a computational linguistics course at the University of Pennsylvania in 2001 (Loper and Bird, 2002). It was designed with three pedagogical applications in mind: assignments, demonstrations, and projects. Assignments. NLTK supports assignments of varying difficulty and scope. In the simplest assign- ments, students experiment with existing compo- nents to perform a wide variety of NLP tasks. As students become more familiar with the toolkit, they can be asked to modify existing components, or to create complete systems out of existing compo- nents. Figure 1: Interactive Chart Parsing Demonstration Demonstrations. NLTK’s interactive graphical demonstrations have proven to be very useful for students learning NLP concepts. The 2 Design demonstrations give a step-by-step execution NLTK is implemented as a large collection of of important algorithms, displaying the current minimally interdependent modules, organized state of key data structures. A screenshot of the into a shallow hierarchy. A set of core modules chart parsing demonstration is shown in Figure 1. defines basic data types that are used throughout the Projects. NLTK provides students with a flexible toolkit. The remaining modules are task modules, framework for advanced projects. Typical projects each devoted to an individual natural language might involve implementing a new algorithm, processing task. For example, the nltk.parser developing a new component, or implementing a module encompasses to the task of parsing, or new task. deriving the syntactic structure of a sentence; We chose Python because it has a shallow learn- and the nltk.tokenizer module is devoted to ing curve, its syntax and semantics are transparent, the task of tokenizing, or dividing a text into its and it has good string-handling functionality. As constituent parts. 2.1 Tokens and other core data types 2.2 The corpus module To maximize interoperability between modules, we Many language processing tasks must be developed use a single class to encode information about nat- and tested using annotated data sets or corpora. ural language texts – the Token class. Each Token Several such corpora are distributed with NLTK, instance represents a unit of text such as a word, as listed in Table 1. The corpus module defines sentence, or document, and is defined by a (partial) classes for reading and processing many of these mapping from property names to values. For exam- corpora. The following code fragment illustrates ple, the TEXT property is used to encode a token’s how the Brown Corpus is accessed. text content:1 >>> from nltk.corpus import brown >>> from nltk.token import * >>> brown.groups() >>> Token(TEXT="Hello World!") ['skill and hobbies', 'popular lore',
Table 1: Corpora and Corpus Samples Distributed with NLTK must be initialized by training on a tagged corpus function, and variable in the toolkit. Technical before it can be used. Third, subclassing can be used reports explain and justify the toolkit’s design and to create specialized versions of a given algorithm. implementation. All are available from http:// Each processing module defines an interface nltk.sf.net/docs.html. for its task. Interface classes are distinguished by naming them with a trailing capital “I,” such as 3 Installing NLTK ParserI. Each interface defines a single action NLTK is available from nltk.sf.net, and is method which performs the task defined by the packaged for easy installation under Unix, Mac interface. For example, the ParserI interface OS X and Windows. The full distribution consists defines the parse method and the Tokenizer of four packages: the Python source code (nltk); interface defines the tokenize method. When the corpora (nltk-data); the documentation appropriate, an interface defines extended action (nltk-docs); and third-party contributions methods, which provide variations on the basic (nltk-contrib). Before installing NLTK, it is action method. For example, the ParserI interface necessary to install Python version 2.3 or later, defines the parse n method which finds at most n available from www.python.org. Full installation parses for a given sentence; and the TokenizerI instructions and a quick start guide are available interface defines the xtokenize method, which from the NLTK homepage. outputs an iterator over subtokens instead of a list As soon as NLTK is installed, users can run the of subtokens. demonstrations. On Windows, the demonstrations NLTK includes the following modules: can be run by double-clicking on their Python cfg, corpus, draw (cfg, chart, corpus, source files. Alternatively, from the Python featurestruct, fsa, graph, plot, rdparser, interpreter, this can be done as follows: srparser, tree), eval, featurestruct, parser (chart, chunk, probabilistic), probability, sense, set, stemmer (porter), >>> import nltk.draw.rdparser tagger, test, token, tokenizer, tree, and >>> nltk.draw.rdparser.demo() util. Please see the online documentation for >>> nltk.draw.srparser.demo() >>> nltk.draw.chart.demo() details. 2.4 Documentation 4 Using and contributing to NLTK Three different types of documentation are avail- NLTK has been used at the University of Pennsylva- able. Tutorials explain how to use the toolkit, with nia since 2001, and has subsequently been adopted detailed worked examples. The API documentation by several NLP courses at other universities, includ- describes every module, interface, class, method, ing those listed in Table 2. Third party contributions to NLTK include: Graz University of Technology, Austria Brill tagger (Chris Maloof), hidden Markov model Information Search and Retrieval tagger (Trevor Cohn, Phil Blunsom), GPSG-style Macquarie University, Australia feature-based grammar and parser (Rob Speer, Bob Intelligent Text Processing Berwick), finite-state morphological analyzer (Carl Massachusetts Institute of Technology, USA de Marcken, Beracah Yankama, Bob Berwick), Natural Language Processing decision list and decision tree classifiers (Trevor National Autonomous University of Mexico, Mexico Cohn), and Discourse Representation Theory Introduction to Natural Language Processing implementation (Edward Ivanovic). in Python Ohio State University, USA NLTK is an open source project, and we wel- Statistical Natural Language Processing come any contributions. There are several ways University of Amsterdam, Netherlands to contribute: users can report bugs, suggest fea- Language Processing and Information Access tures, or contribute patches on Sourceforge; users University of Colorado, USA can participate in discussions on the NLTK-Devel Natural Language Processing mailing list2 or in the NLTK public forums; and University of Edinburgh, UK users can submit their own NLTK-based projects Introduction to Computational Linguistics for inclusion in the nltk contrib directory. New University of Magdeburg, Germany code modules that are relevant, substantial, orig- Natural Language Systems inal and well-documented will be considered for University of Malta, Malta inclusion in NLTK proper. All source code is dis- Natural Language Algorithms tributed under the GNU General Public License, and University of Melbourne, Australia all documentation is distributed under a Creative Human Language Technology Commons non-commercial license. Thus, poten- University of Pennsylvania, USA tial contributors can be confident that their work Introduction to Computational Linguistics will remain freely available to all. Further infor- University of Pittsburgh, USA mation about contributing to NLTK is available at Artificial Intelligence Application Development http://nltk.sf.net/contrib.html. Simon Fraser University, Canada Computational Linguistics
5 Conclusion Table 2: University Courses using NLTK NLTK is a broad-coverage natural language toolkit that provides a simple, extensible, uniform frame- References work for assignments, demonstrations and projects. Edward Loper and Steven Bird. 2002. NLTK: It is thoroughly documented, easy to learn, and sim- The Natural Language Toolkit. In Proceedings ple to use. NLTK is now widely used in research of the ACL Workshop on Effective Tools and and teaching. Readers who would like to receive Methodologies for Teaching Natural Language occasional announcements about NLTK are encour- Processing and Computational Linguistics, pages aged to sign up for the low-volume, moderated mail- 62–69. Somerset, NJ: Association for Computa- 3 ing list NLTK-Announce. tional Linguistics. http://arXiv.org/abs/ cs/0205028. 6 Acknowledgements Edward Loper. 2004. NLTK: Building a pedagogi- cal toolkit in Python. In PyCon DC 2004. Python We are indebted to our students and colleagues for Software Foundation. http://www.python. feedback on the toolkit, and to many contributors org/pycon/dc2004/papers/. listed on the NLTK website. Guido Van Rossum. 2003a. An Introduction to Python. Network Theory Ltd. Guido Van Rossum. 2003b. The Python Language 2http://lists.sourceforge.net/ Reference. Network Theory Ltd. lists/listinfo/nltk-devel 3http://lists.sourceforge.net/ lists/listinfo/nltk-announce