Automated Knowledge Base Construction (2021) Conference paper NatCat: Weakly Supervised Text Classification with Naturally Annotated Resources Zewei Chu [email protected] University of Chicago, Chicago, IL 60637, USA Karl Stratos [email protected] Rutgers University, Piscataway, NJ 08854, USA Kevin Gimpel [email protected] Toyota Technological Institute at Chicago, Chicago, IL 60637, USA Abstract We describe NatCat, a large-scale resource for text classification constructed from three data sources: Wikipedia, Stack Exchange, and Reddit. NatCat consists of document- category pairs derived from manual curation that occurs naturally within online communities. To demonstrate its usefulness, we build general purpose text classifiers by training on NatCat and evaluate them on a suite of 11 text classification tasks (CatEval), reporting large improvements compared to prior work. We benchmark different modeling choices and resource combinations and show how tasks benefit from particular NatCat data sources.1 1. Introduction Websites with community contributed content contain ample knowledge for natural lan- guage processing. In this paper, we seek to improve text classification by leveraging this knowledge in the form of texts paired with natural category annotations. In particular, we present NatCat, a large-scale resource constructed automatically from three data sources: Wikipedia, Stack Exchange,2 and Reddit.3 With rich knowledge about text and categories, we show that NatCat is a useful resource for text classification. To demonstrate the usefulness of NatCat, we use it to train models that compute a score for any document-category pair. As a result, we get weakly supervised (i.e., \dataless"; Chang et al., 2008) text classifiers that can be used off-the-shelf to produce interpretable and relevant topics for any document. They can also be effortlessly ported to a specific topic classification task by computing the score for only the labels in the task. Table1 illustrates the use of a NatCat-trained model on a document from AGNews. To evaluate, we propose CatEval, a standardized benchmark for evaluating weakly supervised text classification with a choice of datasets, label descriptions for each dataset, and baseline results. CatEval comprises a diverse choice of 11 text classification tasks including both topic and sentiment labels, and contains both single and multi-label classification tasks. We show that NatCat is a valuable resource for weakly supervised text classification and study the impact of data domain and pretrained model choice on tasks in CatEval. We analyze the gap between weakly supervised and supervised models and show that the 1. NatCat is available at https://github.com/ZeweiChu/NatCat 2. https://stackexchange.com/ 3. https://www.reddit.com/ 1 Chu, Stratos, & Gimpel Israeli ambassador calls peace conference idea `counterproductive'. A broad Text international peace conference that has reportedly been suggested by Egypt could be \counterproductive" and shouldn't be discussed until after ... AGNews international, sports, science technology, business NatCat (Wiki.) invasions, diplomats, peace, diplomacy, environmentalism, Egypt, patriotism ... Table 1: An example from AGNews, showing the text, the provided AGNews categories (true class in bold), and predicted categories from a text classifier trained on the Wikipedia version of NatCat (in decreasing order by score). The NatCat-trained model can be used to score any document-category pair, so it can be applied to the AGNews task by scoring each category and choosing the one with the highest score. The AGNews row shows the category ordering from the NatCat classifier. mistakes of our models are reasonable and humanlike, suggesting that the construction process of NatCat is a promising approach to mining data for building text classifiers. 2. The NatCat Resource In this section, we describe the creation of the NatCat resource. NatCat is constructed from three different data sources with natural category annotation: Wikipedia, Stack Exchange, and Reddit. Therefore, NatCat naturally contains a wide range of world knowledge that is useful for topical text classification. For each data source, we describe ways of constructing document-category pairs in which the document can be labeled with the category. Wikipedia. Wikipedia documents are annotated with categories by the community con- tributors. The categories of each Wikipedia document can be found at the bottom of the page. We obtained Wikipedia documents from Wikimedia Downloads. Wikipedia page-to-category mappings were generated from Wiki SQL dumps using the \categorylinks" and \page" tables. We removed hidden categories by SQL filtering, which are typically maintenance and tracking categories that are unrelated to the document content. We also removed disambiguation categories. After filtering, there are 5.75M documents with at least one category, and a total of 1.19M unique categories. We preprocessed the Wikipedia articles by removing irrelevant information such as the external links at the end of each article. We then removed Wikipedia documents with fewer than 100 non-stopwords. Some category names are lengthy and specific, e.g., \Properties of religious function on the National Register of Historic Places in the United States Virgin Islands". These categories are unlikely to be as useful for end users or downstream applications as shorter and more common categories. Therefore, we consider multiple ways of augmenting the given categories with additional categories. The first way is to use a heuristic method of breaking long category names into shorter ones. We first use stopwords as separators and keep each part of the non-stopword word sequence as a category name. For each category name of a document, we also run a named entity recognizer [Honnibal and Montani, 2017] to find all named entities in that category 2 NatCat: Weakly Supervised Text Classification Wikipedia Stack Exchange Reddit # categories 1,730,447 156 3,000 # documents 2,800,000 2,138,022 7,393,847 avg. # cats. per doc. 86.9 1 1 mode # cats. per doc. 46 1 1 avg. # words per doc. 117.9 58.6 11.4 Table 2: Statistics of training sets sampled from NatCat for each of its three data sources. name, and add them to the category set of the document. This way we expand the existing category names from Wikipedia. For the example category above, this procedure yields the following categories: \religious function", \the national register of historic places", \properties", \historic places", \the united states virgin islands", \properties of religious function on the national register of historic places in the united states virgin islands", \united states virgin islands", and \national register". Our second method of expansion is based on the fact that Wikipedia categories can have parent categories and therefore form a hierarchical structure. When expanding the category set by adding its ancestors, there is a trade-off between specificity/relevance and generality/utility of category names. Using only the categories provided for the article yields a small set of high-precision, specific categories. Adding categories that are one or two edges away in the graph increases the total number of training pairs and targets more general/common categories, but some of them will be less relevant to the article. In NatCat, we include all categories of documents that are up to two edges away. Stack Exchange. Stack Exchange is a question answering platform where users post and answer questions as a community. Questions on Stack Exchange fall into 308 subareas, each area having its own site. We create document-category pairs by pairing question titles or descriptions with their corresponding subareas. Question titles, descriptions, and subareas are available from Chu et al.[2020]. Many Stack Exchange subareas have their own corresponding \meta" sites. A meta site is meant to discuss the website itself regarding its policy, community, and bugs, etc. When creating this dataset, we merge the subareas with their corresponding meta areas. This gives us over 2 million documents with 156 categories. Reddit. Inspired by Puri and Catanzaro[2019], we construct a category classification dataset from Reddit. In our dataset, we propose to classify Reddit post titles to their corresponding subreddit names. We use the OpenWebText4 toolkit to get Reddit posts with more than 3 karma and their subreddit names. We keep only the top 3000 most frequent subreddits as they better capture the common categories that we are interested in.5 This gives us over 7 million documents with 3000 categories. 4. https://github.com/jcpeterson/openwebtext 5. Subreddit names are generally more noisy than categories in the other two data sources, and many are abbreviations or shorthand that are meaningful only to particular groups. Wikipedia category names are more formal and understandable by most people. Hence we decide to only keep the most common categories from Reddit. Some example subreddit names (with ranking by frequency in parentheses): 3 Chu, Stratos, & Gimpel dataset # test docs. # labels # sents./doc. # words/doc. # words/sent. AGNews 7,600 4 1.3 48.8 36.8 DBpedia 70k 14 2.4 58.7 24.4 Yahoo 60k 10 5.7 115.8 20.3 20 News Groups 7,532 20 15.9 375.4 Emotion 16k 10 1.6 19.5 12.4 SST-2 1,821 2 1.0 19.2 19.1 Yelp-2 38k 2 8.4 155.1 18.4 Amazon-2 400k 2 4.9 95.7 19.5 NYTimes 10k 100 30.0 688.3 22.9 Comment 1,287 28 1.3 13.8 10.5 Situation 3,525 12 1.8 44.0 24.7 Table 3: Statistics of CatEval datasets. Constructed by these three data sources, NatCat covers a wide range of topics and world knowledge. Table2 summarizes statistics of training sets we sampled from NatCat. Note that all documents from Stack Exchange and Reddit have only one associated category, while a document from Wikipedia may have multiple categories describing it. 3. CatEval Tasks To evaluate NatCat, we will use it to build general purpose text classifiers and test them on a variety of text classification tasks.