Data Extraction from Charts via Single Deep Neural Network Xiaoyi Liu 1 Diego Klabjan 1 Patrick N Bless 2 Abstract charts (Savva et al., 2011; Huang & Tan, 2007) based on traditional computer vision methods, which rely on compli- Automatic data extraction from charts is challeng- cated human-defined rules and thus are not robust. With the ing for two reasons: there exist many relations proliferation of deep learning, it is conceivable that the accu- among objects in a chart, which is not a common racy of chart component detection can be improved without consideration in general computer vision prob- complicated rules, i.e., by using raw images as input with no lems; and different types of charts may not be feature engineering or employment of other rules. Despite processed by the same model. To address these of this belief there is still lack of a single deep learning problems, we propose a framework of a single model for data extraction from charts. deep neural network, which consists of object detection, text recognition and object matching We introduce a deep learning solution that automatically ex- modules. The framework handles both bar and tracts data from bar and pie charts and essentially converts a pie charts, and it may also be extended to other chart to a relational data table. The approach first detects the types of charts by slight revisions and by aug- type of a chart (bar or pie), and then employs a single deep menting the training data. Our model performs learning model that extracts all of the relevant components successfully on 79.4% of test simulated bar charts and data. There is one single model for bar charts and a and 88.0% of test simulated pie charts, while for different one for pie charts. The entire framework has three charts outside of the training domain it degrades stages: 1. chart type identification, 2. element detection, for 57.5% and 62.3%, respectively. text recognition, and bounding box matching through which the actual numerical data is extracted, and 3. inference. The first phase is a standard image classification problem. The 1. Introduction most interesting part is the second phase where we rely on the Faster-RCNN model (Ren & Sun, 2015). We add several “Data everywhere information nowhere” is a common say- components to the feature maps of regional proposals, e.g., ing in the business world. Consider all of the presentations text detection and recognition. The most significant part and reports lingering in folders of a company. They are is the addition of relation network components that match, embellished with eye-appealing charts as images forming e.g., part of the legend with a matching bar, a bar with the formidable data, but getting information from these charts y-axis value, a slice in the pie chart with part of the legend. is challenging. To this end, a system that automatically In order to make extraction from pie charts work, additional extracts information from charts would provide great bene- novel tricks are needed; e.g., the model detects the angle fits in knowledge management within the company. Such of each slice by attaching an RNN to regional proposals knowledge can be combined with other data sets to further (since slices form a sequence when traversed in a clockwise arXiv:1906.11906v1 [cs.CV] 6 Jun 2019 enhance business value. We address this problem by devel- manner), multiplies the feature map matrix of the regional oping a deep learning model that takes an image of a chart proposal of the entire pie with an angle-dependent rotation as input and it extracts information in the form of categories matrix, and then uses this rotated matrix in the relation net- being displayed, the relevant text such as the legend, axis work. The last inference phase is using heuristics to produce labels, and numeric values behind the data displayed. the final objects and data. There are existing tools for information extraction from The model is trained on simulated charts based on Microsoft Excel and the Matplotlib library. It is then evaluated on 1Department of Industrial Engineering and Manage- ment Science, Northwestern University, Evanston, USA simulated test data, the Microsoft FigureQA charts data set 2Intel Corporation, Chandler, USA. Correspondence to: (Kahou & Bengio, 2017), and manually inspected charts Xiaoyi Liu <[email protected]>, Diego from Google Images. The results on the simulated test set Klabjan <[email protected]>, Patrick N Bless show our model performs successfully on 79.4% simulated <[email protected]>. bar charts and 88.0% on simulated pie charts. On charts from FigureQA and Google Images the performance drops Data Extraction from Charts via Single Deep Neural Network Histograms of Oriented Gradients and the Scale Invariant Feature Transform descriptors for feature extraction and Support Vector Machine (SVM) for classification. Savva et al. (2011) have built a system called Revision to redesign charts, which includes chart classification, data extraction and visualization. Low-level image features are extracted for chart classification by SVMs. In recent years, deep learning techniques have made great progress in general image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2012; He & Sun, 2016), which can be applied to chart classification. Among these methods, convolutional neural networks based methods are (a) Bar chart extraction (b) Pie chart extraction the most widely used, and Siegel et al. (2016) have trained both AlexNet (Krizhevsky et al., 2012) and ResNet-50 (He Figure 1. Framework for charts data extraction & Sun, 2016) on their annotated datasets including 60,000 charts and 7 categories. VGG16 (Simonyan & Zisserman, to 57.5% and 62.3% for bar and pie charts, respectively. 2012) is employed in our system. Our main contributions are as follows. A chart includes a set of structural texts and data. In bar charts, texts can be categorized as title, axis-title, axis-tick • We propose a single deep learning model that extracts or legend, and data information is encoded into the height information from bar charts. The new ideas of the or width of bars. To extract the textual and graphical infor- model are the combination of text recognition, text mation from bar charts, one must firstly detect the bounding detection, and pairwise matching of components within boxes of texts and bars. Object detection is a common a single model. In particular, we design a new approach problem in computer vision. With the development of deep for matching candidate components, e.g, an actual bar learning techniques, there are two main kinds of methods: bounding box with an entry in the legend. 1. RCNN (Girshick & Malik, 2014) includes two stages of generating region proposals and subsequent classification; 2. • We also propose another single deep learning model for derivations fast-RCNN (Girshick, 2015) and Faster-RCNN pie charts. This model introduces an RNN component (Ren & Sun, 2015) of RCNN, YOLO (Redmon & Farhadi, to detect angles in a pie chart, and a different strategy 2016) and SSD (Liu & Berg, 2016) use only one stage for matching non-rectangular patches. including both region proposing and classification, which usually perform better on training speed but worse on accu- • We use a pipeline where we first identify a chart type racy of bounding box prediction (Huang & Murphy, 2017). by standard CNN-based classification. Once the chart It is worth pointing out that Faster-RCNN produces higher type is identified, we employ one of the aforementioned accuracy than YOLO and SSD at the expense of a higher models to extract information. training time. There are also some specially designed mod- els (Tian & Qiao, 2016; Shi & Belongie, 2017) which only In Section2, related work and methods for charts data extrac- focus on text detection. Tian et al. (2016) use an anchor box tion are reviewed. We show all components of our model method to predict text bounding boxes. Shi et al. (2017) and inference methods in Section3. The computational introduce a segment linking method that can handle oriented results are discussed in Section4. text detection. In terms of chart component detection, there are many works 2. Literature Review done with traditional computer vision techniques. Zhou et Automated chart analysis has been studied for many years, al. (2000) combined Hough transform and boundary tracing and the process of extracting data from charts in documents to detect bars. Huang et al. (2007) have employed rules to can be divided into four steps: chart localization and extrac- detect chart components using edge maps. In (Savva et al., tion, chart classification, text and element detection, data 2011), bars or pies are detected by their shapes and color reconstruction. Our work focuses on the last three steps. information in pixels. By using deep learning techniques, all the texts and chart components can be detected in a model Chart classification is a specific kind of image classification automatically. There are already some works based on deep problems. In 2007, Prasad et al. (2007) have presented learning techniques; Cliche et al. (2017) have trained three a traditional computer vision-based approach to classify separate object detection models ReInspect (Stewart & Ng, charts in five categories. This approach is based on the Data Extraction from Charts via Single Deep Neural Network 2016) to detect tick marks, tick labels and points in different 3.1. Background resolutions, which are finally combined to extract data from 3.1.1. FASTER-RCNN a scatterplot. Poco et al. (2017) have employed a CNN to classify each pixel as text or not in a chart and then remove Faster RCNN uses a single convolutional neural network to all non-text pixels. create feature maps for each predefined regional proposal Data reconstruction is followed after chart component de- called also anchor proposal and predicts both a class label tection.