BigARTM Documentation Release 1.0 Konstantin Vorontsov May 04, 2019 Contents 1 Introduction 3 2 Downloads 5 3 Installation 7 3.1 Installation for Windows users......................................7 3.2 Installation for Linux and Mac OS-X users...............................8 3.3 Installation as Docker container..................................... 11 4 User’s Guide 13 4.1 Input Data Formats and Datasets..................................... 13 4.2 BigARTM Command Line Utility.................................... 16 4.3 Python Tutorial.............................................. 20 4.4 Python Guide............................................... 28 4.5 Regularizers Description......................................... 43 4.6 Scores Description............................................ 46 5 API References 49 5.1 Python Interface............................................. 49 5.2 C++ interface............................................... 84 5.3 C Interface................................................ 88 6 VisARTM 97 7 Release Notes 99 7.1 Changes in Python API......................................... 99 7.2 Changes in Protobuf Messages...................................... 103 7.3 Changes in BigARTM CLI........................................ 105 7.4 Changes in c_interface.......................................... 107 7.5 BigARTM v0.7.X Release Notes.................................... 108 8 BigARTM Developer’s Guide 121 8.1 Downloads (Windows).......................................... 121 8.2 Source code............................................... 122 8.3 Build C++ code on Windows...................................... 122 8.4 Python code on Windows........................................ 123 8.5 Compiling .proto files on Windows................................... 124 i 8.6 Working with iPython notebooks remotely............................... 124 8.7 Build C++ code on Linux........................................ 125 8.8 Creating New Regularizer........................................ 127 8.9 Code style................................................ 131 8.10 Release new version........................................... 131 8.11 Messages................................................. 132 Python Module Index 163 ii BigARTM Documentation, Release 1.0 Getting help • Learn more about BigARTM from IPython Notebooks, NLPub.ru, MachineLearning.ru and several publica- tions. • Search for information in the archives of the bigartm-users mailing list, or post a question. • Report bugs with BigARTM in our ticket tracker. • Try the Q&A – it’s got answers to many common questions. Contents 1 BigARTM Documentation, Release 1.0 2 Contents CHAPTER 1 Introduction Warning: Please note that this is a beta version of the BigARTM library which is still undergoing final testing before its official release. Should you encounter any bugs, lack of functionality or other problems with our library, please let us know immediately. Your help in this regard is greatly appreciated. This is the documentation for the BigARTM library. BigARTM is a tool to infer topic models, based on a novel technique called Additive Regularization of Topic Models. This technique effectively builds multi-objective models by adding the weighted sums of regularizers to the optimization criterion. BigARTM is known to combine well very different objectives, including sparsing, smoothing, topics decorrelation and many others. Such combinations of regularizers significantly improves several quality measures at once almost without any loss of the perplexity. Online. BigARTM never stores the entire text collection in the main memory. Instead the collection is split into small chunks called ‘batches’, and BigARTM always loads a limited number of batches into memory at any time. Parallel. BigARTM can concurrently process several batches, and by doing so it substantially improves the throughput on multi-core machines. The library hosts all computation in several threads withing a single process, which enables efficient usage of shared memory across application threads. Extensible API. BigARTM comes with an API in Python, but can be easily extended for all other languages that have an implementation of Google Protocol Buffers. Cross-platform. BigARTM is known to be compatible with gcc, clang and the Microsoft compiler (VS 2012). We have tested our library on Windows, Ubuntu and Fedora. Open source. BigARTM is released under the New BSD License. Acknowledgements. BigARTM project is supported by Russian Foundation for Basic Research (grants 14-07-00847, 14-07-00908, 14-07-31176), Skolkovo Institute of Science and Technology (project 081-R), Moscow Institute of Physics and Technology. 3 BigARTM Documentation, Release 1.0 Partners 4 Chapter 1. Introduction CHAPTER 2 Downloads • Windows – Latest 64 bit release: BigARTM_v0.10.0_win64 – Latest build from master branch: BigARTM_master_win64.7z (warning, use this with caution) – All previous releases are available at https://github.com/bigartm/bigartm/releases • Linux, Mac OS-X To run BigARTM on Linux and Mac OS-X you need to clone BigARTM repository (https://github.com/bigartm/ bigartm). • Docker container On any OS it is possible to run BigARTM as docker container, as described in Installation as Docker container 5 BigARTM Documentation, Release 1.0 6 Chapter 2. Downloads CHAPTER 3 Installation 3.1 Installation for Windows users 3.1.1 Download Download latest binary distribution of BigARTM from https://github.com/bigartm/bigartm/releases. Explicit down- load links can be found at Downloads section (64 bit only). The distribution will contain pre-build binaries, command-line interface and BigARTM API for Python. The distri- bution also contains a simple dataset. More datasets in BigARTM-compatible format are available in the Downloads section. You may also try BigARTM from pre-built docker container, as described in Installation as Docker container. 3.1.2 Use BigARTM from command line Download the latest package from https://github.com/bigartm/bigartm/releases. Unpack it to C:\BigARTM. Open command line or power shell and change working directory to C:\BigARTM\bin. Now you may use BigARTM command line utility bigartm.exe. Go to BigARTM Command Line Utility for examples and further directions. 3.1.3 Configure BigARTM Python API 1. Install Python, for example from Anaconda distribution (https://www.continuum.io/downloads). You are free to choose Python 2.7 or Python 3, both are supported with BigARTM. You have to choose 64 bit Python installation. 2. Install Python packages that BigARTM depend on. • numpy >= 1.9.2 (skip this if you install via Anaconda) • pandas >= 0.16.2 (skip this if you install via Anaconda) 7 BigARTM Documentation, Release 1.0 • protobuf >= 3.0.0 (not included in Anaconda; install by running pip install protobuf==3.0.0) • tqdm (not included in Anaconda; install by running pip install tqdm) 3. Add C:\BigARTM\bin folder to your PATH system variable, and add C:\BigARTM\python to your PYTHONPATH system variable: set PATH=%PATH%;C:\BigARTM\bin set PYTHONPATH=%PYTHONPATH%;C:\BigARTM\Python Remember to change C:\BigARTM if you unpack to another location. 4. Now you can used BigARTM from Python shell or from ipython notebook. Refer to Python Tutorial or Python Guide for examples, or to Python Interface for documentation. If you are getting errors when configuring or using Python API, please refer to Troubleshooting chapter in Installation for Linux and Mac OS-X users. The list of issues is common between Windows and Linux. Normally you should install protobuf for python with pip install. An alternative way is to install it from the package that comes with BigARTM. This is described here. 3.2 Installation for Linux and Mac OS-X users BigARTM had been tested on several Linux and MAC distributions, and it is known to work well on • Ubuntu 16.04.1 • Linux Mint 18 • Ubuntu 14.04.5 • Linux Mint 17.3 • Arch Linux • Manjaro • Fedora 24 • openSUSE Leap 42.1 To install BigARTM you should build it from source code. Or, you may run BigARTM from pre-built docker container, as described in Installation as Docker container. If you are looking for old instructions, refer to Build C++ code on Linux. 3.2.1 Script to install BigARTM on Ubuntu The following script is tested with Ubuntu 14.04. Scroll further below for more OS-specific insructinos. # Step 1. Update and install dependencies apt-get --yes update apt-get --yes install git make cmake build-essential libboost-all-dev # Step 2. Insall python packages apt-get --yes install python-numpy python-pandas python-scipy wget https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install protobuf tqdm wheel (continues on next page) 8 Chapter 3. Installation BigARTM Documentation, Release 1.0 (continued from previous page) # Step 3. Clone repository and build git clone --branch=stable https://github.com/bigartm/bigartm.git cd bigartm mkdir build&& cd build cmake .. make # Step 4. Install BigARTM make install export ARTM_SHARED_LIBRARY=/usr/local/lib/libartm.so # Alternative step 4 - installing only the Python package pip install python/bigartm*.whl Now you should be able to use BigARTM command line utility (try bigartm --help), or run Bi- gARTM from python, like this: import artm; print(artm.version()); print(artm. ARTM(num_topics=10).info). 3.2.2 Step 1. Install system dependencies Ubuntu, Linux Mint: sudo apt-get install git build-essential libboost-all-dev sudo apt-get install cmake # For Ubuntu 16.04 and Linux Mint 18 For Ubuntu 14.04 and Linux Mint 17.3 ubstakk cmake from PPA: sudo add-apt-repository