Thrift Tutorial Release 1.0 Stratos Dimopoulos Sep 21, 2017 Contents 1 Introduction to Thrift 3 2 Thrift installation 5 3 Thrift protocol stack 9 4 Thrift type system 13 5 Writing a .thrift file 15 6 Thrift by Example 19 7 Indices and tables 51 i ii Thrift Tutorial, Release 1.0 The purpose of this document is to describe step by step Thrift’s installation and give simple examples of its usage. Contents 1 Thrift Tutorial, Release 1.0 2 Contents CHAPTER 1 Introduction to Thrift Thrift is a lightweight, language-independent software stack with an associated code gen- eration mechanism for RPC. Thrift provides clean abstractions for data transport, data se- rialization, and application level processing. Thrift was originally developed by Facebook and now it is open sourced as an Apache project. Apache Thrift is a set of code- generation tools that allows developers to build RPC clients and servers by just defining the data types and service interfaces in a sim- ple definition file. Given this file as an input, code is generated to build RPC clients and servers that communicate seamlessly across programming languages. In this tutorial I will describe how Thrift works and provide a guide for build and in- stallation steps, how to write thrift files and how to generate from those files the source code that can be used from different client libraries to communicate with the server. Thrift supports a variety of languages includ- ing C++, Java, Python, PHP, Ruby but for simplicity I will focus this tutorial on exam- ples that include Java and Python. 3 Fig. 1.1: Thrift Architecture. Image from wikipedia.org Thrift Tutorial, Release 1.0 Fig. 1.2: Services using Thrift 4 Chapter 1. Introduction to Thrift CHAPTER 2 Thrift installation Detailed information on how to install Thrift can be found here: http://thrift.apache.org/docs/install/ On Ubuntu Linux for example you just need to first install the dependencies and then you are ready to install Thrift. 1. Install the languages with which you plan to use thrift. To use with Java for example, install a Java JDK you prefer. In this demo I am using Oracle JDK 7 for Ubuntu, but you shouldn’t have problem using the one you like. • To use with Java you will also need to install Apache Ant sudo apt-get install ant 2. Installing required tools and libraries: sudo apt-get install libboost-dev libboost-test-dev libboost-program- ,!options-dev libboost-filesystem-dev libboost-thread-dev libevent-dev ,!automake libtool flex bison pkg-config g++ libssl-dev • You can check for specific requirements for each language you wish to use here: http://thrift.apache. org/docs/install/ 1. Download Thrift: http://thrift.apache.org/download 2. Copy the downloaded file into the desired directory and untar the file tar-xvf thrift-0.9.3.tar.gz 3. For detailed instuctions on how to build Apache Thrift on your specific system you can read here: http://thrift.apache.org/docs/BuildingFromSource • For an Ubuntu linux distribution you just need to go to the thrift directory and type: ./bootstrap.sh ./configure • At the end of the output you should be able to see a list of all the libraries that are currently built in your system and ready to use with your desired programming languages. If a component is missing you should download the missing language and repeat the above step. 5 Thrift Tutorial, Release 1.0 thrift 0.9.3 Building C++ Library......... : yes Building C (GLib) Library.... : no Building Java Library........ : yes Building C# Library .......... : no Building Python Library...... : yes Building Ruby Library........ : no Building Haskell Library..... : no Building Perl Library........ : no Building PHP Library......... : no Building Erlang Library...... : no Building Go Library.......... : no Building D Library........... : no C++ Library: Build TZlibTransport...... : yes Build TNonblockingServer.. : yes Build TQTcpServer (Qt).... : no Java Library: Using javac............... : javac Using java................ : java Using ant.................:/usr/bin/ant Python Library: Using Python..............:/usr/bin/python • Here http://thrift.apache.org/docs/install/debian/ you can find all the packages you might need to support your desired language in case some of them are missing. • On the same directory run make to build Thrift sudo make • (Optional) Run the test suite if you want sudo make check • And finally you are ready to install Thrift by running sudo make install Verify installation Now your Thrift installation is completed! To verify that you have succesfully installed Thrift just type thrift-version and you should be able to see something like the following: Thrift version 0.9.0 Additionaly you can go to the tutorial directory and follow the instructions located on the README files on each of the targeted languages directories. For example for JAVA you should be able to verify the following: 6 Chapter 2. Thrift installation Thrift Tutorial, Release 1.0 thrift/tutorial/java$ file ../../lib/java/build/libthrift-${version}-${release}.jar ../../lib/java/libthrift.jar: Zip archive data, at least v1.0 to extract thrift/tutorial/java$ file ../../compiler/cpp/thrift ../../compiler/cpp/thrift: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), ,!dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped thrift/tutorial/java$ ls ../../lib/java/build/lib/ commons-lang-2.5.jar junit-4.4.jar servlet-api-2.5.jar slf4j-api-1.5.8.jar slf4j- ,!simple-1.5.8.jar 2.1. Verify installation 7 Thrift Tutorial, Release 1.0 8 Chapter 2. Thrift installation CHAPTER 3 Thrift protocol stack To understand Thrift’s protocol stack I highly recommend to take a look at the white-paper: http://thrift.apache.org/static/files/thrift-20070401.pdf and to the Thrift Architecture part of this very nice description here: http://jnb.ociweb.com/jnb/jnbJun2009.html - Part of the following con- tent is directly derived from these sources. I will now briefly present Thrift’s architecture. Runtime Library The protocol and transport layer are part of the runtime library. This means that it is pos- sible to define a service and change the pro- tocol and transport without recompiling the code. Protocol Layer The protocol layer provides serialization and deserialization. Thrift supports the following protocols : • TBinaryProtocol - A straight-forward binary format encoding numeric val- ues as binary, rather than converting to text. • TCompactProtocol - Very efficient, dense encoding of data (See details be- low). • TDenseProtocol - Similar to TCom- pactProtocol but strips off the meta 9 Fig. 3.1: Thrift Architecture. Image from wikipedia.org Thrift Tutorial, Release 1.0 information from what is transmitted, and adds it back in at the receiver. TDenseProtocol is still experimental and not yet available in the Java im- plementation. • TJSONProtocol - Uses JSON for en- coding of data. • TSimpleJSONProtocol - A write-only protocol using JSON. Suitable for parsing by scripting languages • TDebugProtocol - Uses a human- readable text format to aid in debug- ging. Tranport Layer The transport layer is responsible for reading from and writing to the wire. Thrift supports the following: • TSocket - Uses blocking socket I/O for transport. • TFramedTransport - Sends data in frames, where each frame is preceded by a length. This transport is required when using a non-blocking server. • TFileTransport - This transport writes to a file. While this transport is not in- cluded with the Java implementation, it should be simple enough to imple- ment. • TMemoryTransport - Uses memory for I/O. The Java implementation uses a simple ByteArrayOutputStream in- ternally. • TZlibTransport - Performs compres- sion using zlib. Used in conjunction with another transport. Not available in the Java implementation. Processor The processor takes as arguments an input and an output protocol. Reads data from the input, processes the data throught the Handler specified by the user and then writes the data to the output. 10 Chapter 3. Thrift protocol stack Thrift Tutorial, Release 1.0 Supported Servers A server will be listening for connections to a port and will send the data it receives to the Processor to handle. • TSimpleServer - A single-threaded server using std blocking io. Useful for testing. • TThreadPoolServer - A multi-threaded server using std blocking io. • TNonblockingServer - A multi-threaded server using non-blocking io (Java implementation uses NIO channels). TFramedTransport must be used with this server. 3.3. Supported Servers 11 Thrift Tutorial, Release 1.0 12 Chapter 3. Thrift protocol stack CHAPTER 4 Thrift type system The thrift type system includes base types like bool, byte, double, string and integer but also special types like binary and it also supports structs (equivalent to classes but without inheritance) and also containers (list, set, map) that correspond to commonly available interfaces in most programming languages. The type system focuses on key types available in all programming languages and omits types that are specific to only some programming languages. • A detailed reference to Thrift type system follows next and more details can be found here: http://thrift.apache. org/docs/types • If you want to check the Thrift interface description language that allows for the definition of Thrift types you can read here: http://thrift.apache.org/docs/idl Base types • bool: A boolean value (true or false) • byte: An 8-bit signed integer • i16: A 16-bit signed integer • i32: A 32-bit signed integer • i64: A 64-bit signed integer • double: A 64-bit floating point number • string: A text string encoded using UTF-8 encoding Note: There is no support for unsigned integer types, due to the fact that there are no native unsigned integer types in many programming languages. Signed integers can be safely cast to their unsigned counterparts when necessary. Special Types binary: a sequence of unencoded bytes 13 Thrift Tutorial, Release 1.0 Structs A struct has a set of strongly typed fields, each with a unique name identifier.