Introduction to MS-HMSHTTP

Bin Guo SQL OIPI Test Team 1 Background & Overview

Agenda 2 Transport Direction & Stack

3 Message Structure & Examples

4 Generation Demo

5 Resource Protocol Background and Overview SQL Server 2019 new feature: Big Data Cluster

Apache Hadoop Big Data Cluster Hive Metastore cluster HTTP Server • • Hive Metastore • Hive Metadata Service Database Derby SQL Server Database MS-HMSHTTP

Client MSThe-HiveHMSHTTPMetastoreHTTPprotocol Comparison with Apache Hive IMetaStoreClient

Hive MS-HMSHTTP IMetaStoreClient

Lightweight Hive All kinds of Metastore management functions Interface Protocol Transport Direction and Layers Knox, the Gateway in

HTTP Post to https://{clusterIp}:{knoxPort}/path

https://ClusterIP:30443/gateway/default/hmshttpthrift/api/hms Transport Stack (Layers) Message Structure and Examples All MS-HMSHTTP Message Operations

• get_all_databases • get_all_tables • get_partition_names • get_databases • get_tables • get_partitions • get_database • get_table • get_tables_by_type Message Data Structures

• Database • MetaException • Table • NoSuchObjectException • Partition struct Database { 1: string name, 2: string description, 3: string locationUri, 4: map parameters, // properties associated with the database 5: optional PrincipalPrivilegeSet privileges, 6: optional string ownerName, 7: optional PrincipalType ownerType } Message body in TJSON array format

Position 1 Position 2 Position 3: Position 4 Position 5

• version • message • message • message • payload name type seqid

message type

1: CALL 2: REPLY 3: EXCEPTION 4: ONEWAY Request body Example

Position 1 Position 2 Position 3: Position 4 Position 5

• version • message • message • message • payload name type seqid

[1,"get_all_databases",1,1,{}] Response Body Example

Position 1 Position 2 Position 3: Position 4 Position 5

• version • message • message • message • payload name type seqid

[1,"get_all_databases",2,1,{"0":{"lst":["str",2,"default","h mshttptestdatabase"]}}] Hive Metastore Thrift API signature of get_all_databases()

list get_all_databases() throws(1:MetaException o1) Thrift in Apache Hive Architecture Protocol Client and Payload Thrift Generation

A Hive Metastore MS-HMSHTTP ApacheThrift Tool Thrift HTTP Client Thrift Appendix Chain • JavaScript API , PHP API, # API, … Thrift Schema in Protocol Appendix Demo

Protocol client can be created by using the Apache Thrift [ApacheThrift] toolchain to query the Hive Metastore Thrift API. 演示 Demo C#/JS HMSHTTP Thrift Client ▅ 1) Download Thrift EXE Generation Steps ▅ 2) Prepare On Windows 10 ThriftHiveMetastore.thrift and with VS2019 fb303.thrift files

According to ▅ 3) Use VS Developer CMD: https://thrift.apache.org/ ▅ thrift --gen

▅ thrift-0.13.0.exe --gen netstd/JS hive_metadata.thrift Resource

[MS-HMSHTTP]: Hive Metastore HTTP Protocol https://docs.microsoft.com/en-us/openspecs/sql_server_protocols/ms-hmshttp/1f53d4a7-19be- 41dc-aecc-f762096f2e92

Apache Hive IMetaStoreClient https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/metastore/IMetaStoreClient .html

[ApacheThrift] https://thrift.apache.org/

What are SQL Server Big Data Clusters? https://docs.microsoft.com/en-us/sql/big-data- cluster/big-data-cluster-overview?view=sqlallproducts-allversions

SQL-On-Hadoop: Hive - Part I https://www.sqlservercentral.com/articles/sql-on-hadoop-hive-part-i Summery • Protocol Background & Overview • Transport Direction & Stack • Message Structure • Message Example • Apache Thrift Generation Demo This is a Sample QR Code, We will replace it when the new QR code created.