Release Notes 3 HDP 3.1.4 Release Notes Date of Publish: 2019-08-26 https://docs.hortonworks.com Release Notes | Contents | ii Contents HDP 3.1.4 Release Notes..........................................................................................4 Component Versions.................................................................................................4 Descriptions of New Features..................................................................................5 Deprecation Notices.................................................................................................. 6 Terminology.......................................................................................................................................................... 6 Removed Components and Product Capabilities.................................................................................................6 Testing Unsupported Features................................................................................ 6 Descriptions of the Latest Technical Preview Features.......................................................................................7 Upgrading to HDP 3.1.4...........................................................................................7 Behavioral Changes.................................................................................................. 7 Apache Patch Information.....................................................................................11 Accumulo............................................................................................................................................................ 11 Atlas.....................................................................................................................................................................12 Calcite..................................................................................................................................................................16 DataFu................................................................................................................................................................. 16 Hadoop................................................................................................................................................................ 16 HBase.................................................................................................................................................................. 22 Hive..................................................................................................................................................................... 26 Kafka................................................................................................................................................................... 33 Knox.................................................................................................................................................................... 34 Livy..................................................................................................................................................................... 34 Oozie................................................................................................................................................................... 34 Phoenix................................................................................................................................................................38 Pig........................................................................................................................................................................38 Ranger................................................................................................................................................................. 39 Spark....................................................................................................................................................................41 Sqoop...................................................................................................................................................................42 Storm................................................................................................................................................................... 42 Tez.......................................................................................................................................................................44 Zeppelin...............................................................................................................................................................45 ZooKeeper...........................................................................................................................................................45 Fixed Common Vulnerabilities and Exposures................................................... 46 Release Notes | Contents | iii Fixed Issues..............................................................................................................48 Known Issues...........................................................................................................64 Legal Information...................................................................................................73 Release Notes HDP 3.1.4 Release Notes HDP 3.1.4 Release Notes This document provides you with the latest information about the Hortonworks Data Platform (HDP) 3.1.4 release. Component Versions List of the official Apache component versions for Hortonworks Data Platform (HDP) 3.1.4. To ensure that you are working with the most recent stable software available, you must be familiar with the latest Apache component versions in HDP 3.1.4. You should also be aware of the available Technical Preview components and use them only in a testing environment. The Hortonworks approach is to provide patches only when necessary, to ensure the interoperability of components. Unless you are explicitly directed by Hortonworks Support to install a patch, each of the HDP components should remain at the following package version levels to ensure a certified and supported copy of HDP 3.1.4. Official Apache component versions for HDP 3.1.4: • Apache Accumulo 1.7.0 • Apache Atlas 1.1.0 • Apache Calcite 1.16.0 • Apache DataFu 1.3.0 • Apache Druid 0.12.1 (incubating) • Apache Hadoop 3.1.1 • Apache HBase 2.0.2 • Apache Hive 3.1.0 • Apache Kafka 2.0.0 • Apache Knox 1.0.0 • Apache Livy 0.5.0 • Apache Oozie 4.3.1 • Apache Phoenix 5.0.0 • Apache Pig 0.16.0 • Apache Ranger 1.2.0 • Apache Spark 2.3.2 • Apache Sqoop 1.4.7 • Apache Storm 1.2.1 • Apache TEZ 0.9.1 • Apache Zeppelin 0.8.0 • Apache ZooKeeper 3.4.6 You can use the following Technical Previews components, but only in a testing environment: • Apache Superset 0.23.3 Note: For information about open source software licensing and notices, refer to the Licenses and Notices files included with the software installation package. Related reference Fixed Common Vulnerabilities and Exposures Apache Patch Information Fixed Issues 4 Release Notes Descriptions of New Features Descriptions of New Features Hortonworks Data Platform (HDP) 3.1.4 provides you with several new capabilities. You must learn about the new features and their benefits to use the new features in HDP 3.1.4. Apache component Feature areas, benefits, and links to further information Atlas Many fixes on top of Apache Atlas 1.1.0, including these new features: • Support for process execution entities that represent separate execution instances for a single process. • Hive Metastore hook for collecting metadata. • Bulk API for retrieving entities by unique attributes. Hive Core Capabilities • Automatic partition management synchronizes changes in the metadata and on the file system. • You can now configure how long you can retain partition data and metadata. • Hive Warehouse Connector now validates mapping of columns against those in Hive to alert the user to input errors. • Writing a DataFrame to Hive supports specifying partitions. • A new MergeBuilder interface for HiveWarehouseSession API operations supports merging tables. Upgrade • HiveStrictManagedMigration has two new options: Specify the number of threads for processing tables in parallel. Default: The number of CPU cores. Specify the the type of tables to process. For example, MANAGED_TABLE, EXTERNAL_TABLE. Default: All tables. Kakfa Core Capabilities • New Kafka Streams configuration parameter upgrade allows rolling bounce upgrade from an older version. • Updated ProcessorStateManager APIs in Kafka Streams for registering state stores to the processor topology. For more details please read the Streams Upgrade Guide. • Dynamic broker configuration options can be stored in ZooKeeper using kafka-configs.sh before brokers are started. This option can be used to avoid storing clear passwords in server.properties as all password configs may be stored encrypted in ZooKeeper. • ZooKeeper hosts are now re-resolved if the connection attempt fails. But if your ZooKeeper host names resolve to multiple addresses, and some of them are not reachable, you may need to increase the connection timeout zookeeper.connection.timeout.ms property. • Ability to define ACLs on prefixed resources, for example, any topic starting with 'foo'. Ranger Core Capabilities • Support for time-bound classification or business catalog mapping. Ranger policy engine recognizes the start and end