Migrating Oracle Big Data Cloud to Oracle Cloud Infrastructure
Total Page:16
File Type:pdf, Size:1020Kb
Oracle® Cloud Migrating Oracle Big Data Cloud to Oracle Cloud Infrastructure E68641-01 August 2019 Oracle Cloud Migrating Oracle Big Data Cloud to Oracle Cloud Infrastructure, E68641-01 Copyright © 2019, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency- specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents 1 Learn About Migrating to Oracle Cloud Infrastructure Why Migrate to Oracle Cloud Infrastructure 1-1 About the Migration Scope 1-1 About the Migration Task Flow 1-3 About Migration Tooling 1-3 About Oracle Cloud Infrastructure 1-4 2 Export Data and Metadata from Oracle Big Data Cloud Before You Export from Oracle Big Data Cloud 2-1 Export Data in HDFS 2-2 Export Data from the Oracle Cloud Infrastructure Classic Object Store 2-5 Install the rclone Utility 2-5 Configure the Object Store as Source for rclone 2-6 Configure the Object Store as Target for rclone 2-7 Verify the rclone Source and Target Configurations 2-7 Copy Data 2-7 Export Metadata 2-8 Stop the Hive, Zeppelin, and Spark Services 2-9 Prepare to Run the Export Script 2-9 Run the Export Script 2-9 Start the Hive, Zeppelin, and Spark Services 2-10 Location of Exported Content 2-11 3 Import Data and Metadata to Oracle Cloud Infrastructure Before You Import to Oracle Cloud Infrastructure 3-1 Import Data in HDFS 3-2 Import Data in the Oracle Cloud Infrastructure Classic Object Store 3-2 Import Metadata 3-2 Stop Hive, Zeppelin, and Spark from Ambari 3-2 Import Hive Metadata 3-2 Import Zeppelin Notebooks 3-3 iii Start Hive, Zeppelin, and Spark from Ambari 3-4 4 Complete the Post-Migration Tasks Validate the Migration 4-1 iv 1 Learn About Migrating to Oracle Cloud Infrastructure Learn about migrating your existing Oracle Big Data Cloud clusters to Oracle Cloud Infrastructure, and get an overview of the migration process and tools. Topics: • Why Migrate to Oracle Cloud Infrastructure • About the Migration Scope • About the Migration Task Flow • About Migration Tooling • About Oracle Cloud Infrastructure Why Migrate to Oracle Cloud Infrastructure Oracle encourages you to migrate your existing cloud resources to Oracle Cloud Infrastructure regions. You can gain several advantages by doing so. In Oracle Cloud, you provision resources in specific regions, which are localized to geographic locations. Certain regions support the Oracle Cloud Infrastructure platform. Oracle Cloud Infrastructure is Oracle's modern cloud platform that's based on the latest cloud technologies and standards. It provides more consistent performance and better features at lower costs. Oracle continues to invest in Oracle Cloud Infrastructure, including the addition of new regions, services, and features. See Data Regions for Platform and Infrastructure Services. You can benefit from these additional administrative features when you migrate your cloud resources to Oracle Cloud Infrastructure: • Organize cloud resources into a hierarchy of logical compartments. • Create fine-grained access policies for each compartment. To learn more, see Upgrade Your Classic Services to Oracle Cloud Infrastructure. About the Migration Scope An Oracle Big Data Cloud cluster can be migrated from Oracle Cloud Infrastructure Classic to Oracle Cloud Infrastructure. This document describes how to perform that migration. 1-1 Chapter 1 About the Migration Scope Note: You can also migrate Oracle Big Data Cloud clusters to Cloudera on Oracle Cloud Infrastructure IaaS. For instructions, see My Oracle Support Doc ID 2572925.1 at http://support.oracle.com. You're advised to retain your Oracle Big Data Cloud cluster(s) for a minimum of three months after setting up a functional alternative environment, at least in a stopped state, so if any data, metadata, or configuration is missing you can refer to the cluster to get the required information. When a cluster is in a stopped state you're not charged for compute resources, but are still charged for storage. See Stop, Start, and Restart a Cluster in Using Oracle Big Data Cloud. The following table lists the artifacts that can be migrated from Oracle Big Data Cloud. Artifact Approach More Information Data in HDFS Copy HDFS data to the Export Data in HDFS Oracle Cloud Infrastructure object store Data in the Oracle Copy data to the Oracle Topics in Export Data from the Oracle Cloud Infrastructure Cloud Infrastructure object Cloud Infrastructure Classic Object Classic object store store Store (for clusters based on Oracle Cloud Infrastructure Classic) Hive metadata Dump the Hive database in Topics in Export Metadata the MySQL database in the source Zeppelin notebooks Export notebook definitions Topics in Export Metadata from /user/zeppelin/ notebook in HDFS Configuration files Extract configuration data Topics in Export Metadata (HDFS, YARN, and using the Ambari REST API. Spark) You can refer to these files for classpath settings and performance parameters during migration. You'll need to analyze exported configurations and determine how to apply the configuration changes to the target environments. Versions of various Export service version details Topics in Export Metadata open source using the Ambari REST API. components You can also get version details from the Ambari console (Admin > Stack and Versions). 1-2 Chapter 1 About the Migration Task Flow Note: Oracle Identity Cloud Service (IDCS) integration is not supported for Oracle Big Data Cloud on Oracle Cloud Infrastructure. If you migrate an IDCS- enabled cluster from the Oracle Cloud Infrastructure Classic environment, you won't be able to use IDCS users with Oracle Big Data Cloud on Oracle Cloud Infrastructure. About the Migration Task Flow At a high level, the migration process includes the following tasks: 1. Prepare to export data and metadata from Oracle Big Data Cloud. See Before You Export from Oracle Big Data Cloud. 2. Export data in HDFS to the Oracle Cloud Infrastructure object store. See Export Data in HDFS. 3. Export data from the Oracle Cloud Infrastructure Classic object store to the Oracle Cloud Infrastructure object store using the rclone utility. See Export Data from the Oracle Cloud Infrastructure Classic Object Store. 4. Export metadata (Hive metadata, Zeppelin notebooks, and service configuration and version information) using the exportBDC.py utility script provided by Oracle. See Export Metadata. 5. Prepare to import data and metadata. See Before You Import to Oracle Cloud Infrastructure. 6. Import data in HDFS to the target Oracle Big Data Cloud cluster on Oracle Cloud Infrastructure. See Import Data in HDFS. 7. Import data from the Oracle Cloud Infrastructure Classic object store to the Oracle Cloud Infrastructure object store. See Import Data in the Oracle Cloud Infrastructure Classic Object Store. 8. Import metadata (Hive metadata, Zeppelin notebooks, and service configuration). See Import Metadata. 9. Validate the migration. See Validate the Migration. About Migration Tooling Learn about the tools used to migrate data and metadata from Oracle Big Data Cloud.