Oracle® NoSQL Database Administrator's Guide Release 20.2 E85373-18 October 2020 Oracle NoSQL Database Administrator's Guide, Release 20.2 E85373-18 Copyright © 2011, 2020, Oracle and/or its affiliates. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government’s use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents Preface Conventions Used in This Book xv 1 Installing Oracle NoSQL Database Installation Prerequisites 1-1 Installation 1-2 Installation Configuration Parameters 1-3 Configuring Your KVStore Installation 1-6 Configuring the Firewall 1-10 2 Upgrading an Existing Oracle NoSQL Database Deployment Preparing to Upgrade 2-1 General Upgrade Notes 2-2 Upgrade to a New Release from an Existing Release 2-4 Using a Script to Upgrade to a New Release 2-10 3 Plans Using Plans 3-1 Tracking Plan Progress 3-2 Plan States 3-2 Reviewing Plans 3-3 Plan Ownership 3-3 Pruning Plans 3-3 4 Configuring the KVStore Configuration Overview 4-1 Start the Administration CLI 4-1 The plan Commands 4-3 Configure and Start a Set of Storage Nodes 4-3 iii Name your KVStore 4-3 Create a Zone 4-4 Create an Administration Process on a Specific Host 4-6 Create a Storage Node Pool 4-7 Create the Remainder of your Storage Nodes 4-7 Create Additional Admin Processes 4-8 Create and Deploy Replication Nodes 4-10 Configuring Security with Remote Access 4-11 Configuring with Multiple Zones 4-12 Adding Secondary Zone to the Existing Topology 4-20 Using Master Affinity Zones 4-24 Benefits of Master Affinity Zones 4-25 Adding a Master Affinity Zone 4-25 Losing a Master Affinity Zone Node 4-27 Using a Script to Configure the Store 4-27 Smoke Testing the System 4-28 Troubleshooting 4-30 Where to Find Error Information 4-31 Service States 4-31 Useful Commands 4-32 5 Configuring Multi-Region KVStores Use Case 1: Set up Multi-Region Environment 5-1 Deploy KVStore 5-2 Set Local Region Name 5-3 Configure XRegion Service 5-4 Start XRegion Service 5-8 Create Remote Regions 5-10 Create Multi-Region Tables 5-11 Access and Manipulate Multi-Region Tables 5-14 Stop XRegion Service 5-15 Use Case 2: Expand a Multi-Region Table 5-15 Prerequisites 5-15 Create MR Table in New Region 5-19 Add New Region to Existing Regions 5-20 Access MR Table in New and Existing Regions 5-24 Use Case 3: Contract a Multi-Region Table 5-24 Alter MR Table to Drop Regions 5-24 Use Case 4: Drop a Region 5-25 Prerequisites 5-26 iv Isolate the Region 5-26 Drop MR Tables in the Isolated Region 5-27 Drop the Isolated Region 5-28 6 Determining Your Store's Configuration Steps for Changing the Store's Topology 6-2 Make the Topology Candidate 6-2 Transforming the Topology Candidate 6-3 Increase Data Distribution 6-4 Increase Replication Factor 6-5 Balance a Non-Compliant Topology 6-6 Contracting a Topology 6-7 View the Topology Candidate 6-8 Validate the Topology Candidate 6-8 Preview the Topology Candidate 6-9 Deploy the Topology Candidate 6-9 Verify the Store's Current Topology 6-11 Deploying an Arbiter Node Enabled Topology 6-13 7 Administrative Procedures Backing Up the Store 7-1 Taking a Snapshot 7-1 Snapshot Activities 7-2 Managing Snapshots 7-4 Recovering the Store 7-6 Using the Load Program 7-6 Load Program and Metadata 7-8 Restoring Directly from a Snapshot 7-10 Recovering from Data Corruption 7-11 Detecting Data Corruption 7-11 Data Corruption Recovery Procedure 7-12 Replacing a Failed Disk 7-13 Replacing a Failed Storage Node 7-15 Using a New Storage Node 7-16 Task for an Identical Node 7-18 Repairing a Failed Zone by Replacing Hardware 7-20 Using Oracle NoSQL Migrator 7-20 Overview 7-21 Terminology used with NoSQL Data Migrator 7-22 v Using Oracle NoSQL Data Migrator 7-24 Sources and Sinks 7-26 Supported Sources and Sinks 7-27 Source Configuration Templates 7-27 Sink Configuration Templates 7-36 Transformation Configuration Templates 7-56 Use Case Demonstrations 7-65 Migrate from Oracle NoSQL Database Cloud Service to a JSON file 7-65 Migrate from Oracle NoSQL Database On-Premise to Oracle NoSQL Database Cloud Service 7-70 Migrate from MongoDB-Formatted JSON file to an Oracle NoSQL Database Cloud Service 7-72 Troubleshooting the NoSQL Data Migrator 7-74 Oracle NoSQL Data Migrator Vs. Import/Export Utility 7-77 Transitioning from Import/Export to NoSQL Data Migrator 7-78 Using the Import and Export Utilities 7-79 Import and Export Functionality 7-79 Understanding Data Sources and Data Targets (Sinks) 7-80 Importing Data 7-80 Exporting Data 7-81 Examples 7-81 Increasing Storage Node Capacity 7-83 Managing Storage Directory Sizes 7-87 Managing Disk Thresholds 7-87 Specifying Storage Directory Sizes 7-88 Specifying Differing Disk Capacities 7-89 Monitoring Disk Usage 7-90 Handling Disk Limit Exception 7-92 Increasing Storage Directory Size 7-93 Adding a New Shard 7-98 Managing Admin Directory Size 7-100 Admin is Working 7-101 Admin is not Working 7-101 Disabling Storage Node Agent Hosted Services 7-102 Verifying the Store 7-103 Monitoring the Store 7-107 Events 7-108 Setting Store Parameters 7-109 Changing Parameters 7-110 Setting Store Wide Policy Parameters 7-111 Admin Parameters 7-111 Changing Admin JVM Memory Parameters 7-112 vi Storage Node Parameters 7-114 Replication Node Parameters 7-117 Global Parameters 7-118 Security Parameters 7-118 Admin Restart 7-120 Replication Node Restart 7-121 Removing an Oracle NoSQL Database Deployment 7-121 Modifying Storage Node HA Port Ranges 7-122 Modifying Storage Node Service Port Ranges 7-123 Storage Node Not Deployed 7-123 Storage Node Deployed 7-124 8 Availablity, Failover and Switchover Operations Availability and Failover 8-1 Replication Overview 8-1 Loss of a Read-Only Replica Node 8-2 Loss of a Read/Write Master 8-3 Unplanned Network Partitions 8-3 Master is in the Majority Node Partition 8-4 Master is in the Minority Node Partition 8-5 No Majority Node Partition 8-5 Failover and Switchover Operations 8-6 Repairing a Failed Zone 8-7 Performing