Using Oracle Nosql Database Cloud Service

Oracle® Cloud Using Oracle NoSQL Database Cloud Service

Latest Cloud Release E90090-24 August 2021 Oracle Cloud Using Oracle NoSQL Database Cloud Service, Latest Cloud Release

E90090-24

Primary Author: Vandanadevi Rajamani

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government’s use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle, Java, and MySQL are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents

Preface Audience viii Documentation Accessibility viii Related Resources viii Conventions viii Diversity and Inclusion ix

1 Getting Started About the Service 1-1 Region Availability 1-2 Service Limits 1-2 Service Quotas 1-2 Service Events 1-3 Service Metrics 1-5 Before You Begin 1-6 Typical Workflow 1-7 Key Features 1-7 Always Free Service 1-8 Developer Overview 1-9 Cloud Concepts 1-10 Functional difference between the NoSQL Cloud Service and On-premise database 1-12 Quick Start Tutorials 1-13

2 Connecting your Application Acquiring Credentials 2-1 Connecting your Application using Java 2-3 Connecting your Application using Python 2-4 Connecting your Application using Node.js 2-4 Connecting your Application using Go 2-4 Data Regions and Associated Service Endpoints 2-4

iii 3 Developing in Oracle Cloud Table Design 3-1 Supported Data Types 3-2 Table Fields 3-3 Primary Keys and Shard Keys 3-5 Time to Live 3-6 Table States and Life Cycles 3-7 Table Management 3-8 Data Definition Language Reference 3-8 Using Tables in Java 3-10 About the Oracle NoSQL Database Java SDK 3-10 About Compartments 3-10 Obtaining a NoSQL Handle 3-11 Creating Tables and Indexes 3-12 Adding Data 3-14 Adding JSON Data 3-15 Reading Data 3-16 Using Queries 3-17 Deleting Data 3-18 Modifying Tables 3-19 Dropping Tables and Indexes 3-21 Handling Errors 3-21 Using Tables in Python 3-22 Using Tables in Node.js 3-22 Using Tables in Go 3-22 Query Language Reference 3-22 Oracle NoSQL Database Cloud Service Limits 3-23 Estimating Capacity 3-25 Handling Capacity 3-29

4 Developing in Oracle NoSQL Database Cloud Simulator Downloading the Oracle NoSQL Database Cloud Simulator 4-1 Oracle NoSQL Database Cloud Simulator Compared With Oracle NoSQL Database Cloud Service 4-2

5 Using Plugins for Development About IntelliJ Plugin 5-1 Setting Up IntelliJ Plug-in 5-1 Creating a NoSQL Project in IntelliJ 5-2

iv Connecting to Oracle NoSQL Database Cloud Simulator from IntelliJ 5-2 Connecting to Oracle NoSQL Database Cloud Service from IntelliJ 5-3 Managing Tables Using the IntelliJ Plugin 5-5 About Eclipse Plugin 5-6

6 Using the Console to Manage Tables Accessing the Service from the Infrastructure Console 6-1 Managing Table Data 6-1 Inserting Data Into Tables 6-1 Inserting Data Into Tables: Simple Input Mode 6-2 Inserting Data Into Tables: Advanced JSON Input Mode 6-2 Viewing Table Data 6-3 Updating Table Data 6-3 Deleting Table Data 6-4 Downloading Table Data 6-4 Managing Tables and Indexes 6-5 Creating Tables 6-5 Creating Table: Simple Input Mode 6-5 Creating Table: Advanced DDL Input Mode 6-9 Creating Indexes 6-11 Editing Tables 6-12 Altering Tables 6-14 Adding Table Columns: Simple Input Mode 6-15 Adding Table Columns: Advanced DDL Input Mode 6-16 Deleting Table Columns 6-16 Moving Tables 6-17 Deleting Tables 6-17 Deleting Indexes 6-18 Monitoring Tables and Indexes 6-18 Viewing Tables 6-19 Viewing Indexes 6-19 Viewing Table Details 6-19 Viewing Table DDL 6-19 Viewing Table Metrics 6-20

7 Managing Subscriptions Setting up Your Service 7-1 Estimating Your Monthly Cost 7-2

v Creating a Compartment 7-2

8 Managing Table Access and Security About Oracle NoSQL Database Cloud Service Security Model 8-1 Typical Process to Manage Security for Oracle NoSQL Database Cloud Service 8-3 Setting Up Users, Groups, and Policies 8-3 Policy Reference 8-4 Resource-Types 8-4 Supported Variables 8-5 Details for Verb + Resource-Type Combinations 8-5 Permission Required for Each NoSQL Cloud Driver Request 8-7 Permission Required for Each REST API Operation 8-8 Typical Policy Statements to Manage Tables 8-9 Accessing NoSQL Tables Across Tenancies 8-10 Giving Another User Permission to Manage NoSQL Tables 8-12

9 Using Oracle NoSQL Database Migrator Overview 9-1 Terminology used with Oracle NoSQL Database Migrator 9-2 Using Oracle NoSQL Database Migrator 9-4 Sources and Sinks 9-9 Supported Sources and Sinks 9-9 Source and Sink Security 9-10 Source Configuration Templates 9-11 JSON File 9-11 JSON File in OCI Object Storage Bucket 9-13 MongoDB-Formatted JSON File 9-17 MongoDB-Formatted JSON File in OCI Object Storage bucket 9-18 Oracle NoSQL Database 9-22 Oracle NoSQL Database Cloud Service 9-24 Sink Configuration Templates 9-28 JSON File 9-28 JSON File in OCI Object Storage Bucket 9-31 Oracle NoSQL Database 9-35 Oracle NoSQL Database Cloud Service 9-39 Transformation Configuration Templates 9-46 ignoreFields 9-46 renameFields 9-47 aggregateFields 9-47

vi Use Case Demonstrations 9-48 Migrate from Oracle NoSQL Database Cloud Service to a JSON file 9-49 Migrate from Oracle NoSQL Database On-Premise to Oracle NoSQL Database Cloud Service 9-54 Migrate from MongoDB-Formatted JSON file to an Oracle NoSQL Database Cloud Service 9-56 Troubleshooting the Oracle NoSQL Database Migrator 9-58 Oracle NoSQL Database Migrator Vs. Import/Export Utility 9-61 Transitioning from Import/Export to NoSQL Database Migrator 9-68

vii Preface

Preface

This document describes how to use Oracle NoSQL Database Cloud Service and provides references to related documentation. Audience

This document is intended for developers who want to create tables and query data in Oracle NoSQL Database Cloud Service. This document is also intended for Oracle Cloud administrators who want to manage users and their privileges in Oracle NoSQL Database Cloud Service. Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup? ctx=acc&id=docacc.

Access to Oracle Support Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/ lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired. Related Resources

For more information, see About Oracle Cloud. Conventions

The following text conventions are used in this document:

Convention Meaning boldface Boldface type indicates graphical user interface elements associated with an action, or terms defined in text or the glossary. italic Italic type indicates book titles, emphasis, or placeholder variables for which you supply particular values. monospace Monospace type indicates commands within a paragraph, URLs, code in examples, text that appears on the screen, or text that you enter.

viii Preface

Diversity and Inclusion

Oracle is fully committed to diversity and inclusion. Oracle respects and values having a diverse workforce that increases thought leadership and innovation. As part of our initiative to build a more inclusive culture that positively impacts our employees, customers, and partners, we are working to remove insensitive terms from our products and documentation. We are also mindful of the necessity to maintain compatibility with our customers' existing technologies and the need to ensure continuity of service as Oracle's offerings and industry standards evolve. Because of these technical constraints, our effort to remove insensitive terms is ongoing and will take time and external cooperation.

ix 1 Getting Started

This chapter gives an overview of Oracle NoSQL Database Cloud Service and provides hands-on tutorials to get started with the service.

Topics • About the Service • Key Features • Always Free Service • Cloud Concepts • Quick Start Tutorials About the Service

This section provides an overview of Oracle NoSQL Database Cloud Service. Oracle NoSQL Database Cloud Service is a fully managed database cloud service that is designed for database operations that require predictable, single digit millisecond latency responses to simple queries. NoSQL Database Cloud Service allows developers to focus on application development rather than setting up cluster servers, or performing system monitoring, tuning, diagnosing, and scaling. NoSQL Database Cloud Service is suitable for applications such as Internet of Things, user experience personalization, instant fraud detection, and online display advertising. Once you are authenticated against your Oracle Cloud account, you can create a NoSQL table, and specify throughput and storage requirements for the table. Oracle reserves and manages the resources to meet your requirements, and provisions capacity for you. Capacity is specified using read and write units for throughput and GB for storage units. See Estimating Capacity.

Service Access Options As a developer, you can connect to the Oracle NoSQL Database Cloud Service and work with NoSQL tables using the NoSQL SDKs available in multiple languages. To learn how to download and use the NoSQL SDKs, see Developing in Oracle Cloud. In addition to the NoSQL SDKs, Oracle NoSQL Database Cloud Service also provides Oracle Cloud Infrastructure Console for you to access, manage, and use the NoSQL Database Cloud Service. The Oracle Cloud Infrastructure Console lets you create and manage NoSQL tables and indexes declaratively. You can also manipulate the data in your NoSQL tables and monitor your Oracle NoSQL Database Cloud Service from the console. See Using the Console to Manage Tables.

1-1 Chapter 1 About the Service

Region Availability

For the latest information on the regions where Oracle NoSQL Database Cloud Service is available, see Data Regions for Platform and Infrastructure Services in Oracle Cloud Infrastructure Documentation.

Tip:

To find the service endpoint for a specific data region, see Data Regions and Associated Service Endpoints.

Service Limits

Oracle NoSQL Database Cloud Service has various default limits. Whenever you create an Oracle NoSQL Database Cloud Service table, the system ensures that your requests are within the bounds of the specified limit. This table lists the Oracle NoSQL Database Cloud Service limits that you can reference. For detailed list of service limits, see Oracle NoSQL Database Cloud Service Limits.

Resource Monthly Universal Credits Pay-as-You-Go or Promo Read Units 100,000 units 100,000 units Write Units 40,000 units 40,000 units Table Size 5,000 GB 5,000 GB

You can increase your service limits by submitting a request either from Limits, Quotas, and Usage page in Oracle Cloud Infrastructure Console or using the TableRequest API.

See About Service Limits and Usage in Oracle Cloud Infrastructure Documentation. Service Quotas

You can use quotas to determine how other users allocate Oracle NoSQL Database Cloud Service resources across compartments in Oracle Cloud Infrastructure. A compartment is a collection of related resources (such as instances, virtual cloud networks, block volumes) that can be accessed only by certain groups that have been given permission by an administrator. Whenever you create an Oracle NoSQL Database Cloud Service table or scale up the provisioned throughput or storage, the system ensures that your requests are within the bounds of the quota for that compartment. This table lists the Oracle NoSQL Database Cloud Service quotas that you can reference.

Name Scope Description read-unit-count Regional Read Unit Count write-unit-count Regional Write Unit Count

1-2 Chapter 1 About the Service

Name Scope Description table-size-gb Regional Table Size (GB)

You can set quotas using the Console or API. You can execute quota statements from the Quota Policies page under the Governance option in Oracle Cloud Infrastructure Console.

Example Quota Statements for Oracle NoSQL Database Cloud Service • Limit the number of Oracle NoSQL Database Cloud Service read units that users can allocate to tables they create in my_compartment to 20,000.

set nosql quota read-unit-count to 20000 in compartment my_compartment

• Limit the number of Oracle NoSQL Database Cloud Service write units that users can allocate to tables they create in my_compartment to 5,000.

set nosql quota write-unit-count to 5000 in compartment my_compartment

• Limit the maximum storage space of Oracle NoSQL Database Cloud Service that users can allocate to tables they create in my_compartment to 1,000 GB.

set nosql quota table-size-gb to 1000 in compartment my_compartment

See About Compartment Quotas in Oracle Cloud Infrastructure Documentation. Service Events

Actions that you perform on Oracle NoSQL Database Cloud Service tables emit events. You can define rules that trigger a specific action when an event occurs. For example, you might define a rule that sends a notification to administrators when someone drops a table. See Overview of Events and Get Started with Events in Oracle Cloud Infrastructure Documentation. This table lists the Oracle NoSQL Database Cloud Service events that you can reference.

Friendly Name Event Type Alter Table Begin com.oraclecloud.nosql.altertable.begin

Alter Table End com.oraclecloud.nosql.altertable.end

Change Table Compartment Begin com.oraclecloud.nosql.changecompartment.b egin

1-3 Chapter 1 About the Service

Friendly Name Event Type Change Table Compartment End com.oraclecloud.nosql.changecompartment.e nd

Create Index Begin com.oraclecloud.nosql.createindex.begin

Create Index End com.oraclecloud.nosql.createindex.end

Create Table Begin com.oraclecloud.nosql.createtable.begin

Create Table End com.oraclecloud.nosql.createtable.end

Drop Index Begin com.oraclecloud.nosql.dropindex.begin

Drop Index End com.oraclecloud.nosql.dropindex.end

Drop Table Begin com.oraclecloud.nosql.droptable.begin

Drop Table End com.oraclecloud.nosql.droptable.end

Example This example shows information associated with the event Create Table Begin:

{ "cloudEventsVersion": "0.1", "contentType": "application/json", "source": "nosql", "eventID": "", "eventType": "com.oraclecloud.nosql.createtable.begin", "eventTypeVersion": "", "eventTime": "2019-12-30T00:52:01.343Z", "data": { "additionalDetails": {}, "availabilityDomain": "", "compartmentId": "ocid1.compartment.oc1..", "compartmentName": "my_compartment",

1-4 Chapter 1 About the Service

"freeformTags": { "key":"value" }, "resourceId": "ocid1.nosqltable.oc1..", "resourceName": "my_nosql_table" }, "extensions": { "compartmentId": "ocid1.compartment.oc1.." } }

Service Metrics

Learn about the metrics emitted by the metric namespace oci_nosql (Oracle NoSQL Database Cloud Service). Metrics for Oracle NoSQL Database Cloud Service include the following dimensions: • RESOURCEID The OCID of the NoSQL Table in the Oracle NoSQL Database Cloud Service.

Note:

OCID is an Oracle-assigned unique ID that is included as part of the resource's information in both the console and API.

• TABLENAME The name of the NoSQL table in the Oracle NoSQL Database Cloud Service. Oracle NoSQL Database Cloud Service sends metrics to the Oracle Cloud Infrastructure Monitoring Service. You can view or create alarms on these metrics using the Oracle Cloud Infrastructure Console SDKs or CLI. See OCI SDKs and CLI in Oracle Cloud Infrastructure Documentation. Available Metrics

Metric Metric Display Unit Description Dimensions Name ReadUnits Read Units Units The number of resourceIdtable read units Name consumed during this period. WriteUnits Write Units Units The number of resourceIdtable write units Name consumed during this period.

1-5 Chapter 1 About the Service

Metric Metric Display Unit Description Dimensions Name StorageGB Storage Size GB The maximum resourceIdtable amount of storage Name consumed by the table. As this information is generated hourly, you may see values that are out of date in between the refresh points. ReadThrottleCou Read Throttle Count The number of resourceIdtable nt read throttling Name exceptions on this table in the time period. WriteThrottleCo Write Throttle Count The number of resourceIdtable unt write throttling Name exceptions on this table in the time period. StorageThrottle Storage Throttle Count The number of resourceIdtable Count storage throttling Name exceptions on this table in the time period. Before You Begin

When you order Oracle NoSQL Database Cloud Service through Universal Credits, you automatically get access to other required services, including Oracle Cloud Infrastructure. Here's some information about how Oracle NoSQL Database Cloud Service uses other services and what you need to do if you're setting up Oracle NoSQL Database Cloud Service for the first time.

Service What is it for? Do I need to do anything? Oracle Cloud Infrastructure Identity · You use Oracle Cloud Yes. and Access Management Infrastructure Identity and Before you create your first Oracle Access Management to create NoSQL Database Cloud Service user accounts for the people in table, Oracle recommends that you your organization who will use set up one or more compartments in Oracle NoSQL Database Cloud which you deploy and secure your Service. table. · You can give users permission To know more, see the following to inspect, read, use, or manage resources in Oracle Cloud Oracle NoSQL Database Cloud Infrastructure Documentation: Service tables. See Setting Up · Setting Up Your Tenancy Users, Groups, and Policies · Managing Compartments · You use compartments within your tenancy to organize tables on Oracle Cloud Infrastructure.

1-6 Chapter 1 Key Features

Typical Workflow

Typical sequence of tasks to work with Oracle NoSQL Database Cloud Service. If you're developing applications using Oracle NoSQL Database Cloud Service for the first time, follow these tasks as a guide.

Task Description More Information Connect your application Connect your application to use Oracle NoSQL Connecting your Database Cloud Service tables. Application Develop your application Develop your application after connecting to Oracle Developing in Oracle Cloud NoSQL Database Cloud Service tables. Developing in Oracle NoSQL Database Cloud Simulator

If you're setting up Oracle NoSQL Database Cloud Service for the first time, see Setting up Your Service. Key Features

Learn the key features of Oracle NoSQL Database Cloud Service. • Fully Managed with Zero Administration: Developers do not need to administer data servers or the underlying infrastructure and security. Oracle maintains the hardware and software which allows developers to focus on building applications. • Faster Development Life Cycle: After purchasing access to the service, developers write their applications, and then connect to the service using their credentials. Reading and writing data can begin immediately. Oracle performs Database Management, Storage Management, High Availability, and Scalability which helps developers concentrate on delivering high-performance applications. • High Performance and Predictability: Oracle NoSQL Database Cloud Service takes advantage of the latest component technologies in the Oracle Cloud Infrastructure by providing high performance at scale. Developers know that their applications return data with predictable latencies, even as their throughput and storage requirements increase. • On-Demand Throughput and Storage Provisioning: Oracle NoSQL Database Cloud Service scales to meet application throughput performance requirements with low and predictable latency. As workloads increase with periodic business fluctuations, applications can increase their provisioned throughput to maintain a consistent user experience. As workloads decrease, the same applications can reduce their provisioned throughput, resulting in lower operating expenses. The same holds true for storage requirements. Those can be adjusted based on business fluctuations. You can increase or decrease the storage using the Oracle Cloud Infrastructure Console or the TableRequest API. • Simple APIs: Oracle NoSQL Database Cloud Service provides easy-to-use CRUD (Create Read Update Delete) APIs that allow developers to easily create tables and maintain data in them. • Data Modeling: Oracle NoSQL Database Cloud Service supports both schema-based and schema-less (JSON) modeling.

1-7 Chapter 1 Always Free Service

• Data Safety in Redundancy: The Oracle NoSQL Database Cloud Service stores data across multiple Availability Domains (ADs) or Fault Domains (FDs) in single AD regions. If an AD or FD becomes unavailable, user data is still accessible from another AD or FD. • Data Security: Data is encrypted at rest (on disk) with Advanced Encryption Standard (AES 256). Data is encrypted in motion (transferring data between the application and Oracle NoSQL Database Cloud Service) with HTTPS. • ACID-Compliant Transactions: ACID (Atomicity, Consistency, Isolation, Durability) transactions are fully supported for the data you store in Oracle NoSQL Database Cloud Service. If required, consistency can be relaxed in favor of lower latency. • JSON Data Support: Oracle NoSQL Database Cloud Service allows developers to query schema-less JSON data by using the familiar SQL syntax. • Partial JSON Updates: Oracle NoSQL Database Cloud Service allows developers to update (change, add, and remove) parts of a JSON document. Because these updates occur on the server, the need for a read-modify-write cycle is eliminated, which would consume throughput capacity. • Time-To-Live: Oracle NoSQL Database Cloud Service lets developers set a time frame on table rows, after which the rows expire automatically, and are no longer available. This feature is a critical requirement when capturing sensor data for Internet Of Things (IoT) services. • SQL Queries: Oracle NoSQL Database Cloud Service lets developers access data with SQL queries. • Secondary Indexes: Secondary indexes allow a developer to create an index on any field of a supported data type, thus improving performance over multiple paths for queries using the index. Always Free Service

Always Free NoSQL Database Service As part of the Oracle Cloud Free Tier, the Oracle NoSQL Database Cloud Service participates as an Always Free service. This section describes the restrictions, and details of that offering. Features of Always Free NoSQL Database Service • You may have up to three Always Free NoSQL tables in your tenancy. • You can have both Always Free and regular tables in the same tenancy. • The Always Free NoSQL tables are displayed in the console with an “Always Free” label next to the table name. • An Always Free NoSQL table cannot be changed to a regular table or vice versa. Resource Restrictions for Always Free NoSQL tables • You may have a maximum of three Always Free NoSQL tables at any time. If you have three Always Free NoSQL tables , the toggle button to create an Always Free NoSQL table is disabled. If you delete one or more of those tables, then the toggle button will be re-enabled. • Read Capacity (Read Units) is 50 and cannot be changed.

1-8 Chapter 1 Developer Overview

• Write Capacity (Write Units) is 50 and cannot be changed. • Disk Storage is 25GB and cannot be changed. Regional Availability Always Free NoSQL tables are available in a subset of Oracle Cloud Infrastructure data regions. See Data Regions for more details on where Always Free NoSQL tables are supported. Always Free NoSQL tables - Inactivity and Deletion If an Always Free NoSQL table has not been used or accessed for 30 days, it moves to an ‘inactive’ state. Always Free NoSQL tables that remain inactive for 90 days are deleted. The inactive state is shown in the console next to the table name. A customer notification is sent to the tenancy administrator when the table initially becomes inactive (after 30 days of inactivity). A reminder is sent again at 75 days of inactivity. You may make an Always Free NoSQL table active again by performing any get/put/delete operation on any row(s) in the table. DDL operations do not make an inactive table active again. Developer Overview

Get a high-level overview of the service architecture and select an SDK/Driver that will meet your application development needs. NDCS Developer tasks Oracle NoSQL Database Cloud Service (NDCS) is a fully HA service. It is designed for highly demanding applications that require low latency response times, a flexible data model, and elastic scaling for dynamic workloads. As a fully managed service, Oracle handles all the administrative tasks such as software upgrades, security patches, hardware failures, and patching.

NoSQL cluster HTTPS running in any cloud vendor: Customer NoSQL AWS, Azure, etc NoSQL DB NoSQL DB NoSQL DB Application Instance Instance Instance

OCI Region

HTTPS Availability Domain 1 Availability Domain 2 Availability Domain 3 NoSQL SDK/ Driver NDCS NDCS NDCS NDCS NDCS NDCS Tenant Tenant Tenant Tenant Tenant Tenant

HTTPS

OCI Console NDCS NDCS NDCS NDCS NDCS NDCS Table Table Table Table Table Table

NDCS NDCS NDCS Tenant Tenant Tenant Customer NoSQL Application HTTPS

NDCS NDCS NDCS OCI SDK/ Table Table Table Driver

1-9 Chapter 1 Cloud Concepts

NoSQL Database SDKs/Drivers – These SDKs are licensed under UPL and can be used on either the NoSQL Cloud Service or the on-premise database. These are full featured SDKs and offer a rich set of functionality. These drivers can also be used in applications executing against Oracle NoSQL clusters running in other vendors cloud. 1. NoSQL SDK for Java 2. NoSQL JavaScript SDK 3. NoSQL Python SDK 4. NoSQL Go SDK 5. NoSQL SDK for Spring Data OCI Console – Offers ability to create tables quickly, modify tables, delete tables, load data, create indexes quickly, delete indexes, basic queries, alter table capacities and view metrics. Queries that include ‘Order by’, ‘Group by’, aggregation, ‘Limit’, ‘Offset’, ‘As’, and built-in functions are not supported. OCI SDKs/Drivers – Oracle Cloud Infrastructure provides a number of Software Development Kits (SDKs) to facilitate development of custom solutions. These are typically licensed under UPL. These offer similar functionality to the OCI console through a programmatic interface. 1. Rest API 2. SDK for Java 3. SDK for Python 4. SDK for Javascript 5. SDK for Go 6. SDK for Ruby References: • SQL for NoSQL documentation • Functional difference between the NoSQL Cloud Service and On-premise database Cloud Concepts

Learn the Oracle NoSQL Database Cloud Service concepts. • Table: A Table is a collection of rows where each row holds a data record from your application. Each table row consists of key and data fields which are defined when a table is created. In addition, a table has a specified storage, can support a defined maximum read and write throughput, and has a maximum size. The storage capacity is specified at table creation time and can be changed later. – High-Level Data Types: Oracle NoSQL Database Cloud Service supports all three types of Big Data. You can create NoSQL tables to store structured, unstructured, or semi-structured data. * Structured: This type of data can be organized and stored in tables with a predefined structure or schema. For example, the data stored in regular relational database tables come under this category. They adhere to a fixed schema and are simple to manage and analyze. Data generated

1-10 Chapter 1 Cloud Concepts

from credit card transactions and e-commerce transactions are a few examples of structured data. * Semi-Structured: The data that can not fit into a relational database but can be organized into rows and columns after a certain level of processing is called semi-structured data. Oracle NoSQL Database Cloud Service can store and process semi-structured data by storing key-value pairs in NoSQL tables. XML data is an example of semi-structured data. * Unstructured: The data that can not be organized or stored in tables with a fixed schema or structure are called Unstructured data. Videos, images, and media are a few examples of unstructured data. Oracle NoSQL Database Cloud Service lets you define tables with rows of JSON data type to store unstructured data. – Data Types: A table is created using DDL (Data Definition Language) which defines the data types and primary keys used for the table. Oracle NoSQL Database Cloud Service supports several data types, including several numeric types, string, binary, timestamp, maps, arrays, records, and a special JSON data type which can hold any valid JSON data. Applications can use unstructured tables where a row uses the JSON data type to store the data, or use structured tables where all row types are defined and enforced. See Supported Data Types to view the list of data types supported in Oracle NoSQL Database Cloud Service. Unstructured tables are flexible. But typed data is safer from an enforcement and storage efficiency point of view. Table schema can be modified , but the table structure is less flexible to change. – Indexes: Applications can create an index on any data field which has a data type that permits indexing, including JSON data fields. JSON indexes are created using a path expression into the JSON data. – Capacity: When you create a table, you also specify throughput and storage resources available for the table. Read and write operations to the table are limited by the read and write throughput capacity that you define. The amount of space that the table can use is limited by the storage capacity. See Estimating Capacity to learn how to estimate capacity for your application workload. • Distribution and Sharding: Although not visible to the user, Oracle NoSQL Database Cloud Service tables are sharded and replicated for availability and performance. Therefore, you should consider this during schema design. – Primary and Shard keys: An important consideration for a table is the designation of the primary key, and the shard key. When you create a table in Oracle NoSQL Database Cloud Service, the data in the table is automatically sharded based on a portion of the table primary key, called the shard key. See Primary Keys and Shard Keys for considerations on how to designate the primary and shard keys. – Read Consistency: Read consistency specifies different levels of flexibility in terms of which copy of the data is used to fulfill a read operation. Oracle NoSQL Database Cloud Service provides two levels of consistency, EVENTUAL, and ABSOLUTE. Applications can specify ABSOLUTE consistency, which guarantees that all read operations return the most recently written value for a designated key. Or, applications capable of tolerating inconsistent data can specify EVENTUAL consistency, allowing the database to return a value more quickly even if it is not up-to-date. ABSOLUTE consistency results in a higher cost, consuming twice the number of read units for the same data relative to EVENTUAL consistency, and should only be used

1-11 Chapter 1 Functional difference between the NoSQL Cloud Service and On-premise database

when required. Consistency can be set for a NoSQL handle, or as an optional argument for all read operations. • Identity Access and Management: Oracle NoSQL Database Cloud Service uses the Oracle Cloud Infrastructure Identity and Access Management to provide secure access to Oracle Cloud. Oracle Cloud Infrastructure Identity and Access Management enables you to create user accounts and give users permission to inspect, read, use, or manage Oracle NoSQL Database Cloud Service tables. See Overview of Oracle Cloud Infrastructure Identity and Access Management in Oracle Cloud Infrastructure Documentation. Functional difference between the NoSQL Cloud Service and On-premise database

Table 1-1 High level feature comparison

- NoSQL Database Cloud NoSQL Database Enterprise Service Edition (EE) Infrastructure and software Managed by Oracle Managed by customer management/maintenance (servers, storage, networking, security, OS, and NoSQL software) Database deployment Oracle Cloud only Customer on-premises data centers or BYOL in Oracle Cloud or other cloud vendors. Licensing/Edition Paid subscription or always- Enterprise Edition (paid) or free service Community Edition (free open source) Throughput Throughput capacity is Throughput capacity is managed at each NoSQL managed at each NoSQL Table level through NoSQL cluster. The capacity depends APIs or Oracle Cloud on the size of the NoSQL Infrastructure (OCI) Console. cluster deployed. Larger The capacity is measured in cluster size provides more Write Units, Read Units. throughput capacity for user Throughput capacity per table tables. can be adjusted to meet the dynamic workloads. When the limits for a table is exceeded, users are notified. At the tenancy level, there are maximum service limits. To get more details, see Oracle NoSQL Database Cloud Service Limits.

1-12 Chapter 1 Quick Start Tutorials

Table 1-1 (Cont.) High level feature comparison

- NoSQL Database Cloud NoSQL Database Enterprise Service Edition (EE) Storage Storage capacity is managed Storage capacity is managed at each NoSQL Table level at each NoSQL cluster. The through NoSQL APIs or capacity depends on the Oracle Cloud Infrastructure number of disks and specific (OCI) Console. The capacity is configuration in each storage measured in gigabytes (GB). node deployed in the cluster. Storage capacity per table can Larger cluster size and disk be adjusted to meet the capacity provide more storage dynamic workloads. When the for user tables. limit for a table is exceeded, users are notified. At the tenancy level, there are maximum service limits. To get more details, see Oracle NoSQL Database Cloud Service Limits. Interoperability Interoperates with NoSQL Interoperates with NoSQL Database Enterprise Edition Database Cloud Service through a single programmatic through a single programmatic interface with no application interface with no application code modification. code modification. Installation No customer installs. Customers download and Customers start using the install the software to set up service right away by creating the NoSQL cluster in multiple NoSQL Tables. storage nodes. Quick Start Tutorials

Access the quick start tutorials and get started with the service.

1. Get access to the service. Sign up for a Cloud promotion or purchase an Oracle Cloud subscription. Activate your order, create users (optional). See Setting up Your Service. 2. Acquire credentials to connect your application. See Acquiring Credentials. 3. Create a table in Oracle NoSQL Database Cloud Service.

• To start in Java, see Quick Start: Create a Table in Oracle NoSQL Database Cloud Service.

1-13 2 Connecting your Application

Your application must acquire and provide credentials to establish a secure connection with the Oracle NoSQL Database Cloud Service.

Topics • To connect your application, see: – Acquiring Credentials – Connecting your Application using Java – Connecting your Application using Python – Connecting your Application using Node.js – Connecting your Application using Go • To learn about data regions supported for Oracle NoSQL Database Cloud Service, see Data Regions and Associated Service Endpoints. Acquiring Credentials

Learn how to acquire credentials to connect your application to Oracle NoSQL Database Cloud Service. You need an Oracle Cloud account to connect your application to the Oracle NoSQL Database Cloud Service. If you do not already have an Oracle Cloud account, you can start with Oracle Cloud. The credentials that are used for connecting your application are associated with a specific user. If you want to create a user for your application, see Setting Up Users, Groups, and Policies. Information Comprising the Credentials:

Table 2-1 Credentials

What is it? Where to find it? Tenancy ID, an OCID See Where to Get the Tenancy©s OCID and User©s User ID, an OCID OCID in Oracle Cloud Infrastructure Documentation. API Signing Key For the application user, an API signing key must be generated and uploaded. If this has already been done this step can be skipped. To know more, see the following resources in Oracle Cloud Infrastructure Documentation: · How to Generate an API Signing Key to generate the signing key with an optional pass phrase. · How to Upload the Public Key to upload the API signing key to the user©s account.

2-1 Chapter 2 Acquiring Credentials

Table 2-1 (Cont.) Credentials

What is it? Where to find it? Fingerprint for the Signing Key The fingerprint and pass phrase of the signing key (Optional) Pass Phrase for the Signing Key are created while generating and uploading the API Signing Key. See How to Get the Key©s Fingerprint in Oracle Cloud Infrastructure Documentation.

Tip:

Make a note of the location of your private key, optional pass phrase, and fingerprint of the public key while generating and uploading the API Signing Key.

After performing the tasks discussed above, collect the credentials information and provide them to your application. Providing the Credentials to your Application: The Oracle NoSQL Database SDKs allow you to provide the credentials to an application in multiple ways. The SDKs support a configuration file as well as one or more interfaces that allow direct specification of the information. See the documentation for the programming language driver that you are using to know about the specific credentials interfaces. If you are using a configuration file, the default location is ~/.oci/config. The SDKs allow the file to reside in alternative locations. It's content looks like this:

[DEFAULT] user=ocid1.user.oc1..aaaaaaaas...7ap fingerprint=d1:b2:32:53:d3:5f:cf:68:2d:6f:8b:5f:77:8f:07:13 key_file=~/.oci/oci_api_key_private.pem tenancy=ocid1.tenancy.oc1..aaaaaaaap...keq pass_phrase=mysecretphrase

The [DEFAULT] line indicates that the lines that follow specify the DEFAULT profile. A configuration file can include multiple profiles, prefixed with [PROFILE_NAME]. For example:

[DEFAULT] user=ocid1.user.oc1..aaaaaaaas...7us fingerprint=d1:b2:32:53:d3:5f:cf:68:2d:6f:8b:5f:77:8f:07:15 key_file=~/.oci/oci_api_key_private.pem tenancy=ocid1.tenancy.oc1..aaaaabbap...keq pass_phrase=mysecretphrase

[MYPROFILE] user=ocid1.user.oc1..aaaaaaaas...7ap fingerprint=d1:b2:32:53:d3:5f:cf:68:2d:6f:8b:5f:77:8f:07:13 key_file=~/.oci/oci_api_key_private.pem

2-2 Chapter 2 Connecting your Application using Java

tenancy=ocid1.tenancy.oc1..aaaaaaaap...keq pass_phrase=mysecretphrase

Connecting your Application using Java

Learn how to connect your Java application to Oracle NoSQL Database Cloud Service. Your application connects to Oracle NoSQL Database Cloud Service by specifying credentials and a target region for the connection. See Acquiring Credentials for the required credentials information. You provide credentials in your application using either: • An API that allows you to directly provide the credentials, or • A configuration file Connecting Using API

/* Use the SignatureProvider to supply your credentials to NoSQL Database. * By default, the SignatureProvider will read your OCI configuration file * from the default location, ~/.oci/config. See SignatureProvider for * additional options for reading configurations in other ways.*/ SignatureProvider sp = new SignatureProvider( tenantId, // a string, OCID userId, // a string, OCID fingerprint , // a string privateKey, // a string, content of private key passPhrase // optional, char[] ); //Create an handle to access the cloud service in the us-ashburn-1 region. NoSQLHandleConfig config = new NoSQLHandleConfig(Region.US_ASHBURN_1); config.setAuthorizationProvider(sp); NoSQLHandle handle = NoSQLHandleFactory.createNoSQLHandle(config);

//At this point, your handle is set up to perform data operations.

Connecting Using a Configuration File

//Create an handle to access the cloud service in the us-ashburn-1 region. NoSQLHandleConfig config = new NoSQLHandleConfig(Region.US_ASHBURN_1); config.setAuthorizationProvider(sp); NoSQLHandle handle = NoSQLHandleFactory.createNoSQLHandle(config);

//At this point, your handle is set up to perform data operations.

2-3 Chapter 2 Connecting your Application using Python

Connecting your Application using Python

Learn how to connect your Python application to Oracle NoSQL Database Cloud Service. Before connecting your application to cloud, see the Prerequisites section in NoSQL Database Python SDK to learn about the credentials required to create the connection, and how to access them. To connect your application using Python, see the Configure for the Cloud Service section in NoSQL Database Python SDK. Connecting your Application using Node.js

Learn how to connect your Node.js application to Oracle NoSQL Database Cloud Service. Before connecting your application to cloud, see the Prerequisites section in NoSQL Database Node.js SDK to learn about the credentials required to create the connection, and how to access them. To connect your application using Node.js, see the Connecting an Application to Oracle NoSQL Database Cloud Service tutorial in NoSQL Database Node.js SDK. Connecting your Application using Go

Learn how to connect your Go application to Oracle NoSQL Database Cloud Service. Before connecting your application to cloud, see the Prerequisites section in NoSQL Database Go SDK to learn about the credentials required to create the connection, and how to access them. To connect your application using Go, see the Connecting an Application to the Cloud Service section in NoSQL Database Go SDK. Data Regions and Associated Service Endpoints

Learn about the data regions supported for Oracle NoSQL Database Cloud Service and access region-specific service endpoints.

Data Regions To start with Oracle NoSQL Database Cloud Service, you must create an account (either for free trial or to purchase provisioning). Along with other details, the account application requires you to choose the default data region.

Service Endpoints Associated with Data Regions A service endpoint is the regional network access point to the Oracle NoSQL Database Cloud Service. The general format of a region endpoint is https://nosql. {region}.oci.oraclecloud.com. For example, the service endpoint for the Ashburn Oracle NoSQL Database Cloud Service region identifier in North America region is https://nosql.us-ashburn-1.oci.oraclecloud.com. Different data regions have different {region} components of their URLs.

2-4 Chapter 2 Data Regions and Associated Service Endpoints

This table lists the service endpoints for all the data regions which are or will be supported by Oracle NoSQL Database Cloud Service. See Service Availability for the latest information about the regions that support Oracle NoSQL Database Cloud Service.

Data Region Region Identifier Service Endpoint North America us-ashburn-1 https://nosql.us- ashburn-1.oci.oraclecloud.com North America us-phoenix-1 https://nosql.us- phoenix-1.oci.oraclecloud.com North America ca-toronto-1 https://nosql.ca- toronto-1.oci.oraclecloud.com North America us-sanjose-1 https://nosql.us- sanjose-1.oci.oraclecloud.com North America ca-montreal-1 https://nosql.ca- montreal-1.oci.oraclecloud.com EMEA eu-frankfurt-1 https://nosql.eu- frankfurt-1.oci.oraclecloud.com EMEA uk-cardiff-1 https://nosql.uk- cardiff-1.oci.oraclecloud.com EMEA uk-london-1 https://nosql.uk- london-1.oci.oraclecloud.com EMEA eu-zurich-1 https://nosql.eu- zurich-1.oci.oraclecloud.com EMEA eu-amsterdam-1 https://nosql.eu- amsterdam-1.oci.oraclecloud.com EMEA me-jeddah-1 https://nosql.me- jeddah-1.oci.oraclecloud.com EMEA me-dubai-1 https://nosql.me- dubai-1.oci.oraclecloud.com APAC ap-seoul-1 https://nosql.ap- seoul-1.oci.oraclecloud.com APAC ap-tokyo-1 https://nosql.ap- tokyo-1.oci.oraclecloud.com APAC ap-mumbai-1 https://nosql.ap- mumbai-1.oci.oraclecloud.com APAC ap-sydney-1 https://nosql.ap- sydney-1.oci.oraclecloud.com APAC ap-hyderabad-1 https://nosql.ap- hyderabad-1.oci.oraclecloud.com APAC ap-osaka-1 https://nosql.ap- osaka-1.oci.oraclecloud.com APAC ap-melbourne-1 https://nosql.ap- melbourne-1.oci.oraclecloud.com APAC ap-chuncheon-1 https://nosql.ap- chuncheon-1.oci.oraclecloud.com LAD sa-saopaulo-1 https://nosql.sa- saopaulo-1.oci.oraclecloud.com LAD sa-santiago-1 https://nosql.sa- santiago-1.oci.oraclecloud.com

2-5 Chapter 2 Data Regions and Associated Service Endpoints

Data Region Region Identifier Service Endpoint LAD sa-vinhedo-1 https://nosql.sa- vinhedo-1.oci.oraclecloud.com

2-6 3 Developing in Oracle Cloud

Learn table design concepts, and develop your application using your driver. When you are ready to work with Oracle Cloud, you create a NoSQL application. Your NoSQL application uses tables to organize data.

Topics • To learn how to design and configure tables in Oracle NoSQL Database Cloud Service, see Table Design. • To learn how to create and manage tables to suit your application, see Table Management. • To learn about the APIs that your application uses to insert, read, and delete the data contained within your tables, see: – Using Tables in Java – Using Tables in Python – Using Tables in Node.js – Using Tables in Go • To learn how to use the SQL query language on tables, see Query Language Reference. • To learn about limits that exist for elements of Oracle NoSQL Database Cloud Service, see Oracle NoSQL Database Cloud Service Limits. • To learn how to estimate and handle capacity, see Estimating Capacity and Handling Capacity. Table Design

Learn how to design and configure tables in Oracle NoSQL Database Cloud Service.

Table A table is a collection of rows, where each row holds a data record. Each table row consists of key and data fields, which are defined when a table is created. In addition, a table has a specified storage, can support a defined maximum read and write throughput, and has a maximum size. Learn about the following concepts before designing a table in Oracle NoSQL Database Cloud Service: • Table Fields: Tables are created using DDL (Data Definition Language) that defines the data types and primary keys used for the table. Oracle NoSQL Database Cloud Service supports multiple data types, including several numeric types, string, binary, timestamp, maps, arrays, records, and a special JSON type which can hold valid JSON data. An application can choose data types for the table fields based on the data model. See Supported Data Types for a complete list of data types supported in Oracle NoSQL Database Cloud Service.

3-1 Chapter 3 Table Design

See Table Fields to learn more about table fields. • Primary and Shard Keys: Every table must have one or more fields designated as the primary key. This designation occurs at the time of table creation, and cannot be changed after the table is created. A primary key uniquely identifies every row in the table. In the simplest case, a primary key is used to retrieve a specific row to examine or modify. Shard keys identify which primary key fields are meaningful in terms of shard storage. That is, rows which contain the same values for all the shard keys are guaranteed to be stored on the same shard. This shard storage matters for some operations that promise atomicity of the results. See Primary Keys and Shard Keys to learn more. • Indexes: Indexes represent an alternative way of retrieving table rows. Normally you retrieve table rows using the primary key. By creating an index, you can more efficiently retrieve rows based on fields that are not part of the primary key. Indexes provide more querying capability, but consume both storage and throughput resources. See Creating Tables and Indexes to learn how to create an index. • Capacity: When you create a table, you also specify throughput and storage resources available for the table. Read and write operations to the table are limited by the read and write throughput capacity that you define. The amount of space that the table can use is limited by the storage capacity. See Estimating Capacity for information on how to estimate the capacity that you should specify for your table. See Handling Capacity to learn how to handle the capacity of your table. • Time to Live (TTL): Time to Live allows you to automatically expire table rows. TTL is expressed as the time data is allowed to live in the table. Data which has reached the expiration timeout value can no longer be retrieved, and does not appear in any read operations. See Time to Live to learn more. • Identity Columns: Identity columns are a special type of columns that get their values automatically assigned by Oracle NoSQL Database Cloud Service. These values are generated from a sequence generator. Identity columns are not supported from the NoSQL Database Cloud Service console. For more details about identity columns, see Identity Column from SQL Reference for Oracle NoSQL Database . To learn how to access identity columns from an Oracle NoSQL Database Cloud Service application, see Inserting IDENTITY Values Programmatically from Getting Started with NoSQL Database Table Java Driver. • Table Life cycles: When tables are created, modified, or deleted, they transition through different table life cycle states. See Table States and Life Cycles to learn about the life cycle of a table. Supported Data Types

Oracle NoSQL Database Cloud Service supports many common data types.

Data Type Description BINARY A sequence of zero or more bytes.

3-2 Chapter 3 Table Design

Data Type Description FIXED_BINARY A fixed-size byte array. BOOLEAN A data type with one of two possible values: TRUE or FALSE. DOUBLE A 64 bit (8 bytes) long floating-point number. FLOAT A 32 bit (4 bytes) long floating point number LONG A 64 bit ( 8 bytes) long integer number. INTEGER A 32 bit (4 bytes) long integer number. STRING A sequence of Unicode characters. NUMBER An arbitrary-precision signed decimal number. TIMESTAMP A point in time with a precision. The precision affects the storage size and usage. Timestamp is stored and managed in UTC (Coordinated Universal Time). ENUM An enumeration, represented as an array of strings. ENUM values are symbolic identifiers (tokens) and are stored as a small integer value representing an ordered position in the enumeration. ARRAY An ordered collection of zero of more typed items. Arrays that are not defined as JSON cannot contain NULL values. Arrays declared as JSON can contain any valid JSON, including the special value, null, which is relevant to JSON. MAP An unordered collection of zero or more key-item pairs, where all keys are strings and all items are the same type. All keys must be unique. The key-item pairs are called fields, the keys are field names, and the associated items are field values. Field values can have different types, but maps cannot contain NULL field values. RECORD A fixed collection of one or more key-item pairs, where all keys are strings. All keys in a record must be unique. JSON Any valid JSON data.

Table Fields

Learn how to design and configure data using table fields. An application may choose to use schemaless tables, where a row consists of key fields and a single JSON data field. A schemaless table offers flexibility in what can be stored in a row. Alternatively, the application can choose to use fixed schema tables, where all of the table fields are defined as specific types. Fixed schema tables with typed data are safer to use from an enforcement and storage efficiency standpoint. Even though the schema of fixed schema tables can be modified, their table structure cannot easily be changed. A schemaless table is flexible and the table structure can be easily modified. Finally, an application can also use a hybrid data model approach where a table can have typed data and JSON data fields. The following examples demonstrate how to design and configure data for all three approaches.

3-3 Chapter 3 Table Design

Example 1: Designing a Schemaless Table You have multiple options to store information about browsing patterns in your table. One option is to define a table that uses a cookie ID as a key and keeps audience segmentation data as a single JSON field.

// schema less, data is stored in a JSON field CREATE TABLE audience_info ( cookie_id LONG, audience_data JSON, PRIMARY KEY(cookie_id))

In this case, the audience_info table can hold a JSON object such as:

{ "cookie_id": "", "audience_data": { "ipaddr" : "10.0.00.xxx", "audience_segment: { "sports_lover" : "2018-11-30", "book_reader" : "2018-12-01" } } }

Your application will have a key field and a data field for this table. You have flexibility in what you chose to store as information in your audience_data field. Therefore, you can easily change the types of information available.

Example 2: Designing a Fixed Schema Table You can store information about browsing patterns by creating your table with more explicitly declared fields:

// fixed schema, data is stored in typed fields. CREATE TABLE audience_info( cookie_id LONG, ipaddr STRING, audience_segment RECORD(sports_lover TIMESTAMP(9), book_reader TIMESTAMP(9)), PRIMARY KEY(cookie_id));

In this example, your table has a key field and two data fields. Your data is more compact, and you are able to ensure that all data fields are accurate.

Example 3: Designing a Hybrid Table You can store information about browsing patterns by using both typed and JSON data fields in your table.

// mixed, data is stored in both typed and JSON fields. CREATE TABLE audience_info ( cookie_id LONG,

3-4 Chapter 3 Table Design

ipaddr STRING, audience_segment JSON, PRIMARY KEY(cookie_id));

Primary Keys and Shard Keys

Learn the purpose of primary keys and shard keys while designing your application. Primary keys and shard keys are important elements in your schema and help you access and distribute data efficiently. You create primary keys and shard keys only when you create a table. They remain in place for the life of the table, and cannot be changed or dropped.

Primary Keys You must designate one or more primary key columns when you create your table. A primary key uniquely identifies every row in the table. For simple CRUD operations, Oracle NoSQL Database Cloud Service uses the primary key to retrieve a specific row to read or modify. For example, consider a table has the following fields: • productName • productType • productLine From experience, you know that the product name is important as well as unique to each row, so you set the productName as the primary key. Then, you retrieve rows of interest based on the productName. In such a case, use a statement like this, to define the table.

/* Create a new table called users. */ CREATE TABLE if not exists myProducts ( productName STRING, productType STRING, productLine INTEGER, PRIMARY KEY (productName) )";

Shard Keys The main purpose of shard keys is to distribute data across the Oracle NoSQL Database Cloud Service cluster for increased efficiency, and to position records that share the shard key locally for easy reference and access. Records that share the shard key are stored in the same physical location and can be accessed atomically and efficiently. Your Primary and shard key design has implications on scaling and achieving provisioned throughput. For example, when records share shard keys, you can delete multiple table rows in an atomic operation, or retrieve a subset of rows in your table in a single atomic operation. In addition to enabling scalability, well-designed shard keys can improve performance by requiring fewer cycles to put data to, or get data from, a single shard. For example, suppose that you designate three primary key fields:

PRIMARY KEY (productName, productType, productLine)

3-5 Chapter 3 Table Design

Because you know that your application frequently makes queries using the productName and productType columns, specifying those fields as shard keys is appropriate. The shard key designation guarantees that all rows for these two columns are stored on the same shard. If these two fields are not shard keys, the most frequently queried columns could be stored on any shard. Then, locating all rows for both fields requires scanning all data storage, rather than one shard. Shard keys designate storage on the same shard to facilitate efficient queries for key values. However, because you want your data be distributed across the shards for best performance, you must avoid shard keys that have few unique values.

Note:

If you do not designate shard keys when creating a table, Oracle NoSQL Database Cloud Service uses the primary keys for shard organization.

Important factors to consider when choosing a shard key • Cardinality: Low cardinality fields, such as a user home country, group records together on a few shards. In turn, those shards require frequent data rebalancing, increasing the likelihood of hot shard issues. Instead, each shard key should have high cardinality, where the shard key can express an even slice of records in the data set. For example, identity numbers such as customerID, userID, or productID are good candidates for a shard key. • Atomicity: Only objects that share the shard key can participate in a transaction. If you require ACID transactions that span multiple records, choose a shard key that lets you meet that requirement.

What are the best practices to follow? • Uniform distribution of shard keys: When shard keys are uniformly distributed, no single shard limits the capacity of the system. • Query Isolation: Queries should be targeted to a specific shard to maximize efficiency and performance. If queries are not isolated to a single shard, the query is applied to all shards which is less efficient and increases query latency. See Creating Tables and Indexes to learn how to assign primary and shard keys using the TableRequest object. Time to Live

Learn how to specify expiration times for tables and rows using the Time-to-Live (TTL) feature. Many applications handle data that has a limited useful lifetime. Time-to-Live (TTL) is a mechanism that allows you to set a time frame on table rows, after which the rows expire automatically, and are no longer available. It is the amount of time data is allowed to remain in the Oracle NoSQL Database Cloud Service. Data that reaches expiration time can no longer be retrieved, and does not appear in any storage statistics. By default, every table that you create has a TTL value of zero, indicating no expiration time. You can declare a TTL value when you create a table, specifying the TTL with a number, followed by either HOURS or DAYS. Table rows inherit the TTL value of the table

3-6 Chapter 3 Table Design

in which they reside, unless you explicitly set a TTL value for table rows. Setting a row's TTL value overrides the table's TTL value. If you change the table's TTL value after the row has a TTL value, the row's TTL value persists. You can update the TTL value for a table row at any time before the row reaches the expiration time. Expired data can no longer be accessed. Therefore, using TTL values is more efficient than manually deleting rows, because the overhead of writing a database log entry for the data deletion is avoided. Expired data is purged from the disk after expiration date. Table States and Life Cycles

Learn about the different table states and their significance (table life cycle process). Each table passes through a series of different states from table creation to deletion (drop). For example, a table in the DROPPING state cannot proceed to the ACTIVE state, while a table in the ACTIVE state can change to the UPDATING state. You can track the different table states by monitoring the table life cycle. This section describes the various table states.

Table State Description CREATING The table is in the process of being created. It is not ready to use. UPDATING Updating the table is in process. Further table modifications are not possible while the table is in this state. A table is in the UPDATING state when: · The table limits are being changed · The table schema is evolving · Adding or dropping a table index ACTIVE The table can be used in the current state. The table may have been recently created, or modified, but the table state is now stable. DROPPING The table is being dropped and cannot be accessed for any purpose.

3-7 Chapter 3 Table Management

Table State Description DROPPED The table has been dropped and no longer exists for read, write, or query activities.

Note:

Once dropped, a table with the same name can be created again.

Table Management

Learn how to design tables and use Data Definition Language to work with tables. You can create, modify, and delete NoSQL tables from your application. You can also add indexes to optimize your application common access paths. The table DDL is what you use to specify these actions. • To view the range of options that you have for designing and configuring your tables and indexes, see Query Language Reference. • To view the complete description of the table DDL, see Data Definition Language Reference. The language driver of your choice executes your DDL statement. See Using Tables in Java for examples of how to issue your DDL statements. Data Definition Language Reference

Learn how to use DDL in Oracle NoSQL Database Cloud Service. Use Oracle NoSQL Database Cloud Service DDL to create, alter, and drop tables and indexes. For information on the syntax of the DDL language, see Table Data Definition Language Guide. This guide documents the DDL language as supported by the on- premises Oracle NoSQL Database product. The Oracle NoSQL Database Cloud Service supports a subset of this functionality and the differences are documented in the DDL Differences in the Cloud section. Also, each NoSQL driver provides an API to execute a DDL statement. To write your application, see Using Tables in Java.

Typical DDL Statements Few samples of common DDL statements are as follows: Create Table

CREATE TABLE [IF NOT EXISTS] ( field-definition, field-definition-2 ..., PRIMARY KEY (field-name, field-name-2...), ) [USING TTL ttl]

3-8 Chapter 3 Table Management

For example:

CREATE TABLE IF NOT EXISTS audience_info ( cookie_id LONG, ipaddr STRING, audience_segment JSON, PRIMARY KEY(cookie_id));

Alter Table

ALTER TABLE table-name (ADD field-definition) ALTER TABLE table-name (DROP field-name) ALTER TABLE table-name USING TTL ttl

For example:

ALTER TABLE audience_info USING TTL 7 days;

Create Index

CREATE INDEX [IF NOT EXISTS] index-name ON table-name (path_list)

For example:

CREATE INDEX segmentIdx ON audience_info (audience_segment.sports_lover AS STRING)

Drop Table

DROP TABLE [IF EXISTS] table-name

For example:

DROP TABLE audience_info;

See the reference guides for a complete list: • Table Data Definition Language guide • SQL Reference for Oracle NoSQL Database

DDL Differences in the Cloud The cloud service DDL language differs from what is described in the reference guide in the following way: Table Names • Limited to 256 characters, and are restricted to alphanumeric characters and underscore • Must start with a letter • Cannot include special characters

3-9 Chapter 3 Using Tables in Java

• Child tables are not supported Unsupported Concepts • DESCRIBE and SHOW TABLE statements. • Full text indexes • User and role management • On-premise regions Using Tables in Java

Learn how to create, update, and delete tables from your Java application.

Topics • About the Oracle NoSQL Database Java SDK • About Compartments • Obtaining a NoSQL Handle • Creating Tables and Indexes • Adding Data • Reading Data • Using Queries • Deleting Data • Modifying Tables • Dropping Tables and Indexes • Handling Errors About the Oracle NoSQL Database Java SDK

Learn about the Oracle NoSQL Database Java SDK. The Oracle NoSQL Database Java Driver contains the jar files that enable an application to communicate with the on-premises or the Oracle NoSQL Database Cloud Service or the Oracle NoSQL Database Cloud Simulator. Download the SDK for Java from GitHub. The Oracle NoSQL SDK for Java provides you with all the Java classes, methods, interfaces and examples. Documentation is available as javadoc in GitHub or from Java API Reference Guide. About Compartments

Learn how to specify the compartment while creating and working with Oracle NoSQL Database Cloud Service tables using the Oracle NoSQL Database Java Driver. Oracle NoSQL Database Cloud Service tables are created in a compartment and are scoped to that compartment. When authenticated as a specific user, your tables are managed in the root compartment of your tenancy unless otherwise specified. Organizing tables into different compartments will help with respect to organization and security.

3-10 Chapter 3 Using Tables in Java

If you have been authenticated using an instance principal (accessing the service from an OCI compute instance), you must specify a compartment using its id (OCID), as there is no default in this case. See Calling Service From an Instance in Oracle Cloud Infrastructure Documentation. There are several ways to specify a compartment in your application code: 1. Use a default compartment in NoSQLHandleConfig so that it applies to all the operations using the handle. See Obtaining a NoSQL Handle for an example. 2. Use the compartment name or id (OCID) in each request in addition to the table name. This overrides any default compartment. For example:

GetRequest getReq = new GetRequest().setTableName("mytable") .setCompartment("mycompartment");

3. Use the compartment name as a prefix on the table name. This overrides any default compartment as well as a compartment specified using API. For example:

GetRequest getReq = new GetRequest().setTableName("mycompartment:mytable");

When using a named compartment, the name can be the simple name of a top-level compartment or a path to a nested compartment. In the latter case, the path is a "." (dot) separated path.

Note:

While specifying the path to a nested compartment, do not include the top-level compartment's name in the path as that is inferred from the tenancy.

Obtaining a NoSQL Handle

Learn how to access tables using the Oracle NoSQL Database Java Driver. Start developing your application by creating a NoSQL Handle. Use the NoSQLHandle to access the tables and execute all operations. To create a connection represented by a NoSQLHandle, obtain a handle using the NoSQLHandleFactory.createNoSQLHandle method and the NoSQLHandleConfig class. The NoSQLHandleConfig class allows an application to specify the handle configuration. See the Java API Reference guide to learn more.

Obtain a NoSQL Handle Use the following code to obtain a NoSQL handle:

/* Configure a handle for the desired Region and AuthorizationProvider. * By default this SignatureProvider constructor reads authorization * information from ~/.oci/config and uses the default user profile and * private key for request signing. Additional SignatureProvider * constructors are available if a config file is not available or * desirable. */

3-11 Chapter 3 Using Tables in Java

AuthorizationProvider ap = new SignatureProvider();

/* Use the us-ashburn-1 region */ NoSQLHandleConfig config = new NoSQLHandleConfig(Region.US - ASHBURN - 1, ap); config.setAuthorizationProvider(ap);

/* Sets a default compartment for all requests from this handle. This * may be overridden in individual requests or by using a * compartment-name prefixed table name. */ config.setDefaultCompartment("mycompartment");

// Open the handle NoSQLHandle handle = NoSQLHandleFactory.createNoSQLHandle(config);

// Use the handle to execute operations

A handle has memory and network resources associated with it. Use the NoSQLHandle.close method to free up the resources when your application is done using the handle. To minimize network activity and resource allocation and deallocation overheads, it's best to avoid creating and closing handles repeatedly. For example, creating and closing a handle around each operation would result in poor application performance. A handle permits concurrent operations, so a single handle is sufficient to access tables in a multi-threaded application. The creation of multiple handles incurs additional resource overheads without providing any performance benefit. Creating Tables and Indexes

Learn how to create tables and indexes. Creating a table is the first step of developing your application. You use the TableRequest class and methods to execute all DDL statements, such as, creating, modifying, and dropping tables. You also set table limits using the TableRequest.setTableLimits method. Before creating a table, see: • Supported Data Types, and • Oracle NoSQL Database Cloud Service Limits The TableRequest class lets you pass a DDL statement to the TableRequest.setStatement method. Examples of DDL statements are:

/* Create a new table called users */ CREATE IF NOT EXISTS users(id INTEGER, name STRING, PRIMARY KEY(id));

/* Create a new table called users and set the TTL value to 4 days */ CREATE IF NOT EXISTS users(id INTEGER, name STRING, PRIMARY KEY(id)) USING TTL 4 days;

3-12 Chapter 3 Using Tables in Java

/* Create a new index called nameIdx on the name field in the users table */ CREATE INDEX IF NOT EXISTS nameIdx ON users(name);

Note:

The following example considers that the default compartment is specified in NoSQLHandleConfig while obtaining the NoSQL handle. See Obtaining a NoSQL Handle. To explore other options of specifying a compartment for the NoSQL tables, see About Compartments.

To create a table and index using the TableRequest and its methods:

/* Create a simple table with an integer key and a single json data * field and set your desired table capacity. * Set the table TTL value to 3 days. */ String createTableDDL = "CREATE TABLE IF NOT EXISTS users " + "(id INTEGER, name STRING, " + "PRIMARY KEY(id)) USING TTL 3 days";

TableLimits limits = new TableLimits(200, 100, 5); TableRequest treq = new TableRequest().setStatement(createTableDDL) .setTableLimits(limits);

// start the asynchronous operation TableResult tres = handle.tableRequest(treq);

// wait for completion of the operation tres.waitForCompletion(handle, 60000, // wait for 60 sec 1000); // delay in ms for poll

// Create an index called nameIdx on the name field in the users table. treq = new TableRequest().setStatement("CREATE INDEX IF NOT EXISTS nameIdx ON users(name) ");

// start the asynchronous operation handle.tableRequest(treq);

// wait for completion of the operation tres.waitForCompletion(handle, 60000, // wait for 60 sec 1000); // delay in ms for poll

Related Topics • About Time to Live

3-13 Chapter 3 Using Tables in Java

Adding Data

Add rows to your table. When you store data in table rows, your application can easily retrieve, add to, or delete information from a table. The PutRequest class represents the input to a NoSQLHandle.put(oracle.nosql.driver.ops.PutRequest) operation. This request can be used to perform unconditional and conditional puts to: • Overwrite any existing row. Overwrite is the default functionality. • Succeed only if the row does not exist. Use the PutRequest.Option.IfAbsent method in this case. • Succeed only if the row exists. Use the PutRequest.Option.IfPresent method in this case. • Succeed only if the row exists and the version matches a specific version. Use the PutRequest.Option.IfVersion method for this case and the setMatchVersion(oracle.nosql.driver.Version) method to specify the version to match.

Note:

First, connect your client driver to Oracle NoSQL Database Cloud Service to get a handle and then complete other steps. This topic omits the steps for connecting your client driver and creating a table. If you do not yet have a table, see Creating Tables and Indexes.

The following example assumes that the default compartment is specified in NoSQLHandleConfig while obtaining the NoSQL handle. See Obtaining a NoSQL Handle. To explore other options of specifying a compartment for the NoSQL tables, see About Compartments. To add rows to your table:

/* use the MapValue class and input the contents of a new row */ MapValue value = new MapValue().put("id", 1).put("name", "myname");

/* create the PutRequest, setting the required value and table name */ PutRequest putRequest = new PutRequest().setValue(value) .setTableName("users");

/* use the handle to execute the PUT request * on success, PutResult.getVersion() returns a non-null value */ PutResult putRes = handle.put(putRequest); if (putRes.getVersion() != null) { // success } else { // failure }

3-14 Chapter 3 Using Tables in Java

You can perform a sequence of PutRequest operations on a table that share the shard key using the WriteMultipleRequest class. If the operation is successful, the WriteMultipleResult.getSuccess() method returns true. See the Java API Reference Guide for more information about the APIs. You can also add JSON data to your table. You can either convert JSON data into a record for a fixed schema table or you can insert JSON data into a column whose data type is of type JSON. See Adding JSON Data. Adding JSON Data

Learn how to add JSON data to a fixed schema table.

Note:

Table rows are added to the table by using APIs which let you individually specify each table field value. For example, you use the MapValue.put() method to fill in each field value for a row, before inserting the entire row into the table. The PutRequest class also provides the setValueFromJson method which takes a JSON string and uses that to populate a row to insert into the table. The JSON string should specify field names that correspond to the table field names.

Note:

To add JSON data to your table:

/* Construct a simple row, specifying the values for each * field. The value for the row is this: * * { * "cookie_id": 123, * "audience_data": { * "ipaddr": "10.0.00.xxx", * "audience_segment": { * "sports_lover": "2018-11-30", * "book_reader": "2018-12-01" * } * } * } */

3-15 Chapter 3 Using Tables in Java

MapValue segments = new MapValue() .put("sports_lover", new TimestampValue("2018-11-30")) .put("book_reader", new TimestampValue("2018-12-01")); MapValue value = new MapValue() .put("cookie_id", 123) // fill in cookie_id field .put("ipaddr", "10.0.00.xxx") .put("audience_segment", segments); PutRequest putRequest = new PutRequest() .setValue(value) .setTableName(tableName); PutResult putRes = handle.put(putRequest);

The same row can be inserted into the table as a JSON string:

/* Construct a simple row in JSON */ String jsonString = "{\"cookie_id\":123,\"ipaddr\":\"10.0.00.xxx\", \"audience_segment\": {\"sports_lover\":\"2018-11-30\", \"book_reader\":\"2018-12-01\"}}"; PutRequest putRequest = new PutRequest() .setValueFromJson(jsonString, null) // no options .setTableName(tableName); PutResult putRes = handle.put(putRequest);

Reading Data

Learn how to read data from your table. You can read data from your application by using the NoSQLHandle.get() method. This method allows you to retrieve a record based on a single primary key value, or by using queries. The GetRequest class provides a simple and powerful way to read data, while queries can be used for more complex read requests.

Note:

To read data from a table, specify the target table and target key using the GetRequest class and use NoSQLHandle.get() to execute your request. The result of the operation is available in GetResult.

/* GET the row, first create the row key */ MapValue key = new MapValue().put("id", 1);

3-16 Chapter 3 Using Tables in Java

GetRequest getRequest = new GetRequest().setKey(key) .setTableName("users"); GetResult getRes = handle.get(getRequest);

/* on success, GetResult.getValue() returns a non-null value */ if (getRes.getValue() != null) { // success } else { // failure }

Note:

By default, all read operations are eventually consistent. You can change the default Consistency for a NoSQLHandle instance by using the NoSQLHandleConfig.setConsistency(oracle.nosql.driver.Consistency) and GetRequest.setConsistency() methods.

See the Java API Reference Guide for more information about the GET (read) APIs. Using Queries

Learn about some aspects of using queries to your application in Oracle NoSQL Database Cloud Service. Oracle NoSQL Database Cloud Service provides a rich query language to read and update data. See SQL Reference for NoSQL Database for a full description of the query language. To execute your query, you use the NoSQLHandle.query() API. See the Java API Reference Guide for more information about this API.

Note:

The following examples consider that the default compartment is specified in NoSQLHandleConfig while obtaining the NoSQL handle. See Obtaining a NoSQL Handle. To explore other options of specifying a compartment for the NoSQL tables, see About Compartments.

To execute a SELECT query to read data from your table:

/* QUERY a table named "users", using the primary key field "name". * The table name is inferred from the query statement. */ QueryRequest queryRequest = new QueryRequest(). setStatement("SELECT * FROM users WHERE name = \"Taylor\"");

/* Queries can return partial results. It is necessary to loop, * reissuing the request until it is "done" */

do {

3-17 Chapter 3 Using Tables in Java

QueryResult queryResult = handle.query(queryRequest);

/* process current set of results */ List results = queryResult.getResults(); for (MapValue qval : results) { //handle result } } while (!queryRequest.isDone());

When using queries, be aware of the following considerations: • You can use prepared queries when you want to run the same query multiple times. When you use prepared queries, the execution is more efficient than starting with a query string every time. The query language and API support query variables to assist with the reuse. See NoSQLHandle.prepare in the Java API Reference Guide for more information. • You can set important query attributes, such as the usable amount of resources, or the read consistency used by the read operations, using the QueryRequest class. See QueryRequest in the Java API Reference Guide for more information. For example, to execute a SELECT query to read data from your table using a prepared statement:

/* Perform the same query using a prepared statement. This is more * efficient if the query is executed repeatedly and required if * the query contains any bind variables. */ String query = "DECLARE $name STRING; " + "SELECT * from users WHERE name = $name";

PrepareRequest prepReq = new PrepareRequest().setStatement(query);

/* prepare the statement */ PrepareResult prepRes = handle.prepare(prepReq);

/* set the bind variable and set the statement in the QueryRequest */ prepRes.getPreparedStatement() .setVariable("$name", new StringValue("Taylor")); QueryRequest queryRequest = new QueryRequest().setPreparedStatement(prepRes);

/* perform the query in a loop until done */ do { QueryResult queryResult = handle.query(queryRequest); /* handle result */ } while (!queryRequest.isDone());

Deleting Data

Learn how to delete rows from your table. After you insert or load data into a table, you can delete the table rows when they are no longer required.

3-18 Chapter 3 Using Tables in Java

Note:

/* identify the row to delete */ MapValue delKey = new MapValue().put("id", 2);

/* construct the DeleteRequest */ DeleteRequest delRequest = new DeleteRequest().setKey(delKey) .setTableName("users");

/* Use the NoSQL handle to execute the delete request */ DeleteResult del = handle.delete(delRequest);

/* on success DeleteResult.getSuccess() returns true */ if (del.getSuccess()) { // success, row was deleted } else { // failure, row either did not exist or conditional delete failed }

You can perform a sequence of DeleteRequest operations on a table using the MultiDeleteRequest class. See the Java API Reference Guide for more information about the APIs. Modifying Tables

Learn how to modify tables. You modify a table to: • Add new fields to an existing table • Delete currently existing fields in a table • To change the default TTL value • Modify table limits

3-19 Chapter 3 Using Tables in Java

Note:

Examples of DDL statements are:

/* Add a new field to the table */ ALTER TABLE users (ADD age INTEGER);

/* Drop an existing field from the table */ ALTER TABLE users (DROP age);

/* Modify the default TTL value*/ ALTER TABLE users USING TTL 4 days;

Note:

When altering a table, you may also use the TableRequests.setTableLimits method to modify table limits. For example:

/* Alter the users table to modify the TTL value to 4 days. * When modifying the table schema or other table state you cannot also * modify the table limits. These must be independent operations. */ String alterTableDDL = "ALTER TABLE users " + "USING TTL 4 days"; TableRequest treq = new TableRequest().setStatement(alterTableDDL);

/* start the operation, it is asynchronous */ TableResult tres = handle.tableRequest(treq);

/* wait for completion of the operation */ tres.waitForCompletion(handle, 60000, /* wait for 60 sec */ 1000); /* delay in ms for poll */

3-20 Chapter 3 Using Tables in Java

Dropping Tables and Indexes

Learn how to delete a table or index that you have created in Oracle NoSQL Database Cloud Service. To drop a table in Oracle NoSQL Database Cloud Service, you must have the NOSQL_TABLE_DROP permission. See Details for Verb + Resource-Type Combinations to learn about different permissions. To drop a table or index, use the DROP TABLE or DROP INDEX DDL statements. For example:

/* Drop the table named users */ DROP TABLE users;

/* Drop the index called nameIndex on the table users */ DROP INDEX IF EXISTS nameIndex ON users;

Note:

To drop a table using the TableRequests.setStatement method:

/* create the TableRequest to drop the users table */ TableRequest tableRequest = new TableRequest().setStatement("drop table users");

/* start the operation, it is asynchronous */ TableResult tres = handle.tableRequest(tableRequest);

/* wait for completion of the operation */ tres.waitForCompletion(handle, 60000, /* wait for 60 sec */ 1000); /* delay in ms for poll */

Handling Errors

Learn how to handle errors and exceptions. Java errors are thrown as exceptions when you build or run your application. The NoSQlException class is the base for most exceptions thrown by the driver. However, the driver throws exceptions directly for some classes, such as IllegalArgumentException and NullPointerException. In general, NoSQL exception instances are split into two broad categories: • Exceptions that may be retried with the expectation that they may succeed on retry.

3-21 Chapter 3 Using Tables in Python

These exceptions are instances of the RetryableException class. These exceptions usually indicate resource consumption violations such as ThrottlingException. • Exceptions that will fail even after retry. Examples of exceptions that should not be retried are IllegalArgumentException, TableNotFoundException, and any other exception indicating a syntactic or semantic error. See the Java API Reference guide to learn more about these exceptions and how to handle them. Using Tables in Python

Learn how to create, update, and delete tables from your Python application. Oracle NoSQL Database Cloud Service provides a Python SDK that enables your Python application to create, update, and drop tables as well as add, read, and delete data in the tables. See Working with Tables in NoSQL Database Python SDK documentation. Using Tables in Node.js

Learn how to create, update, and delete tables from your Node.js application. Oracle NoSQL Database Cloud Service provides a Node.js SDK that enables your Node.js application to create, update, and drop tables as well as add, read, and delete data in the tables. See Working with Tables in NoSQL Database Node.js SDK documentation. Using Tables in Go

Learn how to create, update, and delete tables from your Go application. Oracle NoSQL Database Cloud Service provides a Go SDK that enables your Go application to create, update, and drop tables as well as add, read, and delete data in the tables. See a simple example that demonstrates working with tables in NoSQL Database Go SDK documentation. Query Language Reference

Learn how to use SQL statements to update and query data in Oracle NoSQL Database Cloud Service. The Oracle NoSQL Database uses the SQL query language to update and query data in NoSQL tables. See SQL Reference for Oracle NoSQL Database to learn the query language syntax.

3-22 Chapter 3 Oracle NoSQL Database Cloud Service Limits

Note:

The SQL Reference for Oracle NoSQL Database documents the SQL query language as supported by the on-premise Oracle NoSQL Database product. Oracle NoSQL Database Cloud Service supports only a subset of this functionality.

Typical Queries

SELECT FROM

[WHERE ] [GROUP BY ] [ORDER BY []] [LIMIT ] [OFFSET ];

For example: SELECT * FROM Users; SELECT id, firstname, lastname FROM Users WHERE firstname = "Taylor";

UPDATE [AS ] [, ]* WHERE [];

For example: UPDATE JSONPersons $j SET TTL 1 DAYS WHERE id = 6 RETURNING remaining_days($j) AS Expires;

Query Language Differences in the Cloud The cloud service query support differs from what is described in the query language reference guide in the following way: Exclusions Joins are not available since child tables are not supported in Oracle NoSQL Database Cloud Service. Restrictions on Expressions Used in the SELECT Clause Oracle NoSQL Database Cloud Service supports grouping expressions or arithmetic expressions among aggregate functions. No other kinds of expressions are allowed in the SELECT clause. For example, CASE expressions are not allowed in the SELECT clause. Each NoSQL Database driver provides an API to execute a query statement. To write your application and use query language, see Using Tables in Java. Oracle NoSQL Database Cloud Service Limits

Current limits that apply to Oracle NoSQL Database Cloud Service.

3-23 Chapter 3 Oracle NoSQL Database Cloud Service Limits

Scope Description Value Table Maximum total storage size per tenant. The 5 TB total space used for one or more tables cannot exceed this value. Table Names Maximum number of characters, allowed Table names can have a maximum of 256 characters, and initial character. characters. All names must begin with a letter (a±z, A±Z). Subsequent characters can be letters (a±z, A±Z), digits (0±9), or underscore. Table Maximum read and write throughput. 40,000 read units and 20,000 write units per table. Tenant Maximum number of tables 30 Table Maximum number columns 50 Table Maximum number of table schema updates 100 Table Maximum number of indexes 5 Table Maximum number of changes for throughput Oracle allows: and storage limits · Unlimited number of throughput and storage increases per day · Up to four throughput or storage decreases per 24-hour period. Index Names Maximum number of characters, allowed Index names can have a maximum of 64 characters, and initial character. characters. All names must begin with a letter (a±z, A±Z). Subsequent characters can be letters (a±z, A±Z), digits (0±9), or underscore. Request Maximum number of individual operations 50 per WriteMultiple request. Request Maximum data size for WriteMultiple 25 MB request. Field Names Maximum number of characters, allowed Field names can have a maximum of 64 characters, and initial character. characters. All names must begin with a letter (a±z, A±Z). Subsequent characters can be letters (a±z, A±Z), digits (0±9), or underscore. Field Maximum index key size. 64 bytes Field Maximum primary key size. 64 bytes Row Maximum row size. 512 KB Query Maximum query string length. 10 KB Tenant Maximum supported rate of DDL operations. 4 per minute Tenant Maximum values for throughput and data Per tenant, Oracle allows: storage resources. · A maximum of 100,000 read units · A maximum of 40,000 write units Oracle allows a maximum storage size of 5- TB per-tenant. The tenant can have a single table with storage size of 5 TB, in which case the tenant is not able to create another table. Or have multiple tables and ensure that the data within all these tables is within the maximum storage size of 5 TB.

3-24 Chapter 3 Estimating Capacity

Estimating Capacity

Learn how to estimate throughput and storage capacities for your Oracle NoSQL Database Cloud Service.

Basics Behind the Calculation Before you learn how to estimate throughput and storage for the service, let's review the throughput and storage unit definitions. • Write Unit (WU): One Write Unit is defined as throughput for up to 1 kilobyte (KB) of data per second. A write operation is any Oracle NoSQL Database Cloud Service API call that results in insertion, update, or deletion of a record. A NoSQL table has a write limit value which specifies the number of write units that may be used each second. Index updates also consume write units. For example, a record size of less than 1 KB requires one WU for a write operation. A record size of 1.5 KB requires two WUs for the write operation. • Read Unit (RU): One Read Unit is defined as throughput for up to 1 KB of data per second for an eventually consistent read operation. Your NoSQL table has a read limit value which specifies the number of read units that may be used each second. For example, a record size of less than 1 KB requires one RU for an eventually consistent read operation. A record size of 1.5 KB requires two RUs for an eventually consistent read operation and four RUs for an absolutely consistent read operation. • Storage Capacity: One storage unit is a single gigabyte (GB) of data storage. • Absolute Consistency: The data returned is expected to be the most recently written data to the database. • Eventual Consistency: The data returned may not be the most recently written data to the database; if no new updates are made to the data, eventually all accesses to that data return the latest updated value.

Factors that Impact the Capacity Unit Before you provision the capacity units, it is important to consider the following factors that impact the read, write, and storage capacities. • Record size: As the record size increases, the number of capacity units consumed to write or read data also increases. • Data consistency: Absolute consistency reads are twice the cost of eventual consistency reads. • Secondary Indexes: In a table, when an existing record is modified (added, updated, or deleted), updating secondary indexes consume Write Units. The total provisioned throughput cost for a write operation is the sum of write units consumed by writing to the table and updating the local secondary indexes. • Data modeling choice: With schema-less JSON, each document is self-describing which adds metadata overhead to the overall size of the record. With fixed schema tables, the overhead for each record is exactly 1 byte. • Query pattern: The cost of a query operation depends on the number of rows retrieved, number of predicates, the size of the source data, projections, and the presence of indexes. The least expensive queries specify a shard key or index key (with an associated index) to allow the system to take advantage of primary and secondary

3-25 Chapter 3 Estimating Capacity

indexes. An application can try different queries and examine the consumed throughput to help tune the operations.

Real World Example: How to Estimate your Application Workload Consider an E-commerce application example to learn how to estimate reads and writes per second. In this example, Oracle NoSQL Database Cloud Service is used to store the product catalog information of the application.

1. Identify the data model (JSON or fixed-table), record size, and key size for the application. Assume that the E-commerce application follows the JSON data model and the developer has created a simple table with two columns. A record identifier (primary key) and a JSON document for the product features and attributes. The JSON document, which is under 1 KB is as follows:

{ "additionalFeatures": "Front Facing 1.3MP Camera", "os": "Macintosh OS X 10.7", "battery": { "type": "Lithium Ion (Li-Ion) (7000 mAH)", "standbytime" : "24 hours" }, "camera": { "features": ["Flash","Video"], "primary": "5.0 megapixels" }, "connectivity": { "bluetooth": "Bluetooth 2.1", "cell": "T-mobile HSPA+ @ 2100/1900/AWS/850 MHz", "gps": true, "infrared": false, "wifi": "802.11 b/g" }, "description": "Apple iBook is the best in class computer for your professional and personal work.", "display": { "screenResolution": "WVGA (1280 x 968)", "screenSize": "13.0 inches" }, "hardware": { "accelerometer": true, "audioJack": "3.5mm", "cpu": "Intel i7 2.5 GHz", "fmRadio": false, "physicalKeyboard": false, "usb": "USB 3.0" }, "id": "appleproduct_1", "images": ["img/apple-laptop.jpg"], "name": "Myshop.com : Apple iBook", "sizeAndWeight": { "dimensions": [ "300 mm (w)", "300 mm (h)", "12.4 mm (d)" ], "weight": "1250.0 grams" }, "storage": { "hdd": "750GB",

3-26 Chapter 3 Estimating Capacity

"ram": "8GB" } }

Assume that the application has 100,000 such records and the primary key is about 20 bytes in size. Also, assume that there are queries that would read records via secondary index. For example, to find all the records that have screen size of 13 inches. So, an index is created on the screenSize field. The information in summarized as follows:

Tables Rows per Columns per Key Size in Value Size in Indexes Index Key Table Table Bytes Bytes (sum Size in Bytes of all columns) 1 100000 2 20 2 KB 1 20

2. Identify the list of operations (typically CRUD operations and Index reads) on the table and at what rate (per second) they are expected.

Operation Number of Example Operations (per second) Create Records. 3 To create a product. Read records using the primary key. 200 To read product details using the product ID. Read records using the secondary index. 1 To fetch all products that have a screen size of 13 inches. Update or add an attribute to a record. 5 To update the product description of a camera or To add information about the weight of a camera. Delete record. 5 To delete an existing product.

3. Identify the read and write consumption in KB.

Operation Assumptions (If Formula Read Consumption Write any) in KB Consumption in KB Create Records. Assume that the Record size (rounded 0 1 KB + 1 KB (1 ) records are to next KB) + 1 = 2 KB created without KB(index) * (number performing any of indexes) condition checks (if they exist). Read records using Minimum Read (1 KB) + Record size = 1 KB 0 the primary key. Record size round up to KB + Overhead of 2 KB

3-27 Chapter 3 Estimating Capacity

Operation Assumptions (If Formula Read Consumption Write any) in KB Consumption in KB Read records using Assume that 100 [1 KB(index) + 100 * (1 KB + 1 KB) = 0 the secondary index. records are record_size] * 200 KB returned. number_of_records_mat 200 KB + 10 KB = 210 ched KB Where 10 KB accounts for variable overhead that may occur depending on the number of batches returned and the size limit set for the query. Update existing Assume that the Read consumption = [1 2 KB + 2 KB = 4 KB 1 KB + 1 KB + 1 records updated record KB(index) * 2] + KB = 3 KB size is the same [record_size * 2] as the old record Write consumption = size (1 KB). original_record_size + new_record_size + 1 KB (index) * (number of writes) Delete record Read consumption = 1 1 KB + 2 KB = 2 KB 1 KB + 1 KB (1 KB (index) * 2 index) = 2 KB Write consumption = record_size + 1KB (index) * (number_of_indexes)

Using steps 2 and 3, let us determine read and write units for our application workload.

Operations Rate of Operations Reads per Second Writes per Second Create records 3 0 6 Read records using the primary Key 300 300 0 Read records using the secondary index 10 2100 0 Update existing record 5 20 15 Delete record 1 2 2

Total Read Units: 2422 Total Write Units: 23 Therefore, the E-commerce application is estimated to have a workload of 2422 reads per second and 23 writes per second. Download the Capacity Estimator tool available on Oracle Technology Network to input these values and estimate throughput and storage of your application.

3-28 Chapter 3 Handling Capacity

Note:

The preceding calculations assume eventually consistent read requests. For an absolute consistency read request, the operation consumes double the capacity units. Therefore, the read capacity units would be 4844 Read Units.

Handling Capacity

Learn how to handle throughput and storage consumed by your table. Before you run your application in the Oracle NoSQL Database Cloud Service, estimate throughput and storage capacity of the table as described in Estimating Capacity.

Setting Table Limits When you create a table, you use the setTableLimits method within the TableRequest class to specify throughput and capacity that the table consumes. You can also change the table limits of an existing table by using the same class. See the Java API Reference guide for more information. To change limits after a table exists:

/* Create a new TableLimits object, setting values for read, write, and storage units */ TableLimits newLimits = new TableLimits(30, 10, 10);

/* create the TableRequest object. Set the table limits and the table name.*/ TableRequest treq = new TableRequest().setTableLimits(newLimits). setTableName(tableName); TableResult tres = handle.tableRequest(treq); System.out.println("Altering table limits");

/* Wait for the operation to complete */ tres.waitForCompletion(handle, 20000, 1000);

To read the new table limits, use the GetTableRequest class:

GetTableRequest gtr = new GetTableRequest().setTableName(tableName); tres = handle.getTable(gtr); System.out.println("New table limits: " + "read units=" + tres.getTableLimits().getReadUnits() + ", write units=" + tres.getTableLimits().getWriteUnits() + ", table size=" + tres.getTableLimits().getStorageGB() );

See the NoSQLHandle.getTableUsage API in the Java API Reference guide to get information about your table usage characteristics.

3-29 4 Developing in Oracle NoSQL Database Cloud Simulator

Get familiar with the Cloud APIs by using the Oracle NoSQL Database Cloud Simulator. The Oracle NoSQL Database Cloud Simulator simulates the cloud service and lets you write and test applications locally without accessing Oracle NoSQL Database Cloud Service. The Oracle NoSQL Database Java SDK contains a few examples for the developer to get started with. You can start developing your application in the Oracle NoSQL Database Cloud Simulator, using and understanding the basic examples, before you get started with Oracle NoSQL Database Cloud Service. Extract Oracle NoSQL Database Java SDK and the Oracle NoSQL Database Cloud Simulator bundles. Create your application using the Cloud APIs. After building, debugging, and testing your application with the Oracle NoSQL Database Cloud Simulator, move your application to Oracle NoSQL Database Cloud Service.

Topics

1. Downloading the Oracle NoSQL Database Cloud Simulator 2. Oracle NoSQL Database Cloud Simulator Compared With Oracle NoSQL Database Cloud Service Downloading the Oracle NoSQL Database Cloud Simulator

Learn how to download and extract the Oracle NoSQL Database Cloud Simulator. The Oracle NoSQL Database Cloud Simulator is available for download from the Oracle Cloud website. To install the Oracle NoSQL Database Cloud Simulator, first download and extract the file.

Note:

Your local system should meet the following requirements to run the Oracle NoSQL Database Cloud Simulator: • Java JDK version 10 or higher installed in your machine. • A minimum of 5-GB available disk space where you plan to install the Oracle NoSQL Database Cloud Simulator.

Perform the following steps: 1. Download the Oracle NoSQL Database Java SDK file from the Oracle NoSQL Cloud SDK download page.

4-1 Chapter 4 Oracle NoSQL Database Cloud Simulator Compared With Oracle NoSQL Database Cloud Service

2. Gunzip and untar the .tar.gz package (or extract the files if you have downloaded a .zip package). The SDK package oracle-nosql-cloud-simulator-1.2.0 is used in this example. The actual package name and the .jar file names can change, depending on the release version that you download.

$ tar xvfz oracle-nosql-cloud-simulator-1.2.0.tar.gz

The output displays all directories and files that are part of the package. All the Oracle NoSQL Database Cloud Simulator related .jar files are placed in the cloudsim/lib directory. After extracting the package, read the oracle-nosql-cloud-simulator-1.2.0/ README.txt file for instructions on how to start and stop the simulator.

In order to use the Oracle NoSQL Database Cloud Simulator you must download one of the supported Oracle NoSQL language SDKs. The SDKs have instructions and example code to connect to either the Oracle NoSQL Database Cloud Simulator or the Oracle NoSQL Database Cloud Service. Oracle NoSQL Database Cloud Simulator Compared With Oracle NoSQL Database Cloud Service

Learn the difference between Oracle NoSQL Database Cloud Simulator and Oracle NoSQL Database Cloud Service. The differences helps determine important design considerations that you should make before using your application in a production environment. Oracle NoSQL Database Cloud Simulator is a local version of the Oracle NoSQL Database Cloud Service. The server instance that you create in Oracle NoSQL Database Cloud Simulator supports relatively limited aggregate throughput when compared to the Oracle NoSQL Database Cloud Service. Also, the performance of NoSQL operations on the Oracle NoSQL Database Cloud Simulator is based on the speed and capability of the machine on which it is deployed. By comparison, Oracle NoSQL Database Cloud Service is suitable for production use because of features such as scalability, availability, and durability. Oracle NoSQL Database Cloud Simulator has the following limitations when compared to Oracle NoSQL Database Cloud Service: • The Oracle NoSQL Database Cloud Simulator can be used for only development and testing purposes. Do not use Oracle NoSQL Database Cloud Simulator for performance measurements or in a production environment. • At least 5 GB of disk drive space must be available to run the Oracle NoSQL Database Cloud Simulator. • A single instance of the Oracle NoSQL Database Cloud Simulator should be started in a root directory (directory where the Oracle NoSQL Database Cloud Simulator data is located). Oracle NoSQL Database Cloud Simulator assumes exclusive control over the data storage directory. • The Oracle NoSQL Database Cloud Simulator does not support or require security-relevant configurations.

4-2 Chapter 4 Oracle NoSQL Database Cloud Simulator Compared With Oracle NoSQL Database Cloud Service

• No hard limit is enforced on the number of tables, size of tables, number of indexes, or maximum throughput specified for tables (except for the amount of storage on the local disk drive). • Data Definition Language (DDL) operations, such as creating or dropping a table, and creating or dropping an index, are not throttled. • Operational history is not maintained.

4-3 5 Using Plugins for Development

Get familiar with the plugins available for developing NoSQL applications in the Oracle NoSQL Database Cloud Service from external integrated development environments or IDEs.

Topics

1. About IntelliJ Plugin 2. About Eclipse Plugin About IntelliJ Plugin

Browse tables and execute queries on your Oracle NoSQL Database Cloud Service instance or simulator from IntelliJ. The Oracle NoSQL Database Cloud Service IntelliJ plugin connects to a running instance of Oracle NoSQL Database Cloud Service or simulator and allows you to: • Quickly get started with Oracle NoSQL Database Cloud Service by using the examples available with the plugin. • View the tables in your cloud account or simulator. • Retrieve columns, indexes, primary keys, and shard keys for each table. • Build and test your SQL queries on a table and obtain results in a tabular format. • View the data in each column in the JSON format.

Topics: • Setting Up IntelliJ Plug-in • Creating a NoSQL Project in IntelliJ • Connecting to Oracle NoSQL Database Cloud Simulator from IntelliJ • Connecting to Oracle NoSQL Database Cloud Service from IntelliJ • Managing Tables Using the IntelliJ Plugin Setting Up IntelliJ Plug-in

Learn how to set up the IntelliJ plug-in for Oracle NoSQL Database Cloud Service instance or simulator. Perform the following steps: 1. Download and start Oracle NoSQL Database Cloud Simulator. See Downloading the Oracle NoSQL Database Cloud Simulator. 2. Download and extract Oracle NoSQL Database Java SDK. See About the Oracle NoSQL Database Java SDK. 3. Install the IntelliJ plugin, and restart the IDE.

5-1 Chapter 5 About IntelliJ Plugin

You have two options to install the plugin: • Search the Oracle NoSQL Database Connector in the JetBrains plug-in repository, and install it, or • Download the IntelliJ plugin from Oracle Technology Network, and install the plugin from disk.

Tip:

Don't extract the downloaded plugin zip file. Select the plugin in the zip format while installing it from disk.

After you successfully set up your IntelliJ plugin, create a NoSQL project, and connect it to your Oracle NoSQL Database Cloud Service instance or simulator. Creating a NoSQL Project in IntelliJ

Learn how to create a NoSQL project in IntelliJ. Perform the following steps: 1. Open IntelliJ IDEA. Click File > New > Project. 2. Select Oracle NoSQL examples from the explorer window, and click Next. 3. Browse to the location where you extracted Oracle NoSQL Database Java SDK on your hard-disk, and click OK. For example, if you extracted the Oracle NoSQL Database Java SDK in your D:\ drive, the path looks like D:\oracle-nosql-java-sdk-5.2.11 4. Click Next. 5. Enter a value for Project Name and Project Location, and click Finish. 6. Once your NoSQL project is created, you can browse the example java files from the Project Explorer window. After you successfully create a NoSQL project in IntelliJ, connect your project to your Oracle NoSQL Database Cloud Service or simulator. Connecting to Oracle NoSQL Database Cloud Simulator from IntelliJ

Learn how to connect your NoSQL project to Oracle NoSQL Database Cloud Simulator using the IntelliJ plugin. Perform the following steps: 1. Open your NoSQL project in IntelliJ.

2. Click the icon in the Schema Explorer window to open the Settings dialog for the plugin. 3. Expand Tools > Oracle NoSQL in the Settings Explorer, and click Connections. 4. Select Cloudsim from the drop-down menu for the connection type. 5. Enter values for the following connection parameters, and click OK.

5-2 Chapter 5 About IntelliJ Plugin

Table 5-1 Connection Parameters

Parameter Description Sample Value Service URL The IP address and port on The default value is http:// which the Oracle NoSQL localhost:8080 Database Cloud Simulator is running. Tenant Identifier Unique identifier to identify the The default value is exampleId. tenant. Retain this value if you want to test the examples. SDK Path Complete path to the directory D:\oracle-nosql-java- where you extracted the Oracle sdk-5.2.11 NoSQL Database Java SDK.

6. The Intellij plugin connects your project to the Oracle NoSQL Database Cloud Simulator and displays its schema in the Schema Explorer window.

Note:

Before connecting your project to Oracle NoSQL Database Cloud Simulator, it must be started and running. Otherwise, your connection request will fail in IntelliJ.

After you successfully connect your project to your Oracle NoSQL Database Cloud Simulator, you can manage the tables and data in your schema. Connecting to Oracle NoSQL Database Cloud Service from IntelliJ

Learn how to connect your NoSQL project to Oracle NoSQL Database Cloud Service using the IntelliJ plugin Perform the following steps: 1. Open your NoSQL project in IntelliJ.

2. Click the icon in the Schema Explorer window to open the Settings dialog for the plugin. 3. Expand Tools > Oracle NoSQL in the Settings Explorer, and click Connections. 4. Select Cloud from the drop-down menu for the connection type. 5. Enter values for the following connection parameters, and click OK.

5-3 Chapter 5 About IntelliJ Plugin

Table 5-2 Connection Parameters

Parameter Description Sample Value Endpoint Regional network access https://nosql.us- point to the Oracle NoSQL ashburn-1.oci.oraclecl Database Cloud Service. oud.com (for the Ashburn Oracle NoSQL Database Cloud Service region identifier in the North America region. See Data Regions and Associated Service Endpoints for a list of service endpoints. SDK Path Complete path to the D:\oracle-nosql-java- directory where you sdk-5.2.11 extracted the Oracle NoSQL Database Java SDK. Tenant ID Tenancy©s OCID for your See Where to get the Oracle NoSQL Database Tenancy©s OCID and User©s Cloud Service. OCID in Oracle Cloud User ID User©s OCID for your Oracle Infrastructure NoSQL Database Cloud Documentation. Service. Fingerprint The fingerprint and See the following resources Passphrase(Optional) passphrase of the signing in Oracle Cloud key created while generating Infrastructure and uploading the API Documentation: Signing Key. · How to Generate an API Signing Key to generate the signing key with an optional passphrase. · See How to Get the Key©s Fingerprint to get the key©s fingerprint Private Key The private key generated for For the application user, an the user. API signing key must be generated and uploaded. See How to Generate an API Signing Key to generate the signing key with an optional passphrase. Compartment (Optional) The compartment ID for your If no value is specified, it NoSQL database schema. defaults to the Root compartment.

6. The Intellij plugin connects your project to the Oracle NoSQL Database Cloud Service and displays its schema in the Schema Explorer window. 7. If required, you can change your service endpoint or compartment from the Schema Explorer window itself. To do this, click the icon in the Schema Explorer window. A dialog window appears where you can provide the new values for Endpoint and Compartment. Enter the values that you want to modify, and click OK. You can provide values for: • Both Endpoint and Compartment, or

5-4 Chapter 5 About IntelliJ Plugin

• Endpoint alone. In this case, the Compartment defaults to the Root compartment in that region. After you successfully connect your project to your Oracle NoSQL Database Cloud Service, you can manage the tables and data in your schema. Managing Tables Using the IntelliJ Plugin

Learn how to create tables and view table data in Oracle NoSQL Database Cloud Service or Oracle NoSQL Database Cloud Simulator from IntelliJ. After connecting to the Oracle NoSQL Database Cloud Simulator or Oracle NoSQL Database Cloud Service, you can execute the examples downloaded with Oracle NoSQL Database Java SDK to create a sample table. With the help of the IntelliJ Plugin, you can view the tables and their data in the Schema Explorer window. To execute an example program: 1. Open the NoSQL project connected to your Oracle NoSQL Database Cloud Service or simulator. 2. Locate and click BasicTableExample in the Project Explorer window. By looking at the code, you can notice that this program creates a table called audienceData, puts two rows into this table, queries the inserted rows, deletes the inserted rows, and finally drops the audienceData table. 3. To pass the required arguments, click Run > Edit Configurations. Depending on the connection type, enter the following program arguments, and click OK.

Table 5-3 Program Arguments

Connection Type Program Arguments More Information Cloudsim http://localhost:8080 If you started the Oracle NoSQL Database Cloud Simulator on a different port, you must replace 8080 with that port number. Cloud us-ashburn-1 -configFile The first argument indicates the D:\OCI_PROP\config data region of your Oracle NoSQL Database Cloud Service. The second argument passes a configuration file that contains the credentials to connect to the Oracle NoSQL Database Cloud Service.

4. To execute this program, click Run > Run 'BasicExampleTable' or press Shift + 10. 5. Verify the logs in the terminal to confirm that the code executed successfully. You can see the display messages that indicate table creation, rows insertion, and so on.

Tip:

As the BasicExampleTable deletes the inserted rows and drops the audienceData table, you can't view this table in the Schema Explorer. If you want to see the table in the Schema Explorer, comment the code that deletes the inserted rows and drops the table, and rerun the program.

5-5 Chapter 5 About Eclipse Plugin

6. To view the tables and their data:

a. Locate the Schema Explorer, and click the icon to reload the schema. b. Locate the audienceData table under your tenant identifier, and expand it to view its columns, primary key, and shard key details. c. Double-click the table name to view its data. Alternatively, you can right-click the table and select Browse Table. d. A record viewer window appears in the main editor. Click Execute to run the query and display table data.

Note:

As of the current release, only SELECT queries are supported on the NoSQL tables from the Schema Explorer.

e. To view individual cell data separately, double-click the cell. About Eclipse Plugin

Build and run your Oracle NoSQL Database Cloud Service application quickly from the Eclipse IDE. To enhance your experience of building an Oracle NoSQL Database Cloud Service application, a plugin is available in Eclipse. This plugin connects to a running instance of Oracle NoSQL Database Cloud Service or Cloud Simulator and allows you to: • Quickly get started with Oracle NoSQL Database Cloud Service by using the examples available with the plugin. • Explore development/test data from tables in your Oracle NoSQL Database Cloud Service or Cloud Simulator. • Build and test your queries. • Retrieve columns, indexes, primary keys, and shard keys for each table. • Build and test your SQL queries on a table and obtain results in a tabular format. To use the Eclipse plugin: 1. Download the Eclipse plugin from Oracle Technology Network. 2. Follow the instructions given in the README file and install the plugin. 3. After installing the Eclipse plugin, you can connect to your Oracle NoSQL Database Cloud Service or Oracle NoSQL Database Cloud Simulator and execute the code to read/write the tables. For more details, you can access the help content embedded with Eclipse. To access the help content: a. Click Help Contents from the Help menu. b. Locate and expand the Oracle NoSQL Plugin Help Contents section. c. This lists all the help topics available for Oracle NoSQL Plugin. d. Refer the help topic as per your requirement.

5-6 6 Using the Console to Manage Tables

Learn how to manage Oracle NoSQL Database Cloud Service tables using the console.

Topics: • Learn how to read, update, and delete data from Oracle NoSQL Database Cloud Service tables from the console. See Managing Table Data. • Learn how to create and manage Oracle NoSQL Database Cloud Service tables and indexes from the console. See Managing Tables and Indexes. • Learn how to monitor Oracle NoSQL Database Cloud Service tables from the console. See Monitoring Tables and Indexes. Accessing the Service from the Infrastructure Console

Learn how to access the Oracle NoSQL Database Cloud Service service from the Infrastructure Console.

1. Locate the service URL from the welcome email, and then sign in to your Oracle NoSQL Database Cloud Service. 2. Open the navigation menu, click Databases, and then click NoSQL Database. 3. Select the compartment created for your tables by the service administrator.

To view help for the current page, click the help icon at the top of the page. Managing Table Data

Learn how to manage Oracle NoSQL Database Cloud Service table data for your organization.

Topics: • Inserting Data Into Tables • Viewing Table Data • Updating Table Data • Deleting Table Data • Downloading Table Data Inserting Data Into Tables

Learn how to insert data into Oracle NoSQL Database Cloud Service tables from the NoSQL console. The NoSQL console lets you insert new rows into the Oracle NoSQL Database Cloud Service tables in two modes:

6-1 Chapter 6 Managing Table Data

1. Simple Input Mode: You can use this mode to provide the values for the new rows declaratively. 2. Advanced JSON Input Mode: You can use this mode to provide the values for the new rows in JSON format. Inserting Data Into Tables: Simple Input Mode

Learn how to insert data into Oracle NoSQL Database Cloud Service tables by using the Simple Input insertion mode. To insert data into tables: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. Click Insert Row. 4. In the Insert Row window, select Simple Input for Entry Mode. 5. All the columns in the table are listed. Input the data for the columns of the table. For some column types, such as Binary, you upload the data.

Note:

Entering a value is mandatory for all non-nullable columns of the table.

6. Click Insert Row. The record is inserted into the table. To view help for the current page, click the help link at the top of the page. Inserting Data Into Tables: Advanced JSON Input Mode

Learn how to insert data into Oracle NoSQL Database Cloud Service tables by using the Advanced JSON input mode. To insert data into tables: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up.

6-2 Chapter 6 Managing Table Data

3. Click Insert Row. 4. In the Insert Record window, select Advanced JSON Input for Entry Mode. 5. Paste or upload the Record Definition in JSON format. 6. Click Insert Row. The record is inserted into the table. To view help for the current page, click the help link at the top of the page. Viewing Table Data

Learn how to view data in Oracle NoSQL Database Cloud Service tables from the NoSQL console. To view table data: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Table Rows tab under Resources. 4. By default, the query text is populated with a SQL query that will retrieve all the records from the table. You can modify this query with any valid SQL for Oracle NoSQL statement. See SQL Reference for Oracle NoSQL Database. 5. Click Run Query. The table data is displayed in the Records section. Updating Table Data

Learn how to update data in Oracle NoSQL Database Cloud Service tables from the NoSQL console. To update table data: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Table Rows tab under Resources. 4. By default, the query text is populated with a SQL query that will retrieve all the records from the table. You can modify this query with any valid SQL for Oracle NoSQL statement. See SQL Reference for Oracle NoSQL Database.

6-3 Chapter 6 Managing Table Data

5. Click the action menu corresponding to the row you wish to update, and select Update Row. 6. Modify the values in Simple Input or Advanced JSON Input Updation Mode. 7. Click Update Row. To view help for the current page, click the help link at the top of the page. Deleting Table Data

Learn how to delete data in Oracle NoSQL Database Cloud Service tables from the NoSQL console. To delete table data: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Table Rows tab under Resources. 4. By default, the query text is populated with a SQL query that will retrieve all the records from the table. You can modify this query with any valid SQL for Oracle NoSQL statement. See SQL Reference for Oracle NoSQL Database. 5. Click the action menu corresponding to the row you wish to delete, and select Delete. The Delete Row confirmation dialog opens. 6. Click Delete. The row is deleted. Downloading Table Data

Learn how to download data in Oracle NoSQL Database Cloud Service tables from the NoSQL console. To download table data: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Table Rows tab under Resources.

6-4 Chapter 6 Managing Tables and Indexes

4. By default, the query text is populated with a SQL query that will retrieve all the records from the table. You can modify this query with any valid SQL for Oracle NoSQL statement. See SQL Reference for Oracle NoSQL Database. 5. Click the action menu corresponding to the row you wish to download, and select Download JSON. The row downloads in JSON format. Managing Tables and Indexes

Learn how to create and manage Oracle NoSQL Database Cloud Service tables and indexes for your organization.

Topics: • Creating Tables • Creating Indexes • Editing Tables • Moving Tables • Altering Tables • Deleting Tables • Deleting Indexes Creating Tables

You can create new Oracle NoSQL Database Cloud Service table from the NoSQL console. The NoSQL console lets you create the Oracle NoSQL Database Cloud Service tables in two modes: 1. Simple Input Mode: You can use this mode to create the NoSQL Database Cloud Service table declaratively, that is, without writing a DDL statement. 2. Advanced DDL Input Mode: You can use this mode to create the NoSQL Database Cloud Service table using a DDL statement. Creating Table: Simple Input Mode

Learn how to create a table from the NoSQL console by using the Simple Input table creation mode. To create a table: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. Click Create Table. 3. In the Create Table dialog, select Simple input for Table Creation Mode. 4. Under Reserved Capacity, enable the toggle button to create an Always Free NoSQL table. Disabling the toggle button creates a regular NoSQL table. You can create up to three Always Free NoSQL tables in the tenancy. If you have three Always Free NoSQL tables in the tenancy, the toggle button to create an Always Free SQL table is disabled.

6-5 Chapter 6 Managing Tables and Indexes

5. If you enable the toggle button to create an Always Free NoSQL table, the Read capacity, Write capacity and Disk storage fields are assigned default values. These values cannot be changed.

6. If you want to create a regular table, then disable the toggle button. You will be able to enter the appropriate capacity values for the table. • Read Capacity (ReadUnits): Enter the number of read units. See Estimating Capacity to learn about read units. • Write Capacity (WriteUnits): Enter the number of write units. See Estimating Capacity to learn about write units. • Disk Storage (GB): Specify the disk space in gigabytes (GB) to be used by the table. See Estimating Capacity to learn about storage capacity.

6-6 Chapter 6 Managing Tables and Indexes

7. In the Name field, enter a table name that is unique within your tenancy. Table names must conform to Oracle NoSQL Database Cloud Service naming conventions. See Oracle NoSQL Database Cloud Service Limits. 8. In the Primary Key Columns section, enter primary key details: • Column Name: Enter a column name for the primary key in your table. See Oracle NoSQL Database Cloud Service Limits to learn about column naming requirements. • Type: Select the data type for your primary key column. • Precision:This is applicable for TIMESTAMP typed columns only. Timestamp values have precision in fractional seconds that range from 0 to 9. For example, a precision of 0 means that no fractional seconds are stored, 3 means that the timestamp stores milliseconds and 9 means a precision of nanoseconds. 0 is the minimum precision, and 9 is the maximum. • Set as Shard Key: Click this option to set this primary key column as shard key. Shard key is to distribute data across the Oracle NoSQL Database Cloud Service cluster for increased efficiency, and to position records that share the shard key locally for easy reference and access. Records that share the shard key are stored in the same physical location and can be accessed atomically and efficiently. • + Another Primary Key Column: Click this button to add more columns while creating a composite (multi-column) primary key. • Use the up and down arrows to change the sequence of columns while creating a composite primary key.

6-7 Chapter 6 Managing Tables and Indexes

9. In the Columns section, enter non-primary column details:

• Column Name: Enter the column name. Ensure that you conform to column naming requirements described in Oracle NoSQL Database Cloud Service Limits. • Type: Select the data type for your column. • Precision:This is applicable for TIMESTAMP typed columns only. Timestamp values have precision in fractional seconds that range from 0 to 9. For example, a precision of 0 means that no fractional seconds are stored, 3 means that the timestamp stores milliseconds and 9 means a precision of nanoseconds. 0 is the minimum precision, and 9 is the maximum. • Size: This is applicable for BINARY typed columns only. Specify the size in bytes to make the binary a fixed binary. • Default Value: (optional) Supply a default value for the column.

Note:

Default values can not be specified for binary and JSON data type columns.

• Value is Not Null: Click this option to specify that a column must always have a value. • + Another Column: Click this button to add more columns. • Click the delete icon to delete a column. 10. (Optional) To specify advanced options, click Show Advanced Options and enter advanced details: • Table Time to Live (Days): (optional) Specify expiration duration (no. of days) for the rows in the table. After the number of days, the rows expire automatically, and are no longer available. The default value is zero, indicating no expiration time.

6-8 Chapter 6 Managing Tables and Indexes

Note:

Updating Table Time to Live (TTL) will not change the TTL value of any existing data in the table. The new TTL value will only apply to those rows that are added to the table after this value is modified and to the rows for which no overriding row-specific value has been supplied.

In the Tags section, enter: • Tag Namespace: Select a tag namespace from the select list. A tag namespace is like a container for your tag keys. It is case insensitive and must be unique across the tenancy. • Tag Key: Enter the name to use to refer to the tag. A tag key is case insensitive and must be unique within a namespace. • Value: Enter the value to give your tag. • + Additional Tag: Click to add more tags.

11. Click Create table. The table is created and listed in the NoSQL console. To view help for the current page, click the help link at the top of the page. Creating Table: Advanced DDL Input Mode

Learn how to create a table from the NoSQL console by using the Advanced DDL Input table creation mode. To create a table: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. Click Create Table. 3. In the Create Table window, select Advanced DDL Input for Table Creation Mode. 4. Under Reserved Capacity, enable the toggle button to create an Always Free NoSQL table. Disabling the toggle button creates a regular NoSQL table. You can create up to

6-9 Chapter 6 Managing Tables and Indexes

three Always Free NoSQL tables in the tenancy. If you have three Always Free NoSQL tables in the tenancy, the toggle button to create an Always Free SQL table is disabled.

5. If you enable the toggle button to create an Always Free NoSQL table, the Read capacity, Write capacity and Disk storage fields are assigned default values. These values cannot be changed.

6-10 Chapter 6 Managing Tables and Indexes

7. In the DDL input section, enter the create table DDL statement for Query. See SQL Reference for Oracle NoSQL Database. 8. (Optional) To specify advanced options, click Show Advanced Options and enter advanced details: • Tag Namespace: Select a tag namespace from the select list. A tag namespace is like a container for your tag keys. It is case insensitive and must be unique across the tenancy. • Tag Key: Enter the name to use to refer to the tag. A tag key is case insensitive and must be unique within a namespace. • Value: Enter the value to give your tag. • + Additional Tag: Click to add more tags.

9. Click Create Table. The table is created and listed in the NoSQL console. To view help for the current page, click the help link at the top of the page. Creating Indexes

Learn how to create indexes in Oracle NoSQL Database Cloud Service tables from the NoSQL console. To create indexes: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Indexes tab under Resources.

6-11 Chapter 6 Managing Tables and Indexes

You will see a list of all the indexes added to the table. 4. Click Add Index. 5. In the Create Index window, enter a name for the index that is unique within the table. See Oracle NoSQL Database Cloud Service Limits to learn about the naming restrictions for indexes. 6. In the Index Columns section, enter index details: • Index Column Name: Select the column that you would like included in the index. • + Another Index Column: Click this button to include another column in the index. • Use the up and down arrow to change the sequence of the columns in the index being created. • Click the delete icon next to any column to remove it from the index being created. 7. Click Create Index. The index is created. To view help for the current page, click the help link at the top of the page. Editing Tables

You can update reserved capacity (if the table is not an Always Free NoSQL table) and Time to Live (TTL) values for your Oracle NoSQL Database Cloud Service tables from the NoSQL console. To edit tables: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. The value of Time to Live (TTL) can be updated. • To update the value of Time to Live (TTL), click the Edit link next to the Time to live (Days) field.

6-12 Chapter 6 Managing Tables and Indexes

• You can also update the value of Time to Live (TTL) by clicking the action menu corresponding to the table name you wish to change and select Edit default time to live.

• Table Time to Live (Days): (optional) Specify the default expiration time for the rows in the table. After this time, the rows expire automatically, and are no longer available. The default value is zero, indicating no expiration time.

Note:

4. If your table is not an Always Free NoSQL table, then the reserved capacity can be modified. • Under More Actions, click Edit reserved capacity.

6-13 Chapter 6 Managing Tables and Indexes

• You can also update the Reserved Capacity by clicking the action menu corresponding to the table name you wish to change and select Edit reserved capacity.

Modify the following values for the table: • Read Capacity (ReadUnits): Enter the number of read units. See Estimating Capacity to learn about read units. • Write Capacity (WriteUnits): Enter the number of write units. See Estimating Capacity to learn about write units. • Disk Storage (GB): Specify the disk space in gigabytes (GB) to be used by the table. See Estimating Capacity to learn about storage capacity. 5. (Optional) To dismiss the changes, click Cancel. To view help for the current page, click the help link at the top of the page. Altering Tables

Learn how to alter Oracle NoSQL Database Cloud Service tables by adding in simple or advanced mode, or deleting columns using the NoSQL console. The NoSQL console lets you alter the Oracle NoSQL Database Cloud Service tables in two modes: 1. Simple Input Mode: You can use this mode to alter the NoSQL Database Cloud Service table declaratively, that is, without writing a DDL statement. 2. Advanced DDL Input Mode: You can use this mode to alter the NoSQL Database Cloud Service table using a DDL statement.

6-14 Chapter 6 Managing Tables and Indexes

Adding Table Columns: Simple Input Mode

Learn how to add table columns to an Oracle NoSQL Database Cloud Service table by using the Simple Input table column update mode. To add table columns: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Columns tab under Resources. You will see a list of all the columns added to the table. 4. Click Add Columns. 5. In the Add Columns window, select Simple Input for Table Column Update Mode. 6. In the Columns section, enter non-primary column details:

6-15 Chapter 6 Managing Tables and Indexes

Note:

Default values can not be specified for binary and JSON data type columns.

• Value is Not Null: Click this option to specify that a column must always have a value. • + Another Column: Click this button to add more columns. • Click the delete icon to delete a column. 7. Click Add Columns. The new columns are added to the table. To view help for the current page, click the help link at the top of the page. Adding Table Columns: Advanced DDL Input Mode

Learn how to add table columns to an Oracle NoSQL Database Cloud Service table by using the Advanced DDL table column update mode. To add table columns: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Columns tab under Resources. You will see a list of all the columns added to the table. 4. Click Add Columns. 5. In the Add Columns window, select Advanced DDL Input for Table Column Update Mode. 6. Enter the update table DDL statement. For an example, See Alter Statement in SQL Reference for Oracle NoSQL Database . 7. Click Add Columns. The new columns are added to the table. Deleting Table Columns

Learn how to delete table columns from an Oracle NoSQL Database Cloud Service table by using the NoSQL console. To delete columns from a table: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console.

6-16 Chapter 6 Managing Tables and Indexes

2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Columns tab under Resources. You will see a list of all the columns added to the table. 4. Click the action menu corresponding to the column you wish to delete and select Delete. 5. In the Delete Column confirmation dialog, click Delete to confirm delete. The column is deleted from the table. Moving Tables

Learn how to move Oracle NoSQL Database Cloud Service table to a different compartment from the NoSQL console. To move a table: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, click Move Table. 4. Alternatively, Click the action menu corresponding to the table name and select Move table. 5. In the Move Resource to a Different Compartment window, modify the following values for the table: • Choose New Compartment: Select the new compartment from the select list. 6. Click Move table. 7. (Optional) To dismiss the changes, click the Cancel link on the top right corner. To view help for the current page, click the help link at the top of the page. Deleting Tables

Learn how to delete Oracle NoSQL Database Cloud Service tables from the NoSQL console. To delete tables: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To delete the table, do either of the following:

6-17 Chapter 6 Monitoring Tables and Indexes

• Click the table name. In the Table Details page, click the Delete button, or • Click the action menu corresponding to the table name you wish to delete and select Delete. The Delete Table confirmation dialog opens. 3. Click Delete. The table is deleted. Deleting Indexes

Learn how to delete Oracle NoSQL Database Cloud Service indexes from the NoSQL console. To delete indexes: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Indexes tab under Resources. You will see a list of all the indexes added to the table. 4. Click the action menu corresponding to the index you wish to delete, and select Delete. The Delete Index confirmation dialog opens. 5. Click Delete. The index is deleted. Monitoring Tables and Indexes

Learn how to monitor the Oracle NoSQL Database Cloud Service tables and their indexes.

Topics: • Viewing Tables • Viewing Indexes • Viewing Table Details • Viewing Table Metrics

6-18 Chapter 6 Monitoring Tables and Indexes

Viewing Tables

You can view Oracle NoSQL Database Cloud Service tables from the NoSQL console. To view tables: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. You can view all the tables in your tenancy from the NoSQL console. Viewing Indexes

You can view Oracle NoSQL Database Cloud Service the list of indexes created for a NoSQL table from the NoSQL console. To view indexes: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Indexes tab under Resources. You will see a list of all the indexes added to the table. Viewing Table Details

Learn how to view Oracle NoSQL Database Cloud Service table details from the NoSQL console. To view table details: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. From the Table Details page, you can view all table columns, indexes, rows, and metrics. Viewing Table DDL

You can view the DDL statement used to create a table for the Table Details page. To view table DDL: 1. In the Table Details page, click View Table DDL.

6-19 Chapter 6 Monitoring Tables and Indexes

The View Table DDL window displays the table DDL statement. 2. Now, you can select and copy the table DDL statement from the window. Click OK to close the window. Viewing Table Metrics

Learn how to view Oracle NoSQL Database Cloud Service table metrics from the NoSQL console. To view table metrics: 1. Access the NoSQL console from the Infrastructure Console. See Accessing the Service from the Infrastructure Console. 2. The NoSQL console lists all the tables in the tenancy. To view table details, do either of the following: • Click the table name, or • Click the action menu corresponding to the table name and select View Details. The Table Details page opens up. 3. In the Table Details page, select the Metrics tab under Resources. Table metrics such as Read Units, Write Units, Storage GB, Read Throttle Count, Write Throttle Count, and Storage Throttle Count show up. You can filter the metrics by date, change interval, and statistic value. 4. For each of the metrics displayed on this page, you can perform the following actions: • View Query in Metrics Explorer: This page lets you write and edit queries in Monitoring Query Language (MQL), using metrics from either your application or an Oracle Cloud Infrastructure service. If you're not familiar with MQL, see Monitoring Query Language (MQL) Reference. To learn more about this page, see Metrics Explorer. • Copy Chart URL: Click this option to copy the default metrics chart URL for any future reference. • Copy Query (MQL): Click this option to copy the MQL query used to create the default metrics chart. If you're not familiar with MQL, see Monitoring Query Language (MQL) Reference. • Create an Alarm on this Query: Click this option to create alarms to monitor your cloud resources. To learn about alarms, see Managing Alarms.

6-20 7 Managing Subscriptions

Learn how to manage Oracle NoSQL Database Cloud Service subscriptions and it's users.

Topics • To get an overview of setting up Oracle NoSQL Database Cloud Service, see Setting up Your Service. • To learn how to estimate the monthly cost of your Oracle NoSQL Database Cloud Service, see Estimating Your Monthly Cost. • To learn about creating compartments in your Oracle NoSQL Database Cloud Service, see Creating a Compartment. • To learn how to set up users, groups, and policies, see Setting Up Users, Groups, and Policies. Setting up Your Service

If you're setting up Oracle NoSQL Database Cloud Service for the first time, follow these tasks as a guide.

Task Reference Related Information Place an order for Oracle NoSQL Signing Up for Oracle Cloud To learn how to estimate the Database Cloud Service or sign up for Infrastructure in Oracle Cloud monthly cost of your Oracle the Oracle Free Trial. Infrastructure Documentation. NoSQL Database Cloud Service subscription, see Estimating Your Monthly Cost. To upgrade your free account or to change your payment method, see Changing Your Payment Method in Oracle Cloud Infrastructure Documentation. Activate your Oracle Cloud account Signing In to the Console in Oracle To familiarize yourself with Oracle and sign in for the first time. Cloud Infrastructure Documentation. Cloud Infrastructure Console, see Using the Console in Oracle Cloud Infrastructure Documentation. (Recommended) Create a Creating a Compartment If you©re not familiar with compartment for your service. compartments, see Understanding Compartments in Oracle Cloud Infrastructure Documentation. Manage security for your service. Typical Process to Manage Security for To familiarize yourself with Oracle NoSQL Database Cloud NoSQL Database Cloud Service Service security model, see About Oracle NoSQL Database Cloud Service Security Model.

7-1 Chapter 7 Estimating Your Monthly Cost

Estimating Your Monthly Cost

Learn how to estimate the monthly cost of your Oracle Cloud subscription. When you are ready to order your Oracle Cloud service, Oracle provides you with a cost estimator to figure out your monthly usage and costs before you commit to a subscription model or an amount. The Cost Estimator automatically calculates your monthly cost based on your input of read units, write units, and storage. But for you to understand how to calculate the read and write units for your application, follow these steps:

1. Step 1: Navigate to the Estimating Capacity topic. Estimate your application workload by using the example and formulas described in this topic. Step 2: Download and use the Capacity Estimator from Oracle Technology Network to estimate write units, read units, and the storage capacity for your application based on the application workload and database operations criteria. 2. Step 2: Access the Cost Estimator on the Oracle Cloud website. Select the Data Management check box. Scroll through to locate Oracle NoSQL Database Cloud, and click Add to add an entry for Oracle NoSQL Database Cloud under the Configuration Options. Expand NoSQL Database to find the different Utilization and configuration options. Input values for the Utilization and Configuration parameters to estimate the cost for Oracle NoSQL Database Cloud Service usage from your Oracle Cloud Pay-As-You-Go and Monthly Flex subscriptions. Creating a Compartment

When you sign up for Oracle Cloud Infrastructure, Oracle creates your tenancy with a root compartment that holds all your cloud resources. You then create additional compartments within the tenancy (root compartment) and corresponding policies to control access to the resources in each compartment. Before you create an Oracle NoSQL Database Cloud Service table, Oracle recommends that you set up the compartment where you want the table to belong. You create compartments in Oracle Cloud Infrastructure Identity and Access Management (IAM). See Setting Up Your Tenancy and Managing Compartments in Oracle Cloud Infrastructure Documentation.

7-2 8 Managing Table Access and Security

As administrator, you manage access to your Oracle NoSQL Database Cloud Service tables using security features in Oracle Cloud Infrastructure Identity and Access Management (IAM).

Topics • To get an overview of Oracle NoSQL Database Cloud Service security model, see About Oracle NoSQL Database Cloud Service Security Model • To learn about the tasks involved in implementing security in Oracle NoSQL Database Cloud Service, see Typical Process to Manage Security for Oracle NoSQL Database Cloud Service • To learn about the writing policies that control access to the Oracle NoSQL Database Cloud Service, see Policy Reference • To view typical policy statements that you might use to authorize access to Oracle NoSQL Database Cloud Service tables, see Typical Policy Statements to Manage Tables • To learn how to manage user permissions, see Giving Another User Permission to Manage NoSQL Tables About Oracle NoSQL Database Cloud Service Security Model

Learn about the security model for Oracle NoSQL Database Cloud Service.

Policies Oracle NoSQL Database Cloud Service uses the Oracle Cloud Infrastructure Identity and Access Management security model that is built on the policies. A policy is a document that specifies who can access which Oracle Cloud Infrastructure resources, including NoSQL tables that your company has, and how they can access these resources. A policy allows a group to work in certain ways with specific types of resources such as NoSQL Tables in a particular compartment. To govern the control of your tables, your company will have at least one policy. Each policy consists of one or more policy statements that follow this basic syntax:

Allow group to in compartment

To learn how policies work, see Overview of Policies in Oracle Cloud Infrastructure Documentation.

Groups In Oracle Cloud Infrastructure Identity and Access Management, you organize Users within groups that usually share the same type of access to a particular set of NoSQL tables or compartments.

8-1 Chapter 8 About Oracle NoSQL Database Cloud Service Security Model

You can grant access to the NoSQL Tables at the group and compartment level, by writing a policy that gives a group a specific type of access within a particular compartment, or to the tenancy itself. If you give a group access to the tenancy, the group automatically gets the same type of access to all the compartments inside the tenancy. For example, after you create a table in the compartment ProjectA, you must write a policy to grant access to the group(s) you want them to manage or use the tables. Otherwise, the tables are not even visible to the groups that don't have access. For example, to allow the Developer group to manage all the NoSQL resources, you can create the following policy: allow group Developers to manage nosql-family in compartment ProjectA

Verbs A verb specifies the type of access being granted by the policy. For example, inspect nosql-tables lets you list the NoSQL tables. Inspect, read, use, and manage are the verbs supported by Oracle NoSQL Database Cloud Service. See Verbs in Oracle Cloud Infrastructure Documentation.

Resource-types Resources are the cloud objects that your company's employees create and use when interacting with the Oracle Cloud Infrastructure (OCI). Oracle defines resource-types you can use in policies. nosql-tables, nosql-rows, and nosql-indexes are three individual resource-types supported by NoSQL Database Cloud Service. By specifying a resource-type in a policy, you give access permissions against that resource type alone. For example, to grant read permissions on the rows of all NoSQL tables in the tenancy, to the viewers group, you can create a policy as: allow group viewers to read nosql-rows in tenancy

To simplify writing policies, NoSQL Database Cloud Service also provides an aggregate resource-type called nosql-family. nosql-family includes nosql-tables, nosql-indexes, and nosql-rows that are often managed together. For example, to grant full access to NoSQL Tables in the tenancy, to the viewers group, you can write a policy as: allow group viewers to manage nosql-family in tenancy

Compartments A compartment is the fundamental component of Oracle Cloud Infrastructure. You can organize the Oracle NoSQL Database Cloud Service resources within compartments. Compartments are used to separate tables for measuring usage and billing, defining access, and isolating the resources between different projects or business units.

Note:

Tenancy is the root compartment that contains all of your organization's Oracle Cloud Infrastructure resources.

8-2 Chapter 8 Typical Process to Manage Security for Oracle NoSQL Database Cloud Service

All the Oracle Cloud Infrastructure Identity and Access Management resources, users, groups, compartments and policies are global and available across all regions, but the master set of definitions reside in a single region, the home region. All the changes to your IAM resources must be made in your home region. To learn more about the IAM components, see Overview of Oracle Cloud Infrastructure Identity and Access Management in Oracle Cloud Infrastructure Documentation. Typical Process to Manage Security for Oracle NoSQL Database Cloud Service

Learn about the typical sequence of tasks that a Oracle NoSQL Database Cloud Service administrator performs to implement a security model in an organization. As a Oracle NoSQL Database Cloud Service administrator, you must execute the following tasks to implement a security model in your organization: • Define users, groups, and one or more compartments to hold the NoSQL tables for your organization. See Setting Up Users, Groups, and Policies • Create one or more policies, each written in the policy language. See Policy Reference • Place users into the appropriate groups depending on the compartments and resources they need to access. See Setting Up Users, Groups, and Policies and Typical Policy Statements to Manage Tables • Provide a one-time password to the users to access the Console and work with the compartments. After you complete the above steps, the users can access the Console, change their one- time password, and work with specific cloud resources, as stated in the policies. Setting Up Users, Groups, and Policies

Oracle NoSQL Database Cloud Service uses Oracle Cloud Infrastructure Identity and Access Management to provide secure access to Oracle Cloud. Oracle Cloud Infrastructure Identity and Access Management enables you to create user accounts and give users permission to inspect, read, use, or manage tables.

1. Sign-in to your Cloud Account as Cloud Account Administrator. 2. In Oracle Cloud Infrastructure Console, add one or more users. a. Open the navigation menu and click Identity & Security. Under Identity, click Users. b. Click Create User. c. Enter details about the user, and click Create. 3. In Oracle Cloud Infrastructure Console, create an OCI group. a. Open the navigation menu and click Identity & Security. Under Identity, click Groups. b. Click Create Group. c. Enter details about the group. For example, if you're creating a policy that gives users permissions to fully manage Oracle NoSQL Database Cloud Service tables you might name the group

8-3 Chapter 8 Policy Reference

nosql_service_admin (or similar) and include a short description such as "Users with permissions to set up and manage Oracle NoSQL Database Cloud Service tables on Oracle Cloud Infrastructure" (or similar). 4. Create a policy that gives users belonging to an OCI group, specific access permissions to Oracle NoSQL Database Cloud Service tables or compartments. a. Open the navigation menu and click Identity & Security. Under Identity, click Policies. b. Select a compartment, and click Create Policy. For details and examples, see Policy Reference and Typical Policy Statements to Manage Tables. If you're unfamiliar about how policies work, see How Policies Work. 5. To manage and use NoSQL tables via Oracle NoSQL Database Cloud Service SDKs, the user must set up the API keys. See Acquiring Credentials.

Note:

Federated users can also manage and use Oracle NoSQL Database Cloud Service tables. This requires the service administrator to set up the federation in Oracle Cloud Infrastructure Identity and Access Management. See Federating with Identity Providers.

Users belonging to any groups mentioned in the policy statement get their new permission when they next sign in to the Console. Policy Reference

This topic covers details for writing policies to control access to the Oracle NoSQL Database Cloud Service. A policy is a document that specifies who can access which Oracle Cloud Infrastructure resources that your company has, and how. See Overview of Policies to learn basics of policies. The overall syntax of a policy statement is as follows:

Allow to in where

For detailed explanation of this syntax, see Policy Syntax in Oracle Cloud Infrastructure Documentation. Resource-Types

Learn about the resource types supported by Oracle NoSQL Database Cloud Service. Resources are the cloud objects that your company's employees create and use when interacting with the Oracle Cloud Infrastructure (OCI).

8-4 Chapter 8 Policy Reference

Individual Resource-Types An Individual Resource-Type represents a specific type of resource. As a user, you can create NoSQL tables, build indexes and populate rows in those tables in Oracle NoSQL Database Cloud Service. Accordingly, three individual resource-types are identified for Oracle NoSQL Database Cloud Service, as: • nosql-tables • nosql-rows • nosql-indexes

Aggregate Resource-Type Multiple individual resource-types that are often managed together are collectively identified as Aggregate Resource-Types. There is only one aggregate resource-type in Oracle NoSQL Database Cloud Service, as: • nosql-family

Note:

A policy that uses nosql-family is equivalent to writing one with a separate statement for each of the individual resource-types.

See the table in Details for Verb + Resource-Type Combinations for a detailed breakout of the API operations covered by each verb, for each individual resource-type included in nosql- family. Supported Variables

Learn about the variables supported by Oracle NoSQL Database Cloud Service. Oracle NoSQL Database Cloud Service supports all the general variables. See General Variables for All Requests. All three NoSQL resource types can use the following variables, except for ListTables and CreateTable.

Table 8-1 Supported Variables

Variable Variable Type Comments target.nosql-table.id OCID Use this variable to control access to specific NoSQL table by OCID. target.nosql-table.name String Use this variable to control access to specific NoSQL table by name. Details for Verb + Resource-Type Combinations

Learn about the permissions and API operations covered by each verb.

8-5 Chapter 8 Policy Reference

The level of access is cumulative as you go from inspect > read > use > manage. A plus sign (+) in a table cell indicates incremental access compared to the cell directly above it, whereas no extra indicates no incremental access. For example, the read verb for the nosql-tables resource-type includes the same permissions and API operations as the inspect verb, plus the NOSQL_TABLE_READ permission and the GetTable API operation. In the case of the nosql-tables resource- type, the use verb covers UpdateTable API operations compared to read. Lastly, manage covers more permissions and operations compared to use. nosql-tables

Table 8-2 nosql-tables

Verb Permissions REST APIs Fully NoSQL Cloud Driver Covered Request Covered INSPECT NOSQL_TABLE_INSP ListTables ListTableRequest ECT READ INSPECT + GetTable GetTableRequest NOSQL_TABLE_REA ListWorkRequests None D GetWorkRequest ListWorkRequestError s ListWorkRequestLogs ListTableUsage TableUsageRequest USE READ + UpdateTable TableRequest NOSQL_TABLE_ALTE DeleteWorkRequest · change R TableLimits · ALTER TABLE MANAGE USE + CreateTable TableRequest NOSQL_TABLE_CRE (CREATE TABLE) ATE NOSQL_TABLE_DRO DeleteTable TableRequest (DROP P TABLE) NOSQL_TABLE_MOV ChangeTableCompart Not supported E ment nosql-rows

Table 8-3 nosql-rows

Verb Permissions REST APIs Fully NoSQL Cloud Driver Covered Request Covered INSPECT None None None READ NOSQL_ROWS_REA GetRow · GetRequest D Query (SELECT) · PrepareRequest PrepareStatement · QueryRequest (SELECT) SummarizeStatement

8-6 Chapter 8 Policy Reference

Table 8-3 (Cont.) nosql-rows

Verb Permissions REST APIs Fully NoSQL Cloud Driver Covered Request Covered USE READ + UpdateRow · PutRequest NOSQL_ROWS_INSE Query (INSERT/ · WriteMultipleReq RT UPSERT, UPDATE) uest(Put) · QueryRequest(IN SERT/UPSERT, UPDATE) MANAGE USE + DeleteRow · DeleteRequest NOSQL_ROWS_DEL Query (DELETE) · MultiDeleteReque ETE st · WriteMultipleReq uest(Delete) · QueryRequest(D ELETE)

nosql-indexes

Table 8-4 nosql-indexes

Verb Permissions REST APIs Fully NoSQL Cloud Driver Covered Request Covered INSPECT None None None READ NOSQL_INDEX_READ ListIndexes GetIndexesRequest + GetIndex indexName GetIndexesRequest USE READ + NONE ListIndexes GetIndexesRequest + GetIndex indexName GetIndexesRequest MANAGE READ + CreateIndex TableRequest(CREATE NOSQL_INDEX_CREAT INDEX) E NOSQL_INDEX_DROP DeleteIndex TableRequest(DROP INDEX) Permission Required for Each NoSQL Cloud Driver Request

Learn about the required permissions for each NoSQL Cloud Driver Request. The table below lists the API operations in a logical order, grouped by resource type. For information about permissions, see Permissions in Oracle Cloud Infrastructure Documentation.

Table 8-5 Permissions

Request Permissions Operation Id (request.operation) DeleteRequest NOSQL_ROWS_DELETE DeleteRow

8-7 Chapter 8 Policy Reference

Table 8-5 (Cont.) Permissions

Request Permissions Operation Id (request.operation) GetIndexesRequest NOSQL_INDEX_READ GetIndex GetRequest NOSQL_ROWS_READ GetRow GetTableRequest NOSQL_TABLE_READ GetTable ListTablesRequest NOSQL_TABLE_INSPECT ListTables MultiDeleteRequest NOSQL_ROWS_DELETE DeleteRow PrepareRequest NOSQL_ROWS_READ GetRow PutRequest NOSQL_ROWS_INSERT UpdateRow QueryRequest (SELECT) NOSQL_ROWS_READ GetRow QueryRequest (INSERT, NOSQL_ROWS_INSERT UpdateRow UPSERT, UPDATE) QueryRequest (DELETE) NOSQL_ROWS_DELETE DeleteRow TableRequest (CREATE TABLE) NOSQL_TABLE_CREATE CreateTable TableRequest (ALTER TABLE) NOSQL_TABLE_ALTER UpdateTable TableRequest (DROP TABLE) NOSQL_TABLE_DROP DeleteTable TableUsageRequest NOSQL_TABLE_READ GetTable WriteMultipleRequest has PutRequest: UpdateRow NOSQL_ROWS_INSERT DeleteTable has DeleteRequest: NOSQL_ROWS_DELETE Permission Required for Each REST API Operation

Learn about the required permissions for each REST API operation request. The table below lists the REST API operations in a logical order, grouped by resource type. For information about permissions, see Permissions in Oracle Cloud Infrastructure Documentation.

Table 8-6 Permissions

Request Permissions ListTables NOSQL_TABLE_INSPECT CreateTable NOSQL_TABLE_CREATE GetTable NOSQL_TABLE_READ UpdateTable NOSQL_TABLE_ALTER DeleteTable NOSQL_TABLE_DROP ListIndexes NOSQL_INDEX_READ CreateIndex NOSQL_INDEX_CREATE GetIndex NOSQL_INDEX_READ DeleteIndex NOSQL_INDEX_DROP GetRow NOSQL_ROWS_READ UpdateRow NOSQL_ROWS_INSERT DeleteRow NOSQL_ROWS_DELETE

8-8 Chapter 8 Typical Policy Statements to Manage Tables

Table 8-6 (Cont.) Permissions

Request Permissions ListTableUsage NOSQL_TABLE_READ ChangeTableCompartment NOSQL_TABLE_ALTER Query (SELECT) NOSQL_ROWS_READ Query (INSERT, UPSERT, UPDATE) NOSQL_ROWS_INSERT Query (DELETE) NOSQL_ROWS_DELETE PrepareStatement NOSQL_TABLE_READ SummarizeStatement NOSQL_TABLE_READ ListWorkRequests NOSQL_TABLE_READ GetWorkRequest NOSQL_TABLE_READ DeleteWorkRequest NOSQL_TABLE_ALTER ListWorkRequestErrors NOSQL_TABLE_READ ListWorkRequestLogs NOSQL_TABLE_READ

When you write a policy with request.operation, use the name of API operations. For Query operations, use the mapping operation of statement in the query. For example:

SELECT => GetRow INSERT, UPSERT or UPDATE => UpdateRow DELETE=> DeleteRow

Typical Policy Statements to Manage Tables

Here are typical policy statements that you might use to authorize access to Oracle NoSQL Database Cloud Service tables. When you create a policy for your tenancy, you grant users access to all compartments by way of policy inheritance. Alternatively, you can restrict access to individual Oracle NoSQL Database Cloud Service tables or compartments. Example 8-1 To allow group Admins to fully manage any Oracle NoSQL Database Cloud Service table

allow group Administrators to manage nosql-tables in tenancy allow group Administrators to manage nosql-rows in tenancy allow group Administrators to manage nosql-indexes in tenancy

Example 8-2 To allow group Admins to do any operations against NoSQL Tables in compartment Dev, use the family resource type.

allow group Admins to manage nosql-family in compartment Dev

Example 8-3 To allow group Analytics to do read-only operations against NoSQL Tables in compartment Dev

allow group Analytics to read nosql-rows in compartment Dev

8-9 Chapter 8 Accessing NoSQL Tables Across Tenancies

Example 8-4 To only allow Joe in Developer to create, get and drop indexes of NoSQL tables in compartment Dev

allow group Developer to manage nosql-indexes in compartment Dev where request.user.id = ©©

Example 8-5 To allow group Admins to create, drop and move NoSQL Tables only but not alter in compartment Dev.

allow group Admins to manage nosql-tables in compartment Dev where any {request.permission = ©NOSQL_TABLE_CREATE©, request.permission = ©NOSQL_TABLE_DROP©, request.permission = ©NOSQL_TABLE_MOVE©}

Example 8-6 To allow group Developer to read, update and delete rows of table "customer" in compartment Dev but not others.

allow group Developer to manage nosql-rows in compartment Dev where target.nosql-table.name = ©customer©

Accessing NoSQL Tables Across Tenancies

This topic describes how to write policies that let your tenancy access NoSQL Tables in other tenancies. If you're new to policies, see Managing Table Access and Security and Getting Started with Policies.

Cross-Tenancy Policies Your organization might want to share resources with another organization that has its own tenancy. It could be another business unit in your company, a customer of your company, a company that provides services to your company, and so on. In cases like these, you need cross-tenancy policies in addition to the required user and service policies described previously. To access and share resources, the administrators of both tenancies need to create special policy statements that explicitly state the resources that can be accessed and shared. These special statements use the words Define, Endorse ,and Admit. Endorse, Admit, and Define Statements Here's an overview of the special verbs used in cross-tenancy statements: Endorse: States the general set of abilities that a group in your own tenancy can perform in other tenancies. The Endorse statement always belongs in the tenancy with the group of users crossing the boundaries into the other tenancy to work with that tenancy's resources. In the examples, you refer to this tenancy as the source. Admit: States the kind of ability in your own tenancy that you want to grant a group from the other tenancy. The Admit statement belongs in the tenancy who is granting "admittance" to the tenancy. The Admit statement identifies the group of users that requires resource access from the source tenancy and identified with a corresponding Endorse statement. In the examples, you refer to this tenancy as the destination.

8-10 Chapter 8 Accessing NoSQL Tables Across Tenancies

Define: Assigns an alias to a tenancy OCID for Endorse and Admit policy statements. A Define statement is also required in the destination tenancy to assign an alias to the source IAM group OCID for Admit statements. Define statements must be included in the same policy entity as the endorse or the admit statement. The Endorse and Admit statements work together, but they reside in separate policies, one in each tenancy. Without a corresponding statement that specifies access, a particular Endorse or Admit statement grants no access. You need an agreement from both the tenancies.

Note:

In addition to policy statements, you must also be subscribed to a region to share resources across regions.

Source tenancy policy statements The source administrator creates policy statements that endorse a source IAM group allowed to manage resources in the destination tenancy.

Note:

The cross-tenancy policies can also be written with other policy subjects. For more details on policy subjects, see Policy Syntax in Oracle Cloud Infrastructure Documentation.

Here is an example of a broad policy statement that endorses the IAM group NoSQLAdmins group to do anything with all NoSQL Tables in any tenancy:

Endorse group NoSQLAdmins to manage nosql-family in any-tenancy

To write a policy that reduces the scope of tenancy access, the destination administrator must provide the destination tenancy OCID. Here is an example of policy statements that endorse the IAM group NoSQLAdmins group to manage NoSQL Tables in the DestinationTenancy only:

Define tenancy DestinationTenancy as ocid1.tenancy.oc1.. Endorse group NoSQLAdmins to manage nosql-family in tenancy DestinationTenancy

Destination tenancy policy statements The destination administrator creates policy statements that: • Defines the source tenancy and IAM group that is allowed to access resources in your tenancy. The source administrator must provide this information. • Admits those defined sources to access NoSQL Tables that you want to allow access to in your tenancy.

8-11 Chapter 8 Giving Another User Permission to Manage NoSQL Tables

Here is an example of policy statements that endorse the IAM group NoSQLAdmins in the source tenancy to do anything with all NoSQL Tables in your tenancy:

Define tenancy SourceTenancy as ocid1.tenancy.oc1.. Define group NoSQLAdmins as ocid1.group.oc1.. Admit group NoSQLAdmins of tenancy SourceTenancy to manage nosql-family in tenancy

Here is an example of policy statements that endorse the IAM group NoSQLAdmins in the source tenancy to manage NoSQL Tables only the Develop compartment:

Define tenancy SourceTenancy as ocid1.tenancy.oc1.. Define group NoSQLAdmins as ocid1.group.oc1.. Admit group NoSQLAdmins of tenancy SourceTenancy to manage nosql-family in compartment Develop

Giving Another User Permission to Manage NoSQL Tables

When you activate your order for Oracle NoSQL Database Cloud Service, you (the first user) are in the Administrators group by default. Being in the Administrators group gives you full administration privileges in Oracle Cloud Infrastructure so you can manage Oracle NoSQL Database Cloud Service tables and much more. There's no need to delegate this responsibility but, if you want to, you can give someone else privileges to create and manage Oracle NoSQL Database Cloud Service tables through the manage nosql-tables permission.

In Oracle Cloud Infrastructure you use IAM security policies to grant permissions. First, you must add the user to a group, and then you create a security policy that grants the group the manage nosql-tables permission on a specific compartment or the tenancy (any compartment in the tenancy). For example, you might create a policy statement that looks like one of these:

allow group MyAdminGroup to manage nosql-tables in tenancy

allow group MyAdminGroup to manage nosql-tables in compartment MyOracleNoSQL

To find out how to create security policy statements specifically for Oracle NoSQL Database Cloud Service, see Setting Up Users, Groups, and Policies.

8-12 9 Using Oracle NoSQL Database Migrator

Learn about Oracle NoSQL Database Migrator and how to use it for data migration. Oracle NoSQL Database Migrator is a tool that supports the movement of Oracle NoSQL tables from one data source to another. This tool can operate on tables in Oracle NoSQL Database Cloud Service, Oracle NoSQL Database on-premise, and handle JSON and MongoDB-formatted JSON input files. The Migrator tool supports several different data formats and physical media types. Supported data formats are JSON and MongoDB- formatted JSON. Supported physical media types are files, OCI Object Storage, Oracle NoSQL Database on-premise, and Oracle NoSQL Database Cloud Service.

Topics: • Overview • Using Oracle NoSQL Database Migrator • Supported Sources and Sinks • Use Case Demonstrations • Troubleshooting the Oracle NoSQL Database Migrator • Oracle NoSQL Database Migrator Vs. Import/Export Utility • Transitioning from Import/Export to NoSQL Database Migrator Overview

Oracle NoSQL Database Migrator lets you move Oracle NoSQL tables from one data source to another, such as Oracle NoSQL Database on-premise or cloud or even a simple JSON file. There can be many situations that require you to migrate NoSQL tables from or to an Oracle NoSQL Database. For instance, a team of developers enhancing a NoSQL Database application may want to test their updated code in the local Oracle NoSQL Database Cloud Service (NDCS) instance using cloudsim. To verify all the possible test cases, they must set up the test data similar to the actual data. To do this, they must copy the NoSQL tables from the production environment to their local NDCS instance, the cloudsim environment. In another situation, NoSQL developers may need to move their application data from on- premise to the cloud and vice-versa, either for development or testing. In all such cases and many more, you can use Oracle NoSQL Database Migrator to move your NoSQL tables from one data source to another, such as Oracle NoSQL Database on- premise or cloud or even a simple JSON file. You can also copy NoSQL tables from a MongoDB-formatted JSON input file into your NoSQL Database on-premise or cloud. Oracle NoSQL Database Migrator is created to replace and enhance the existing on-premise- only import/export utility. To know how the NoSQL Database Migrator is different from the existing import/export utility, see Oracle NoSQL Database Migrator Vs. Import/Export Utility . As depicted in the following figure, the NoSQL Database Migrator utility acts as a connector or pipe between the data source and the target (referred to as the sink). In essence, this utility exports data from the selected source and imports that data into the sink. This tool is

9-1 Chapter 9 Overview

table-oriented, that is, you can move the data only at the table level. A single migration task operates on a single table and supports migration of table data from source to sink in various data formats. Oracle NoSQL Database Migrator is designed such that it can support additional sources and sinks in the future. For a list of sources and sinks supported by Oracle NoSQL Database Migrator as of the current release, see Supported Sources and Sinks.

Transformations

NoSQL NoSQL Table Data Table Data Migration Pipe Source Sink

Terminology used with Oracle NoSQL Database Migrator

Learn about the different terms used in the above diagram, in detail. • Source: An entity from where the NoSQL tables are exported for migration. Some examples of sources are Oracle NoSQL Database on-premise or cloud, JSON file, and MongoDB-formatted JSON file. • Sink: An entity that imports the NoSQL tables from NoSQL Database Migrator. Some examples for sinks are Oracle NoSQL Database on-premise or cloud and JSON file. The NoSQL Database Migrator tool supports different types of sources and sinks (that is physical media or repositories of data) and data formats (that is how the data is represented in the source or sink). Supported data formats are JSON and MongoDB- formatted JSON. Supported source and sink types are files, OCI Object Storage, Oracle NoSQL Database on-premise, and Oracle NoSQL Database Cloud Service. • Migration Pipe: The data from a source will be transferred to the sink by NoSQL Database Migrator. This can be visualized as a Migration Pipe. • Transformations: You can add rules to modify the NoSQL table data in the migration pipe. These rules are called Transformations. Oracle NoSQL Database Migrator allows data transformations at the top-level fields or columns only. It does not let you transform the data in the nested fields. Some examples of permitted transformations are: – Drop or ignore one or more columns, – Rename one or more columns, or – Aggregate several columns into a single field, typically a JSON field. • Configuration File : A configuration file is a JSON file where you define all the parameters required for the migration activity. Later, you pass this JSON file as a single parameter to the runMigrator command from the CLI. A typical configuration file format looks like as shown below.

{ "source": { "type" : , //source-configuration for type. See Source Configuration Templates .

9-2 Chapter 9 Overview

}, "sink": { "type" : , //sink-configuration for type. See Sink Configuration Templates . }, "transforms" : { //transforms configuration. See Transformation Configuration Templates . }, "migratorVersion" : "", "abortOnError" : }

Group Parameters Mandatory (Y/N) Purpose Supported Values source type Y Represents the To know the type source from which value for each to migrate the source, see data. The source Supported provides data and Sources and metadata (if any) Sinks. for migration. source source- Y Defines the See Source configuration for configuration for Configuration type the source. These Templates for the configuration complete list of parameters are configuration specific to the parameters for type of source each source type. selected above. sink type Y Represents the To know the type sink to which to value for each migrate the data. source, see The sink is the Supported target or Sources and destination for the Sinks. migration. sink sink-configuration Y Defines the See Sink for type configuration for Configuration the sink. These Templates for the configuration complete list of parameters are configuration specific to the parameters for type of sink each sink type. selected above. transforms transforms N Defines the See configuration transformations to Transformation be applied to the Configuration data in the Templates for the migration pipe. complete list of transformations supported by the NoSQL Data Migrator. - migratorVersio N Version of the - n NoSQL Data Migrator

9-3 Chapter 9 Using Oracle NoSQL Database Migrator

Group Parameters Mandatory (Y/N) Purpose Supported Values - abortOnError N Specifies whether true, false to stop the migration activity in case of any error or not. The default value is true indicating that the migration stops whenever it encounters a migration error. If you set this value to false, the migration continues even in case of failed records or other migration errors. The failed records and migration errors will be logged as WARNINGs on the CLI terminal.

Note:

As JSON is case-sensitive, all the parameters defined in the configuration file are case-sensitive unless specified otherwise.

Using Oracle NoSQL Database Migrator

Learn about the various steps involved in using the Oracle NoSQL Database Migrator utility for migrating your NoSQL data. The high level flow of tasks involved in using NoSQL Database Migrator is depicted in the below figure.

9-4 Chapter 9 Using Oracle NoSQL Database Migrator

BEGIN

Download the NoSQL Migrator Utility

Identify Source & Sink for Migration

Generate the Create a Conﬁguration JSON File Conﬁguration JSON using runMigrator File Manually

OR You can reuse the Conﬁg JSON File multiple times.

You can reuse the Conﬁg JSON File multiple times.

Proceed to Migration Save the Configuration You can reuse the Run runMigrator by with the Generated JSON File for a Future Config JSON File passing the Configuration Configuration JSON File Migration multiple times. JSON File as a Parameter

END

Download the NoSQL Data Migrator Utility The Oracle NoSQL Database Migrator utility is available for download from the Oracle NoSQL Downloads page. Once you download and unzip it on your machine, you can access the runMigrator command from the command line interface.

Identify the Source and Sink Before using the migrator, you must identify the data source and sink. For instance, if you want to migrate a NoSQL table from Oracle NoSQL Database on-premise to a JSON formatted file, your source will be Oracle NoSQL Database and sink will be JSON file. Ensure that the identified source and sink are supported by the Oracle NoSQL Database Migrator by referring to Supported Sources and Sinks. This is also an appropriate phase to decide the schema for your NoSQL table in the target or sink, and create them. • Identify Sink Table Schema: If the sink is Oracle NoSQL Database on-premise or cloud, you must identify the schema for the sink table and ensure that the source data matches

9-5 Chapter 9 Using Oracle NoSQL Database Migrator

with the target schema. If required, use transformations to map the source data to the sink table. – Default Schema: NoSQL Database Migrator provides an option to create a table with the default schema. The default schema is defined by the migrator itself. If the source is a MongoDB-formatted JSON file, the default schema for the table will be as follows:

CREATE TABLE IF NOT EXISTS (ID STRING, DOCUMENT JSON,PRIMARY KEY(SHARD(ID));

Where: — tablename = value provided for the table attribute in the configuration. — ID = _id value from each document of the mongoDB exported JSON source file. — DOCUMENT = For each document in the mongoDB exported file, the contents excluding the _id field are aggregated into the DOCUMENT column.

Note:

If the _id value is not provided as a string in the MongoDB-formatted JSON file, NoSQL Database Migrator converts it into a string before inserting it into the default schema.

For all the other sources, the default schema will be as follows:

CREATE TABLE IF NOT EXISTS (ID LONG GENERATED ALWAYS AS IDENTITY, DOCUMENT JSON, PRIMARY KEY(ID))

Where: — tablename = value provided for the table attribute in the configuration. — ID = An auto-generated LONG value. — DOCUMENT = The JSON record provided by the source is aggregated into the DOCUMENT column.

Note:

If the _id value is not provided as a string in the MongoDB-formatted JSON file, NoSQL Database Migrator converts it into a string before inserting it into the default schema.

• Providing Table Schema: NoSQL Database Migrator allows the source to provide schema definitions for the table data using schemaInfo attribute. The schemaInfo attribute is available in all the data sources that do not have an implicit schema already defined. Sink data stores can choose any one of the following options.

9-6 Chapter 9 Using Oracle NoSQL Database Migrator

– Use the default schema defined by the NoSQL Database Migrator. – Use the source-provided schema. – Override the source-provided schema by defining its own schema. For example, if you want to transform the data from the source schema to another schema, you need to override the source-provided schema and use the transformation capability of the NoSQL Database Migrator tool.

The table schema file, for example, mytable_schema.ddl can include table DDL statements. The NoSQL Database Migrator tool executes this table schema file before starting the migration. The migrator tool supports no more than one DDL statement per line in the schema file. For example,

CREATE TABLE IF NOT EXISTS(id INTEGER, name STRING, age INTEGER, PRIMARY KEY(SHARD(ID)))

• Create Sink Table: Once you identify the sink table schema, create the sink table either through the Admin CLI or using the schemaInfo attribute of the sink configuration file. See Sink Configuration Templates .

Run the runMigrator command

The runMigrator executable file is available in the extracted NoSQL Database Migrator files. You must install Java 8 or higher version and bash on your system to successfully run the runMigrator command.

You can run the runMigrator command in two ways: 1. By creating the JSON configuration file using the runtime options of the runMigrator command as shown below.

[~]$ ./runMigrator configuration file is not provided. Do you want to generate configuration? (y/n)

[n]: y ......

9-7 Chapter 9 Using Oracle NoSQL Database Migrator

• When you invoke the runMigrator utility, it provides a series of runtime options and creates the configuration JSON file based on your choices for each option. • After the utility creates the configuration JSON file, you have a choice to either proceed with the migration activity in the same run or save the configuration file for a future migration. • Irrespective of your decision to proceed or defer the migration activity with the generated configuration JSON file, the file will be available for edits or customization to meet your future requirements. You can use the customized configuration JSON file for migration later. 2. By passing a manually created JSON configuration file as a runtime parameter using the -c or --config option. You must create the configuration JSON file manually before running the runMigrator command with the -c or --config option. For any help with the source and sink configuration parameters, see Sources and Sinks.

[~]$ ./runMigrator -c

Logging Migrator Progress NoSQL Database Migrator tool provides options, which enables trace, debugging, and progress messages to be printed to standard output or to a file. This option can be useful in tracking the progress of migration operation, particularly for very large tables or data sets. • Log Levels To control the logging behavior through the NoSQL Database Migrator tool, pass the --log-level or -l run time parameter to the runMigrator command. You can specify the amount of log information to write by passing the appropriate log level value.

$./runMigrator --log-level

Example:

$./runMigrator --log-level debug

Table 9-1 Supported Log Levels for NoSQL Database Migrator

Log Level Description warning Prints errors and warnings. info (default) Prints the progress status of data migration such as validating source, validating sink, creating tables, and count of number of data records migrated. debug Prints additional debug information. all Prints everything. This level turns on all levels of logging. • Log File:

9-8 Chapter 9 Sources and Sinks

You can specify the name of the log file using --log-file or -f parameter. If --log- file is passed as run time parameter to the runMigrator command, the NoSQL Database Migrator writes all the log messages to the file else to the standard output.

$./runMigrator --log-file

Example:

$./runMigrator --log-file nosql_migrator.log

Sources and Sinks

Learn about the different sources and sinks supported by the Oracle NoSQL Database Migrator utility and their configuration templates.

Topics: • Supported Sources and Sinks • Source Configuration Templates • Sink Configuration Templates • Transformation Configuration Templates Supported Sources and Sinks

This topic provides the list of the sources and sinks supported by the Oracle NoSQL Database Migrator. You can use any combination of a valid source and sink from this table for the migration activity. However, you must ensure that at least one of the ends, that is, source or sink must be an Oracle NoSQL product. You can not use the NoSQL Database Migrator to move the NoSQL table data from one file to another.

Valid Source Valid Sink Type Format (value) (value)

NA Y Y Oracle NoSQL Database (nosqldb)

NA Y Y Oracle NoSQL Database Cloud Service (nosqldb_cloud)

Y Y File system JSON (file) (json)

Y N MongoDB JSON (mongodb_json)

9-9 Chapter 9 Sources and Sinks

Valid Source Valid Sink Type Format (value) (value)

Y Y OCI Object Storage JSON (object_storage_oci) (json)

Y N MongoDB JSON (mongodb_json)

Note:

Many configuration parameters are common across the source and sink configuration. For ease of reference, the description for such parameters is repeated for each source and sink in the documentation sections, which explain configuration file formats for various types of sources and sinks. In all the cases, the syntax and semantics of the parameters with the same name are identical.

Source and Sink Security

Some of the source and sink types have optional or mandatory security information for authentication purposes. All sources and sinks that use services in the Oracle Cloud Infrastructure (OCI) can use certain parameters for providing optional security information. This information can be provided using an OCI configuration file or Instance Principal. Oracle NoSQL Database sources and sinks require mandatory security information if the installation is secure and uses an Oracle Wallet-based authentication. This information can be provided by adding a jar file to the /lib directory.

Wallet-based Authentication If an Oracle NoSQL Database installation uses Oracle Wallet-based authentication, you need an additional jar file that is part of the EE installation. For more information, see Oracle Wallet. Without this jar file, you will get the following error message: Could not find kvstore-ee.jar in lib directory. Copy kvstore- ee.jar to lib directory.

To prevent the exception shown above, you must copy the kvstore-ee.jar file from your EE server package to the /lib directory. is the nosql-migrator-M.N.O/ directory created by extracting the Oracle NoSQL Database Migrator package and M.N.O represent the software release.major.minor numbers. For example, nosql-migrator-1.1.0/lib.

9-10 Chapter 9 Sources and Sinks

Note:

The wallet-based authentication is supported ONLY in the Enterprise Edition (EE) of Oracle NoSQL Database.

Authenticating with Instance Principals Instance principals is an IAM service feature that enables instances to be authorized actors (or principals) that can perform actions on service resources. Each compute instance has its own identity, and it authenticates using the certificates added to it. Oracle NoSQL Database Migrator provides an option to connect to a NoSQL cloud and OCI Object Storage sources and sinks using instance principal authentication. It is only supported when the NoSQL Database Migrator tool is used within an OCI compute instance, for example, the NoSQL Database Migrator tool running in a VM hosted on OCI. To enable this feature use the useInstancePrincipal attribute of the NoSQL cloud source and sink configuration file. For more information on configuration parameters for different types of sources and sinks, see Source Configuration Templates and Sink Configuration Templates . For more information on instance principals, see Calling Services from an Instance. Source Configuration Templates

Learn about the configuration file formats for each valid source and the purpose of each configuration parameter.

Topics • JSON File • JSON File in OCI Object Storage Bucket • MongoDB-Formatted JSON File • MongoDB-Formatted JSON File in OCI Object Storage bucket • Oracle NoSQL Database • Oracle NoSQL Database Cloud Service JSON File

The configuration file format for JSON File as a source of NoSQL Database Migrator is shown below.

Configuration Template

"source" : { "type" : "file", "format" : "json", "dataPath": "", "schemaInfo": { "schemaPath": "" } }

9-11 Chapter 9 Sources and Sinks

Source Parameters • type • format • dataPath • schemaInfo • schemaInfo.schemaPath type • Purpose: Identifies the source type. • Data Type: string • Mandatory (Y/N): Y • Example: "type" : "file" format • Purpose: Specifies the source format. • Data Type: string • Mandatory (Y/N): Y • Example: "format" : "json" dataPath • Purpose: Specifies the absolute path to a file or directory containing the JSON data for migration. You must ensure that this data matches with the NoSQL table schema defined at the sink. If you specify a directory, the NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. Sub-directories are not supported. • Data Type: string • Mandatory (Y/N): Y • Example: – Specifying a JSON file "dataPath" : "/home/user/sample.json" – Specifying a directory "dataPath" : "/home/user" schemaInfo • Purpose: Specifies the schema of the source data being migrated. This schema is passed to the NoSQL sink. • Data Type: Object • Mandatory (Y/N): N

9-12 Chapter 9 Sources and Sinks

schemaInfo.schemaPath • Purpose: Specifies the absolute path to the schema definition file containing DDL statements for the NoSQL table being migrated. • Data Type: string • Mandatory (Y/N): Y • Example:

"schemaInfo" : { "schemaPath" : "/home/user/mytable/Schema/schema.ddl" }

JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a source of NoSQL Database Migrator is shown below.

Note:

The valid sink types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"source" : { "type" : "object_storage_oci", "format" : "json", "endpoint" : "", "bucket" : "", "prefix" : "

", "compartment" : "", "credentials" : "", "credentialsProfile" : "", "readUnitsPercent" :

, "requestTimeoutMs" : , "useInstancePrincipal" : }

Source Parameters • type

9-24 Chapter 9 Sources and Sinks

• endpoint • table • compartment • credentials • credentialsProfile • readUnitsPercent • requestTimeoutMs • useInstancePrincipal type • Purpose: Identifies the source type. • Data Type: string • Mandatory (Y/N): Y • Example: "type" : "nosqldb_cloud" endpoint • Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service. You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service. • Data Type: string • Mandatory (Y/N): Y • Example: – Region ID: "endpoint" : "us-ashburn-1" – URL format: "endpoint" : "https://nosql.us-ashburn-1.oci.oraclecloud.com/" table • Purpose: Name of the table from which to migrate the data. • Data Type: string • Mandatory (Y/N): Y • Example: "table" :"myTable" compartment • Purpose: Specifies the name or OCID of the compartment in which the table resides. If you do not provide any value, it defaults to the root compartment. You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console. • Data Type: string • Mandatory (Y/N): Yes, if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

9-25 Chapter 9 Sources and Sinks

Note:

If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.

Note:

Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.

• Data Type: string • Mandatory (Y/N): N • Example: 1. "credentials" : "/home/user/.oci/config"

2. "credentials" : "/home/user/security/config" credentialsProfile • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. User account credentials are referred to as a 'profile'. If you do not specify this value, it defaults to the DEFAULT profile.

9-26 Chapter 9 Sources and Sinks

Note:

This parameter is valid ONLY if the credentials parameter is specified.

• Data Type: string • Mandatory (Y/N): N • Example: 1. "credentialsProfile" : "DEFAULT"

2. "credentialsProfile": "ADMIN_USER" readUnitsPercent • Purpose: Percentage of table read units to be used while migrating the NoSQL table. The default value is 90. The valid range is any integer between 1 to 100. Please note that amount of time required to migrate data is directly proportional to this attribute. It's better to increase the read throughput of the table for the migration activity. You can reduce the read throughput after the migration process completes. To learn the daily limits on throughput changes, see Cloud Limits in Using Oracle NoSQL Database Cloud Service. The default value is 90. The valid range is any integer between 1 to 100.

Note:

The time required for the data migration is directly proportional to the writeUnitsPercent value.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed. • Data Type: integer • Mandatory (Y/N): N • Example: "readUnitsPercent" : 90 requestTimeoutMs • Purpose: Specifies the time to wait for each read operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer. • Data Type: integer • Mandatory (Y/N): N • Example: "requestTimeoutMs" : 5000 useInstancePrincipal • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security

9-27 Chapter 9 Sources and Sinks

If not specified, it defaults to false.

Note:

– It is supported ONLY when the NoSQL Database Migrator tool is running within an OCI compute instance, for example, NoSQL Database Migrator tool running in a VM hosted on OCI. – Even though the credentials and useInstancePrincipal parameters are not mandatory individually, one of these parameters MUST be specified. Additionally, these two parameters are mutually exclusive. Specify ONLY one of these parameters, but not both at the same time.

• Data Type: boolean • Mandatory (Y/N): N • Example: "useInstancePrincipal" : true Sink Configuration Templates

Learn about the configuration file formats for each valid sink and the purpose of each configuration parameter.

Topics • JSON File • JSON File in OCI Object Storage Bucket • Oracle NoSQL Database • Oracle NoSQL Database Cloud Service JSON File

The configuration file format for JSON File as a sink of NoSQL Database Migrator is shown below.

Configuration Template

"sink" : { "type" : "file", "format" : "json", "dataPath": "", "schemaPath" : "", "pretty" : , "useMultiFiles" : , "chunkSize" : }

Sink Parameters • type

9-28 Chapter 9 Sources and Sinks

• format • dataPath • schemaPath • pretty • useMultiFiles • chunkSize type • Purpose: Identifies the sink type. • Data Type: string • Mandatory (Y/N): Y • Example: "type" : "file" format • Purpose: Specifies the sink format. • Data Type: string • Mandatory (Y/N): Y • Example: "format" : "json" dataPath • Purpose: Specifies the absolute path to a file where the source data will be copied in the JSON format. If the file does not exist in the specified data path, the NoSQL Database Migrator creates it. If it exists already, the NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory for the file specified in the data path is valid.

Note:

If the useMultiFiles parameter is set to true, specify the path to a directory else specifies the path to the file.

• Data Type: string • Mandatory (Y/N): Y • Example: – With useMultiFiles parameter set to true "dataPath" :"/home/user/data" – With useMultiFiles parameter not specified or it is set to false "dataPath" :"/home/user/sample.json"

9-29 Chapter 9 Sources and Sinks

schemaPath • Purpose: Specifies the absolute path to write schema information provided by the source. If this value is not defined, the source schema information will not be migrated to the sink. If this value is specified, the migrator utility writes the schema of the source table into the file specified here. The schema information is written as one DDL command per line in this file. If the file does not exist in the specified data path, NoSQL Database Migrator creates it. If it exists already, NoSQL Database Migrator will overwrite its contents with the source data. You must ensure that the parent directory for the file specified in the data path is valid. • Data Type: string • Mandatory (Y/N): N • Example: "schemaPath" : "/home/user/schema_file" pretty • Purpose: Specifies whether to beautify the JSON output to increase readability or not. If not specified, it defaults to false. • Data Type: boolean • Mandatory (Y/N): N • Example: "pretty" : true useMultiFiles • Purpose: Specifies whether or not to split the NoSQL table data into multiple files when migrating source data to a file. If not specified, it defaults to false. If set to true, when migrating source data to a file, the NoSQL table data is split into multiple smaller files. For example, .json, where chunk=000000,000001,000002, and so forth.

dataPath |--000000.json |--000001.json

• Data Type: boolean • Mandatory (Y/N): N • Example: "useMultiFiles" : true chunkSize • Purpose: Specifies the maximum size of a "chunk" of table data to be stored at the sink. During migration, a table is split into chunkSize chunks and each chunk is written as a separate file to the sink. When the source data being migrated exceeds this size, a new file is created.

9-30 Chapter 9 Sources and Sinks

If not specified, defaults to 32MB. The valid value is an integer between 1 to 1024.

Note:

This parameter is applicable ONLY when the useMultiFiles parameter is set to true.

• Data Type: integer • Mandatory (Y/N): N • Example: "chunkSize" : 40 JSON File in OCI Object Storage Bucket

The configuration file format for JSON file in OCI Object Storage bucket as a sink of NoSQL Database Migrator is shown below.

Note:

The valid source types for OCI Object Storage source type are nosqldb and nosqldb_cloud.

Configuration Template

"sink" : { "type" : "object_storage_oci", "format" : "json", "endpoint" : "", "bucket" : "", "prefix" : "

", "compartment" : "", "schemaInfo" : { "schemaPath" : "", "defaultSchema" : , "useSourceSchema" : , "readUnits" :

, "writeUnits" :

, "storageSize" : }, "credentials" : "", "credentialsProfile" : "", "writeUnitsPercent" :

, "requestTimeoutMs" : , "useInstancePrincipal" : , "overwrite" : }

Sink Parameters • type • endpoint • table

9-39 Chapter 9 Sources and Sinks

• compartment • schemaInfo • schemaInfo.schemaPath • schemaInfo.defaultSchema • schemaInfo.useSourceSchema • schemaInfo.readUnits • schemaInfo.writeUnits • schemaInfo.storageSize • credentials • credentialsProfile • writeUnitsPercent • requestTimeoutMs • useInstancePrincipal • overwrite type • Purpose: Identifies the sink type. • Data Type: string • Mandatory (Y/N): Y • Example: "type" : "nosqldb_cloud" endpoint • Purpose: Specifies the Service Endpoint of the Oracle NoSQL Database Cloud Service. You can either specify the complete URL or the Region ID alone. See Data Regions and Associated Service URLs in Using Oracle NoSQL Database Cloud Service for the list of data regions supported for Oracle NoSQL Database Cloud Service. • Data Type: string • Mandatory (Y/N): Y • Example: – Region ID: "endpoint" : "us-ashburn-1" – URL format: "endpoint" : "https://nosql.us- ashburn-1.oci.oraclecloud.com/" table • Purpose: Name of the table to which to migrate the data. You must ensure that this table exists in your Oracle NoSQL Database Cloud Service. Otherwise, you have to use the schemaInfo object in the sink configuration to instruct the NoSQL Database Migrator to create the table. The schema of this table must match the source data.

9-40 Chapter 9 Sources and Sinks

• Data Type: string • Mandatory (Y/N): Y • Example: "table" :"myTable" compartment • Purpose: Specifies the name or OCID of the compartment in which the table resides. If you do not provide any value, it defaults to the root compartment. You can find your compartment's OCID from the Compartment Explorer window under Governance in the OCI Cloud Console. • Data Type: string • Mandatory (Y/N): Y if the table is not in the root compartment of the tenancy OR when the useInstancePrincipal parameter is set to true.

Note:

If the useInstancePrincipal parameter is set to true, the compartment must specify the compartment OCID and not the name.

• Example: – Compartment name "compartment" : "mycompartment" – Compartment name qualified with its parent compartment "compartment" : "parent.childcompartment" – No value provided. Defaults to the root compartment. "compartment": "" – Compartment OCID "compartment" : "ocid1.tenancy.oc1...4ksd" schemaInfo • Purpose: Specifies the schema for the data being migrated. If you do not specify this parameter, the NoSQL Database Migrator assumes that the table already exists in your Oracle NoSQL Database Cloud Service. If this parameter is not specified and the table does not exist in the sink, the migration fails. • Data Type: Object • Mandatory (Y/N): N schemaInfo.schemaPath • Purpose: Specifies the absolute path to a file containing DDL statements for the NoSQL table. The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data.

9-41 Chapter 9 Sources and Sinks

The NoSQL Database Migrator does not support more than one DDL statement per line in the schemaPath file. • Data Type: string • Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No. schemaInfo.defaultSchema • Purpose: Setting this parameter to Yes instructs the NoSQL Database Migrator to create a table with default schema. The default schema is defined by the migrator itself. For more information about default schema definitions, see Default Schema. • Data Type: boolean • Mandatory: Y, only when the schemaInfo.defaultSchema parameter is set to No.

Note:

defaultSchema and schemaPath are mutually exclusive schemaInfo.useSourceSchema • Purpose: Specifies whether or not the sink uses the table schema definition provided by the source when migrating NoSQL tables. • Data Type: boolean • Mandatory (Y/N): N

Note:

defaultSchema, schemaPath, and useSourceSchema parameters are mutually exclusive. Specify ONLY one of these parameters.

• Example: – With Default Schema:

"schemaInfo" : { "defaultSchema" : true, "readUnits" : 100, "writeUnits" : 60, "storageSize" : 1 }

– With a pre-defined schema:

"schemaInfo" : { "schemaPath" : "", "readUnits" : 100, "writeUnits" : 100, "storageSize" : 1 }

9-42 Chapter 9 Sources and Sinks

– With source schema:

"schemaInfo" : { "useSourceSchema" : true, "readUnits" : 100, "writeUnits" : 60, "storageSize" : 1 } schemaInfo.readUnits • Purpose: Specifies the read throughput of the new table. • Data Type: integer • Mandatory: Y schemaInfo.writeUnits • Purpose: Specifies the write throughput of the new table. • Data Type: integer • Mandatory: Y schemaInfo.storageSize • Purpose: Specifies the storage size of the new table in GB • Data Type: integer • Mandatory: Y • Example: – With schemaPath

"schemaInfo" : { "schemaPath" : "", "readUnits" : 500, "writeUnits" : 1000, "storageSize" : 5 }

– With defaultSchema

"schemaInfo" : { "defaultSchema" :Yes, "readUnits" : 500, "writeUnits" : 1000, "storageSize" : 5 } credentials • Purpose: Absolute path to a file containing OCI credentials. If not specified, it defaults to $HOME/.oci/config See Example Configuration for an example of the credentials file.

9-43 Chapter 9 Sources and Sinks

Note:

• Data Type: string • Mandatory (Y/N): N • Example: 1. "credentials" : "/home/user/.oci/config"

2. "credentials" : "/home/user/security/config" credentialsProfile • Purpose: Name of the configuration profile to be used to connect to the Oracle NoSQL Database Cloud Service. If you do not specify this value, it defaults to the DEFAULT profile.

Note:

This parameter is valid ONLY if the credentials parameter is specified.

• Data Type: string • Mandatory (Y/N): N • Example: 1. "credentialsProfile" : "DEFAULT"

2. "credentialsProfile": "ADMIN_USER" writeUnitsPercent • Purpose: Specifies the Percentage of table write units to be used during the migration activity. The default value is 90. The valid range is any integer between 1 to 100.

Note:

The time required for the data migration is directly proportional to the writeUnitsPercent value.

See Troubleshooting the Oracle NoSQL Database Migrator to learn how to use this attribute to improve the data migration speed. • Data Type: integer

9-44 Chapter 9 Sources and Sinks

• Mandatory (Y/N): N • Example: "writeUnitsPercent" : 90 requestTimeoutMs • Purpose: Specifies the time to wait for each write operation in the sink to complete. This is provided in milliseconds. The default value is 5000. The value can be any positive integer. • Data Type: integer • Mandatory (Y/N): N • Example: "requestTimeoutMs" : 5000 useInstancePrincipal • Purpose: Specifies whether or not the NoSQL Migrator tool uses instance principal authentication to connect to Oracle NoSQL Database Cloud Service. For more information on Instance Principal authentication method, see Source and Sink Security If not specified, it defaults to false.

Note:

• Data Type: boolean • Mandatory (Y/N): N • Example: "useInstancePrincipal" : true overwrite • Purpose: Indicates the behavior of NoSQL Database Migrator when the record being migrated from the source is already present in the sink. If the value is set to false, when migrating tables the NoSQL Database Migrator skips those records for which the same primary key already exists in the sink. If the value is set to true, when migrating tables the NoSQL Database Migrator overwrites those records for which the same primary key already exists in the sink. If not specified, it defaults to true. • Data Type: boolean • Mandatory (Y/N): N • Example: "overwrite" : false

9-45 Chapter 9 Sources and Sinks

Transformation Configuration Templates

This topic explains the configuration parameters for the different transformations supported by the Oracle NoSQL Database Migrator. Oracle NoSQL Database Migrator lets you modify the data, that is, add data transformations as part of the migration activity. You can define multiple transformations in a single migration. In such a case, the order of transformations is vital because the source data undergoes each transformation in the given order. The output of one transformation becomes the input to the next one in the migrator pipeline. The different transformations supported by the NoSQL Data Migrator are:

Table 9-2 Transformations

Transformation Config Attribute You can use this transformation to ... ignoreFields Ignore the identified columns from the source row before writing to the sink. renameFields Rename the identified columns from the source row before writing to the sink. aggregateFields Aggregate multiple columns from the source into a single column in the sink. As part of this transformation, you can also identify the columns that you want to exclude in the aggregation. Those fields will be skipped from the aggregated column.

You can find the configuration template for each supported transformation below. • ignoreFields • renameFields • aggregateFields ignoreFields

The configuration file format for the ignoreFields transformation is shown below.

Configuration Template

"transforms" : { "ignoreFields" : ["","",...] }

Transformation Parameter

ignoreFields • Purpose: An array of the column names to be ignored from the source records.

9-46 Chapter 9 Sources and Sinks

Note:

You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.

• Data Type: array of strings • Mandatory (Y/N): Y • Example: To ignore the columns named "name" and "address" from the source record: "ignoreFields" : ["name","address"] renameFields

The configuration file format for the renameFields transformation is shown below.

Configuration Template

"transforms" : { "renameFields" : { "" : "", "" : "," ..... } }

Transformation Parameter

renameFields • Purpose: Key-Value pairs of the old and new names of the columns to be renamed.

Note:

You can supply only top-level fields. Transformations can not be applied on the data in the nested fields.

• Data Type: JSON object • Mandatory (Y/N): Y • Example: To rename the column named "residence" to "address" and the column named "_id" to "id": "renameFields" : { "residence" : "address", "_id" : "id" } aggregateFields

The configuration file format for the aggregateFields transformation is shown below.

Configuration Template

"transforms" : { "aggregateFields" : {

9-47 Chapter 9 Use Case Demonstrations

"fieldName" : "name of the new aggregate field", "skipFields" : ["",",...] } }

Transformation Parameter

aggregateFields • Purpose: Name of the aggregated field in the sink. • Data Type: string • Mandatory (Y/N): Y • Example: If the given record is:

{ "id" : 100, "name" : "john", "address" : "USA", "age" : 20 }

If the aggregate transformation is:

"aggregateFields" : { "fieldName" : "document", "skipFields" : "id" }

The aggregated column in the sink looks like:

{ "id": 100, "document": { "name": "john", "address": "USA", "age": 20 } }

Use Case Demonstrations

Learn how to perform data migration using the Oracle NoSQL Database Migrator for specific use cases. You can find detailed systematic instructions with code examples to perform migration in each of the use cases listed below. Topics: • Migrate from Oracle NoSQL Database Cloud Service to a JSON file • Migrate from Oracle NoSQL Database On-Premise to Oracle NoSQL Database Cloud Service

9-48 Chapter 9 Use Case Demonstrations

• Migrate from MongoDB-Formatted JSON file to an Oracle NoSQL Database Cloud Service Migrate from Oracle NoSQL Database Cloud Service to a JSON file

This example shows how to use the Oracle NoSQL Database Migrator to copy data and the schema definition of a NoSQL table from Oracle NoSQL Database Cloud Service (NDCS) to a JSON file.

Use Case An organization decides to train a model using the Oracle NoSQL Database Cloud Service (NDCS) data to predict future behaviors and provide personalized recommendations. They can take a periodic copy of the NDCS tables' data to a JSON file and apply it to the analytic engine to analyze and train the model. Doing this helps them separate the analytical queries from the low-latency critical paths.

Example For the demonstration, let us look at how to migrate the data and schema definition of a NoSQL table called myTable from NDCS to a JSON file. Prerequisites • Identify the source and sink for the migration. – Source: Oracle NoSQL Database Cloud Service – Sink: JSON file • Identify your OCI cloud credentials and capture them in the OCI config file. Save the config file in /home/.oci/config. See Acquiring Credentials in Using Oracle NoSQL Database Cloud Service.

[DEFAULT] tenancy=ocid1.tenancy.oc1.... user=ocid1.user.oc1.... fingerprint= 43:d1:.... key_file= pass_phrase=

• Identify the region endpoint and compartment name for your Oracle NoSQL Database Cloud Service. – endpoint: us-phoenix-1 – compartment: developers Procedure To migrate the data and schema definition of myTable from Oracle NoSQL Database Cloud Service to a JSON file: 1. Open the command prompt and navigate to the directory where you extracted the NoSQL Database Migrator utility. 2. To generate the configuration JSON file using the NoSQL Database Migrator, run the runMigrator command without any runtime parameters.

[~/nosqlMigrator/nosql-migrator-1.0.0]$./runMigrator

9-49 Chapter 9 Use Case Demonstrations

3. As you did not provide the configuration file as a runtime parameter, the utility prompts if you want to generate the configuration now. Type y.

configuration file is not provided. Do you want to generate configuration? (y/n) [n]: y

This command provides a walkthrough of creating a valid config for Oracle NoSQL data migrator.

The following link explain where to find the information required by this script:

4. Based on the prompts from the utility, choose your options for the Source configuration.

Enter a location for your config [./migrator-config.json]: /home/ apothula/nosqlMigrator/NDCS2JSON Select the source: 1) nosqldb 2) nosqldb_cloud 3) file #? 2 Configuration for source type=nosqldb_cloud Enter endpoint URL or region of the Oracle NoSQL Database Cloud: us- phoenix-1 Enter table name: myTable Enter compartment name or id of the source table []: developers Enter path to the file containing OCI credentials [/home/ apothula/.oci/config]: Enter the profile name in OCI credentials file [DEFAULT]: Enter percentage of table read units to be used for migration operation. (1-100) [90]: Enter store operation timeout in milliseconds. (1-30000) [5000]:

5. Based on the prompts from the utility, choose your options for the Sink configuration.

Select the sink: 1) nosqldb 2) nosqldb_cloud 3) file #? 3 Configuration for sink type=file Enter path to a file to store JSON data: /home/apothula/ nosqlMigrator/myTableJSON Would you like to store JSON in pretty format? (y/n) [n]: y Would you like to migrate the table schema also? (y/n) [y]: y Enter path to a file to store table schema: /home/apothula/ nosqlMigrator/myTableSchema

9-50 Chapter 9 Use Case Demonstrations

6. Based on the prompts from the utility, choose your options for the source data transformations. The default value is n.

Would you like to add transformations to source data? (y/n) [n]:

7. Enter your choice to determine whether to proceed with the migration in case any record fails to migrate.

Would you like to continue migration in case of any record/row is failed to migrate?: (y/n) [n]:

8. The utility displays the generated configuration on the screen.

generated configuration is: { "source": { "type": "nosqldb_cloud", "endpoint": "us-phoenix-1", "table": "myTable", "compartment": "developers", "credentials": "/home/apothula/.oci/config", "credentialsProfile": "DEFAULT", "readUnitsPercent": 90, "requestTimeoutMs": 5000 }, "sink": { "type": "file", "format": "json", "schemaPath": "/home/apothula/nosqlMigrator/myTableSchema", "pretty": true, "dataPath": "/home/apothula/nosqlMigrator/myTableJSON" }, "abortOnError": true, "migratorVersion": "1.0.0" }

9. Finally, the utility prompts for your choice to decide whether to proceed with the migration with the generated configuration file or not. The default option is y.

Note:

If you select n, you can use the generated configuration file to run the migration using the ./runMigrator -c or the ./runMigrator --config option.

would you like to run the migration with above configuration? If you select no, you can use the generated configuration file to run the migration using ./runMigrator --config /home/apothula/nosqlMigrator/NDCS2JSON (y/n) [y]:

9-51 Chapter 9 Use Case Demonstrations

10. The NoSQL Database Migrator migrates your data and schema from NDCS to the JSON file.

Records provided by source=10,Records written to sink=10,Records failed=0. Elapsed time: 0min 1sec 277ms Migration completed.

Validation To validate the migration, you can open the JSON Sink files and view the schema and data.

-- Exported myTable Data

[~/nosqlMigrator]$cat myTableJSON { "id" : 10, "document" : { "course" : "Computer Science", "name" : "Neena", "studentid" : 105 } } { "id" : 3, "document" : { "course" : "Computer Science", "name" : "John", "studentid" : 107 } } { "id" : 4, "document" : { "course" : "Computer Science", "name" : "Ruby", "studentid" : 100 } } { "id" : 6, "document" : { "course" : "Bio-Technology", "name" : "Rekha", "studentid" : 104 } } { "id" : 7, "document" : { "course" : "Computer Science", "name" : "Ruby", "studentid" : 100 } }

9-52 Chapter 9 Use Case Demonstrations

{ "id" : 5, "document" : { "course" : "Journalism", "name" : "Rani", "studentid" : 106 } } { "id" : 8, "document" : { "course" : "Computer Science", "name" : "Tom", "studentid" : 103 } } { "id" : 9, "document" : { "course" : "Computer Science", "name" : "Peter", "studentid" : 109 } } { "id" : 1, "document" : { "course" : "Journalism", "name" : "Tracy", "studentid" : 110 } } { "id" : 2, "document" : { "course" : "Bio-Technology", "name" : "Raja", "studentid" : 108 } }

-- Exported myTable Schema

[~/nosqlMigrator]$cat myTableSchema CREATE TABLE IF NOT EXISTS myTable (id INTEGER, document JSON, PRIMARY KEY(SHARD(id)))

9-53 Chapter 9 Use Case Demonstrations

Migrate from Oracle NoSQL Database On-Premise to Oracle NoSQL Database Cloud Service

This example shows how to use the Oracle NoSQL Database Migrator to copy data and the schema definition of a NoSQL table from Oracle NoSQL Database to Oracle NoSQL Database Cloud Service (NDCS).

Use Case As a developer, you are exploring options to avoid the overhead of managing the resources, clusters, and garbage collection for your existing NoSQL Database KVStore workloads. As a solution, you decide to migrate your existing on-premise KVStore workloads to Oracle NoSQL Database Cloud Service because NDCS manages them automatically.

Example For the demonstration, let us look at how to migrate the data and schema definition of a NoSQL table called myTable from the NoSQL Database KVStore to NDCS. We will also use this use case to show how to run the runMigrator utility by passing a pre- created configuration JSON file. Prerequisites • Identify the source and sink for the migration. – Source: Oracle NoSQL Database – Sink: Oracle NoSQL Database Cloud Service • Identify your OCI cloud credentials and capture them in the OCI config file. Save the config file in /home/.oci/config. See Acquiring Credentials in Using Oracle NoSQL Database Cloud Service.

[DEFAULT] tenancy=ocid1.tenancy.oc1.... user=ocid1.user.oc1.... fingerprint= 43:d1:.... key_file= pass_phrase=

• Identify the region endpoint and compartment name for your Oracle NoSQL Database Cloud Service. – endpoint: us-phoenix-1 – compartment: developers • Identify the following details for the on-premise KVStore: – storeName: kvstore – helperHosts: :5000 – table: myTable Procedure To migrate the data and schema definition of myTable from NoSQL Database KVStore to NDCS:

9-54 Chapter 9 Use Case Demonstrations

1. Prepare the configuration JSON file with the identified Source and Sink details. See Source Configuration Templates and Sink Configuration Templates .

{ "source" : { "type" : "nosqldb", "storeName" : "kvstore", "helperHosts" : [":5000"], "table" : "myTable", "requestTimeoutMs" : 5000 }, "sink" : { "type" : "nosqldb_cloud", "endpoint" : "us-phoenix-1", "table" : "myTable", "compartment" : "developers", "schemaInfo" : { "schemaPath" : "", "readUnits" : 100, "writeUnits" : 100, "storageSize" : 1 }, "credentials" : "", "credentialsProfile" : "DEFAULT", "writeUnitsPercent" : 90, "requestTimeoutMs" : 5000 }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

2. Open the command prompt and navigate to the directory where you extracted the NoSQL Database Migrator utility. 3. Run the runMigrator command by passing the configuration JSON file using the -- config or -c option.

[~/nosqlMigrator/nosql-migrator-1.0.0]$./runMigrator --config

4. The utility proceeds with the data migration, as shown below.

Records provided by source=10, Records written to sink=10, Records failed=0. Elapsed time: 0min 10sec 426ms Migration completed.

Validation To validate the migration, you can login to your NDCS console and verify that myTable is created with the source data.

9-55 Chapter 9 Use Case Demonstrations

Migrate from MongoDB-Formatted JSON file to an Oracle NoSQL Database Cloud Service

This example shows how to use the Oracle NoSQL Database Migrator to copy Mongo- DB Formatted Data to the Oracle NoSQL Database Cloud Service (NDCS).

Use Case After evaluating multiple options, an organization finalizes Oracle NoSQL Database Cloud Service as its NoSQL Database platform. As its NoSQL tables and data are in MongoDB, they are looking for a way to migrate those tables and data to Oracle NDCS.

Example For the demonstration, let us look at how to migrate a MongoDB-formatted JSON file to NDCS. We will use a manually created configuration JSON file for this example. Prerequisites • Identify the source and sink for the migration. – Source: MongoDB-Formatted JSON File – Sink: Oracle NoSQL Database Cloud Service • Extract the data from Mongo DB using the mongoexport utility. See mongoexport for more information. • Create a NoSQL table in the sink with a table schema that matches the data in the Mongo-DB-formatted JSON file. As an alternative, you can instruct the NoSQL Database Migratorto create a table with the default schema structure by setting the defaultSchema attribute to true.

Note:

For a MongoDB-Formatted JSON source, the default schema for the table will be as:

CREATE TABLE IF NOT EXISTS (ID STRING, DOCUMENT JSON,PRIMARY KEY(SHARD(ID));

Where: – tablename = value of the table config. – ID = _id value from the mongoDB exported JSON source file. – DOCUMENT = The entire contents of the mongoDB exported JSON source file is aggregated into the DOCUMENT column excluding the _id field.

9-56 Chapter 9 Use Case Demonstrations

• Identify your OCI cloud credentials and capture them in the OCI config file. Save the config file in /home/.oci/config. See Acquiring Credentials in Using Oracle NoSQL Database Cloud Service.

[DEFAULT] tenancy=ocid1.tenancy.oc1.... user=ocid1.user.oc1.... fingerprint= 43:d1:.... key_file= pass_phrase=

• Identify the region endpoint and compartment name for your Oracle NoSQL Database Cloud Service. – endpoint: us-phoenix-1 – compartment: developers Procedure To migrate the MongoDB-formatted JSON data to the Oracle NoSQL Database Cloud Service:

1. Prepare the configuration JSON file with the identified Source and Sink details. See Source Configuration Templates and Sink Configuration Templates .

{ "source" : { "type" : "file", "format" : "mongodb_json", "dataPath" : "" }, "sink" : { "type" : "nosqldb_cloud", "endpoint" : "us-phoenix-1", "table" : "mongoImport", "compartment" : "developers", "schemaInfo" : { "defaultSchema" : true, "readUnits" : 100, "writeUnits" : 60, "storageSize" : 1 }, "credentials" : "", "credentialsProfile" : "DEFAULT", "writeUnitsPercent" : 90, "requestTimeoutMs" : 5000 }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

2. Open the command prompt and navigate to the directory where you extracted the NoSQL Database Migrator utility.

9-57 Chapter 9 Troubleshooting the Oracle NoSQL Database Migrator

3. Run the runMigrator command by passing the configuration JSON file using the --config or -c option.

[~/nosqlMigrator/nosql-migrator-1.0.0]$./runMigrator --config

4. The utility proceeds with the data migration, as shown below.

Records provided by source=29,353, Records written to sink=29,353, Records failed=0. Elapsed time: 9min 9sec 630ms Migration completed.

Validation To validate the migration, you can login to your NDCS console and verify that myTable is created with the source data. Troubleshooting the Oracle NoSQL Database Migrator

Learn about the general challenges that you may face while using the , and how to resolve them.

Migration has failed. How can I resolve this? A failure of the data migration can be because of multiple underlying reasons. The important causes are listed below:

Table 9-3 Migration Failure Causes

Error Message Meaning Resolution Failed to connect to The migrator could not · Check if the values of the Oracle NoSQL Database establish a connection with the storeName and NoSQL Database. helperHosts attributes in the configuration JSON file are valid and that the hosts are reachable. · For a secured store, verify if the security file is valid with correct user name and password values. Failed to connect to The migrator could not · Verify if the endpoint URL Oracle NoSQL Database establish a connection with the or region name specified Cloud Service Oracle NoSQL Database in the configuration JSON Cloud Service. file is correct. · Check if the OCI credentials file is available in the path specified in the configuration JSON file. · Ensure that the OCI credentials provided in the OCI credentials are valid.

9-58 Chapter 9 Troubleshooting the Oracle NoSQL Database Migrator

Table 9-3 (Cont.) Migration Failure Causes

Error Message Meaning Resolution Table not found The table identified for the For the Source: migration could not be located · Verify if the table is by the NoSQL Database present in the source Migrator. database. · Ensure that the table is qualified with its namespace in the configuration JSON file, if the table is created in a non-default namespace. · Verify if you have the required read/write authorization to access the table. · If the source is Oracle NoSQL Database Cloud Service, verify if the valid compartment name is specified in the configuration JSON file, and ensure that you have the required authorization to access the table. For the Sink: · Verify if the table is present in the Sink. If it does not exist, you must either create the table manually or use the schemaInfo config to create it through the migration. DDL Execution failed The DDL commands provided · Check the syntax of the in the input schema definition DDL commands in the file is invalid. schemaPath file. · Ensure that there is only one DDL statement per line in the schemaPath file. failed to write record The input record is not · Check if the data types to the sink table with matching with the table and column names java.lang.IllegalArgume schema of the sink. specified in the target sink ntException table are matching with sink table schema. · If you applied any transformation, check if the transformed records are matching with the sink table schema.

9-59 Chapter 9 Troubleshooting the Oracle NoSQL Database Migrator

Table 9-3 (Cont.) Migration Failure Causes

Error Message Meaning Resolution Request timeout The source or sink©s operation · Verify the network did not complete within the connection. expected time. · Check if the NoSQL Database is up and running. · Try to increase requestTimeout value in the configuration JSON file.

What should I consider before restarting a failed migration? When a data migration task fails, the sink will be at an intermediate state containing the imported data until the point of failure. You can identify the error and failure details from the logs and restart the migration after diagnosing and correcting the error. A restarted migration starts over, processing all data from the beginning. There is no way to checkpoint and restart the migration from the point of failure. Therefore, NoSQL Database Migrator overwrites any record that was migrated to the sink already.

Migration is too slow. How can I speed it up? The time taken for the data migration depends on multiple factors such as volume of data being migrated, network speed, current load on the database. In case of a cloud service, the speed of migration also depends on the read throughput and the write throughput provisioned. So, to improve the migration speed, you can: • Try to reduce the current workload on your Oracle NoSQL Database while migrating the data. • Ensure that the machine that is running the migration, source, and sink all are located in the same data center and the network latencies are minimal. • In case of Oracle NoSQL Database Cloud Service, provision high read/write throughput and verify if the storage allocated for table is sufficient or not. If the NoSQL Database Migrator is not creating the table, you can increase the write throughput. If the migrator is creating the table, consider specifying a higher value for the schemaInfo.writeUnits parameter in the sink configuration. Once the data migration completes, you can lower this value. Be aware of daily limits on throughput changes. see Cloud Limits and Sink Configuration Templates .

I have a long running migration involving huge datasets. How can I track the progress of the migration? You can enable additional logging to track the progress of a long-running migration. To control the logging behavior of Oracle NoSQL Database Migrator, you must set the desired level of logging in the logging.properties file. This file is provided with the NoSQL Database Migrator package and available in the directory where the Oracle NoSQL Database Migrator was unpacked. The different levels of logging are OFF, SEVERE, WARNING, INFO, FINE, and ALL in the order of increasing verbosity. Setting the log level to OFF turns off all the logging information, whereas setting the log level to ALL provides the full log information. The default log level is WARNING. All the logging output is configured to go to the console by default. You can see comments in the logging.properties file to know about each log level.

9-60 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

Oracle NoSQL Database Migrator Vs. Import/Export Utility

This topic explains how the Oracle NoSQL Database Migrator utility is different from the existing Oracle NoSQL import/export utility. The Oracle NoSQL Database Migrator is created to replace and enhance the existing on- premise-only import/export utility. It moves the NoSQL table data and schema definition between a source and a sink or target. It supports multiple sources and sinks as listed in Supported Sources and Sinks. However, the import/export utility lets you import into or export from Oracle NoSQL Database (on-premise) only. That is, using the import/export utility, you can either import data into the Oracle NoSQL Database or export data from Oracle NoSQL Database. When you export, the source type is always Oracle NoSQL Database (where you extract data from) and the sink is the recipient of that data. When you import, the source type is currently limited to a file and the sink is always Oracle NoSQL Database. See Export and Import Functionality. Apart from this fundamental difference, both the utilities are different in many other ways. You can see them in the comparison table below.

Table 9-4 Comparison Table

- Oracle NoSQL Database Oracle NoSQL Import/Export Migrator Sources and Sinks Supports multiple sources and Import sinks such as Oracle NoSQL Source: JSON, binary, or Database (on-premise), Oracle MongoDB-formatted JSON NoSQL Database Cloud Service, Sink: Oracle NoSQL Database JSON file, and MongoDB- (on-premise) only. formatted JSON file. See Supported Sources and Sinks. Export Source: Oracle NoSQL Database (on-premise) only. Sink: Local or network mounted filesystem. Supported Platforms Both the Oracle NoSQL Only Oracle NoSQL Database Database (on-premise) and the (on-premise). Oracle NoSQL Database Cloud Service (NDCS). Migration Level The migration is supported only You can perform the import/ at the table level. You can export operations at a table level, migrate only one table at a time. store level, or an namespace level. You can import/export multiple tables in a single operation. Data Formats Supports only JSON format for Supports both the Binary and the data. The schema definition JSON formats. is represented as a file with one DDL command per line.

9-61 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

Table 9-4 (Cont.) Comparison Table

- Oracle NoSQL Database Oracle NoSQL Import/Export Migrator Unit of Migration The Oracle NoSQL Database When you export data from Migrator migrates the data and Oracle NoSQL Database, the schema definition from the import/export utility creates an Source to the Sink without export package (a directory creating any intermediate structure) that contains schema packages. metadata and table data. When you import data into Oracle NoSQL Database, you can run the utility against this export package, or any directory that contains files with data formats supported by the import/export utility. Key-Value and Large Object Not supported. Supported by the export Data command. NoSQL Database on-premise Supported. Not supported. to on-premise Migration

Example 1: You export data from a table in an on-premise NoSQL datastore in JSON format and import the content into another on-premise NoSQL datastore. A) Steps using Import/Export Utility

1. Export the contents from the table users in kvstore1 and place the export package into the /users/oracle/kvstore_export directory. Specify the format as JSON.

java -jar lib/kvtool.jar export -table users \ -store kvstore1 -helper-hosts :5000 \ -config export_config -format JSON

The export_config file resides in the current working directory and the file content is as below:

{ "path": "/users/oracle/kvstore_export" }

Directories are created in the kvstore_export directory for data and schema definition. Inside the data directory, every table has a directory and the corresponding json file is stored inside the table directory. 2. Import the users table data from the export package created above into a different Oracle NoSQL Database store kvstore2. Use checkpoints to be able to restart the import from where it stopped or got aborted, in case it fails. The parameter -table is the name of the table or tables you want to import. To import multiple tables, specify a comma-delimited list of table names. Specify the format as JSON.

java -jar lib/kvtool.jar import -store kvstore2 \ -table users -helper-hosts :5000 \

9-62 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

-config import_config -format JSON \ -status /users/oracle/checkpoint_dir

The import_config file resides in the current working directory and the file content is as below:

{ "path": "/users/oracle/kvstore_export" }

Steps using NoSQL Database Migrator Utility

1. Prepare the configuration file with the correct source and sink details. Here source is the first NoSQL Database store kvstore1. The table users will be exported. The sink is another NoSQL Database store kvstore2. You specify the absolute path to a file containing DDL statements for creating the NoSQL table in the schemaPath parameter. The NoSQL Database Migrator executes the DDL commands listed in this file before migrating the data. The userSchema file has the following DDL statement.

CREATE TABLE IF NOT EXISTS users (id INTEGER, firstName STRING, lastName STRING, otherNames ARRAY(RECORD(first STRING, last STRING)), age INTEGER, income INTEGER, address JSON, connections ARRAY(INTEGER), expenses MAP(INTEGER), PRIMARY KEY(SHARD(id)))

The NoSQL Database Migrator configuration file is given below.

{ "source" : { "type" : "nosqldb", "storeName" : "kvstore1", "helperHosts" : [":5000"], "table" : "users" }, "sink" : { "type" : "nosqldb", "table" : "users", "schemaInfo" : { "schemaPath" : "/users/oracle/export_dir/kvstore_export/ usersSchema" }, "storeName" : "kvstore2", "helperHosts" : [":5000"], "requestTimeoutMs" : 5000 }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

9-63 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

2. Run the runMigrator command by passing the configuration JSON file using the --config or -c option.

./runMigrator --config /users/oracle/migrator_config/store1tostore2.config

Example 2: You have a script that exports a table every night in binary format. You import this backup when there is a failure and restore the table to a previous night's version A) Steps using Import/Export Utility 1. Export the contents from the table users in kvstore1 and place the export package into the /users/oracle/kvstore_export directory. The default format is binary and so no format is specified.

java -jar lib/kvtool.jar export -table users \ -store kvstore1 -helper-hosts :5000 \ -config export_config

The export_config file resides in the current working directory and the file content is as below:

{ "path": "/users/oracle/kvstore_export" }

Directories are created in the kvstore_export directory for data and schema definition. 2. Whenever there is a failure , use the following import command to restore the table to a previous night's version.

java -jar lib/kvtool.jar import -store kvstore1 \ -table users -helper-hosts :5000 \ -config import_config

The import_config file resides in the current working directory and the file content is as below:

{ "path": "/users/oracle/kvstore_export" }

B) Steps using NoSQL Database Migrator Utility The NoSQL Database Migrator utility does not support binary format. Only JSON format is supported. To use the NoSQL Database Migrator utility here, you need to create two configuration files . The first config file will take the Oracle NoSQL datastore kvstore1 as the source and the sink will be a JSON file. You will run this config file every night to backup the table into a JSON file. The second config file will take the JSON file created above as the source and the sink will be the Oracle NoSQL Database store kvstore1. This config file will be run only when there is a failure and there is a need to restore the table to the previous nights version.

9-64 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

1. Create the configuration file "nightlyusersbackup" to do the nightly backup.

{ "source" : { "type" : "nosqldb", "storeName" : "kvstore1", "helperHosts" : [":5000"], "table" : "users" }, "sink" : { "type" : "file", "format" : "json", "dataPath": "/users/oracle/export_dir/kvstore_export/usersJSON", "schemaPath" : "/users/oracle/export_dir/kvstore_export/usersSchema" }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

2. Run the runMigrator command every night by passing the configuration JSON file using the --config or -c option.

./runMigrator --config /users/oracle/migrator_config/nightlyusersbackup.config

3. Create the configuration file "usersrecovery" to recover the users table to the previous nights version. Here the source is the json file that is created in the nightly backup and sink is the Oracle NoSQL datastore kvstore1.

{ "source" : { "type" : "file", "format" : "json", "dataPath": "/users/oracle/export_dir/kvstore_export/usersJSON", "schemaPath" : "/users/oracle/export_dir/kvstore_export/usersSchema" }, "sink" : { "type" : "nosqldb", "table" : "users", "schemaInfo" : { "schemaPath" : "/users/oracle/export_dir/kvstore_export/ usersSchema" }, "storeName" : "kvstore1", "helperHosts" : [":5000"], "requestTimeoutMs" : 5000 }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

9-65 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

4. Whenever there is a failure and the table users needs to be recovered to the previous night's version, run the runMigrator command by passing the configuration JSON file created above using the --config or -c option.

./runMigrator --config /users/oracle/migrator_config/usersrecovery.config

Example 3: - You have a script that runs nightly to import a directory of JSON files A)Using Import Utility Import the data from the export package already created from an external source into the Oracle NoSQL Database store. Specify the format as JSON. You specify the parameter -external as the data to import has been generated by a source other than Oracle NoSQL Database. Use checkpoints to be able to restart the import from where it stopped or got aborted, in case it fails. java -jar lib/kvtool.jar import -external -store kvstore \ -helper-hosts :5000 -config import_config \ -status /users/oracle/checkpoint_dir -format JSON

The import_config file resides in the current working directory and the file content is as below:

{ "path": "/users/oracle/kvstore_export" }

B) Steps using NoSQL Database Migrator Utility

1. Prepare the configuration file "import_json.config" with the correct source and sink details. Here source is the list of json files that needs to be imported. In the dataPath parameter, specify a directory which contains all the json files. The NoSQL Database Migrator identifies all the files with the .json extension in that directory for the migration. The sink is a NoSQL Database store where the files will be imported into a given table. All the json files are imported into the given table in the Oracle NoSQL Database store. The userSchema file has the following DDL statement.

CREATE TABLE IF NOT EXISTS users (id INTEGER, firstName STRING, lastName STRING, otherNames ARRAY(RECORD(first STRING, last STRING)), age INTEGER, income INTEGER,address JSON, connections ARRAY(INTEGER), expenses MAP(INTEGER), PRIMARY KEY(SHARD(id)))

The NoSQL Database Migrator configuration file is given below.

{ "source" : { "type" : "file", "format" : "json", "dataPath": "/users/oracle/dir_jsonfiles"

9-66 Chapter 9 Oracle NoSQL Database Migrator Vs. Import/Export Utility

}, "sink" : { "type" : "nosqldb", "table" : "users", "schemaInfo" : { "schemaPath" : "/users/oracle/export_dir/kvstore_export/ usersSchema" }, "storeName" : "kvstore", "helperHosts" : [":5000"] }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

2. Run the runMigrator command by passing the configuration JSON file using the --config or -c option.

./runMigrator --config /users/oracle/migrator_config/import_json.config

Example 4: - Import data from a MongoDB export into a KVSTORE A) Steps using Import/Export Utility You can import data files that contain data from a MongoDB export. The parameter - external specifies that the data to import has been generated by a source other than Oracle NoSQL Database. Hence, this flag indicates that the directory structure being read by the import utility is not in the "export package" format. Specify the format as MONGODB_JSON. java -jar lib/kvtool.jar import -external -store kvstore \ -helper-hosts :5000 -config import_config \ -format MONGODB_JSON

The import_config file resides in the current working directory and the file content is as below. Store all of the errors and progress information for the import in /users/oracle/ mongo_db_import_logs. You can specify the DDL statement to be used for creating tables in the ddlSchemaFile parameter.

{ "path": "/users/oracle/my_mongodb_data", "errorOutput": "/users/oracle/mongo_db_import_logs" "ddlSchemaFile" : "/users/oracle/export_dir/kvstore_export/usersSchema" }

B) Steps using NoSQL Database Migrator Utility

1. Prepare the configuration file "import_mongodbjson.config" with the correct source and sink details. The source is MongoDB-formatted JSON File and the sink is Oracle NoSQL Database store. You specify the absolute path to a file containing the MongoDB exported

9-67 Chapter 9 Transitioning from Import/Export to NoSQL Database Migrator

JSON data for migration. You must ensure that this data matches with the NoSQL table schema defined at the sink.

{ "source" : { "type" : "file", "format" : "mongodb_json", "dataPath": "/users/oracle/my_mongodb_data" }, "sink" : { "type" : "nosqldb", "table" : "users", "schemaInfo" : { "schemaPath" : "/users/oracle/export_dir/kvstore_export/ usersSchema" }, "storeName" : "kvstore", "helperHosts" : [":5000"] }, "abortOnError" : true, "migratorVersion" : "1.0.0" }

2. Run the runMigrator command by passing the configuration JSON file using the --config or -c option.

./runMigrator --config /users/oracle/migrator_config/import_mongodbjson.config

Transitioning from Import/Export to NoSQL Database Migrator

Learn how to transition from the import/export utility to the Oracle NoSQL Database Migrator. If you have been using the import/export utility for data migration you would be aware that the kvtool.jar that is part of the Oracle NoSQL Database (on-premise) handles export and import functionality with the help of two commands, export and import. You can achieve the same export and import operations using the NoSQL Database Migrator, as explained below. Equivalent of the export command in NoSQL Database Migrator

The export command is used to export the NoSQL Database (on-premise) data to a local or network mounted file system. You can achieve the equivalent operation by defining the following attributes in the Configuration JSON file as follows: • "source"="nosqldb" • "sink"="file" with "format"="JSON" The other configuration attributes can be defined as per the requirement. As the NoSQL Database Migrator supports migrating only one table at a time, you must run the NoSQL Database Migrator separately for exporting multiple tables.

9-68 Chapter 9 Transitioning from Import/Export to NoSQL Database Migrator

Equivalent of the import command in NoSQL Database Migrator

The import command is used to import a package created by the export command, an external JSON file, or a MongoDB-formatted JSON file. You can achieve the equivalent operation by defining the following attributes in the Configuration JSON file as follows: • "source"="file" with "format"="json" or "format"="mongodb_json" depending on the source file type. • "sink"="nosqldb" Note:The NoSQL Database Migrator treats an external JSON file or a package that was generated using the existing import/export utility similarly.

9-69