DOCSLIB.ORG

AWS Schema Conversion Tool User Guide Version 1.0 AWS Schema Conversion Tool User Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
AWS Schema Conversion Tool User Guide Version 1.0 AWS Schema Conversion Tool User Guide AWS Schema Conversion Tool User Guide Version 1.0 AWS Schema Conversion Tool User Guide AWS Schema Conversion Tool: User Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. AWS Schema Conversion Tool User Guide Table of Contents What Is the AWS SCT? ........................................................................................................................ 1 Schema conversion overview ....................................................................................................... 2 Giving feedback ......................................................................................................................... 2 Installing, verifying, and updating ........................................................................................................ 4 Installing AWS SCT ..................................................................................................................... 4 Installing previous versions .................................................................................................. 5 Verifying the AWS SCT file download .......................................................................................... 5 Verifying the checksum of the AWS SCT file .......................................................................... 5 Verifying the AWS SCT RPM files on Fedora .......................................................................... 6 Verifying the AWS SCT DEB files on Ubuntu .......................................................................... 6 Verifying the AWS SCT MSI file on Microsoft Windows ............................................................ 7 Installing the required database drivers ........................................................................................ 7 Installing JDBC drivers on Linux ........................................................................................... 9 Storing driver paths in the global settings ............................................................................. 9 Updating the AWS SCT ............................................................................................................. 10 Using the AWS SCT user interface ...................................................................................................... 11 The project window .................................................................................................................. 11 Storing AWS Profiles ................................................................................................................. 12 Storing AWS credentials .................................................................................................... 13 Setting the default profile for a project .............................................................................. 15 Storing database passwords ...................................................................................................... 15 Using the Union All view for projects with partitioned tables ......................................................... 15 Using tree filters ...................................................................................................................... 15 ...................................................................................................................................... 16 Importing a file list for the tree filter ................................................................................. 17 Hiding schemas ........................................................................................................................ 18 Keyboard shortcuts ................................................................................................................... 18 Managing the database migration assessment report .................................................................... 19 Starting AWS SCT ..................................................................................................................... 23 Creating a project ..................................................................................................................... 24 Converting your schema ............................................................................................................ 25 Applying the converted schema to your target DB instance ............................................................ 26 Getting started ................................................................................................................................ 28 Sources for AWS SCT ........................................................................................................................ 29 Encrypting Amazon RDS connections .......................................................................................... 29 Using Oracle as a source ........................................................................................................... 31 Permissions needed for Oracle as a source ......................................................................... 32 Connecting to Oracle as a source ....................................................................................... 32 Oracle to PostgreSQL ....................................................................................................... 34 Oracle to MySQL .............................................................................................................. 37 Oracle to Amazon RDS Oracle ........................................................................................... 41 Using SQL Server as a source .................................................................................................... 44 Permissions for Microsoft SQL Server ................................................................................. 44 Using Windows Authentication with Microsoft SQL Server ..................................................... 44 Connecting to SQL Server as a source ................................................................................. 46 SQL Server to MySQL ....................................................................................................... 47 SQL Server to PostgreSQL ................................................................................................. 49 SQL Server to Amazon RDS SQL Server .............................................................................. 52 Using MySQL as a source .......................................................................................................... 53 Permissions needed for MySQL as a source .......................................................................... 53 Connecting to MySQL as a source ...................................................................................... 53 Using PostgreSQL as a source .................................................................................................... 55 Privileges for PostgreSQL ................................................................................................. 55 Connecting to PostgreSQL as a source ............................................................................... 55 Version 1.0 iii AWS Schema Conversion Tool User Guide Using SAP ASE (Sybase ASE) as a source ..................................................................................... 57 Privileges for SAP ASE (Sybase ASE) .................................................................................. 58 Connecting to SAP ASE as a source ................................................................................... 58 Data warehouse sources for AWS SCT ......................................................................................... 61 Using IBM Db2 LUW as a source ........................................................................................ 62 Using Amazon Redshift as a source .................................................................................... 66 Using Oracle Data Warehouse as a source ........................................................................... 68 Using Teradata as a source ................................................................................................ 71 Using Netezza as a source ................................................................................................. 73 Using Greenplum as a source ............................................................................................. 75 Using Vertica as a source .................................................................................................. 77 Using SQL Server Data Warehouse as a Source .................................................................... 79 Using Snowflake as a source .............................................................................................. 81 Creating conversion reports ............................................................................................................... 86 Migration assessment reports .................................................................................................... 86 Creating
Load more

Recommended publications
  • Database Concepts 7Th Edition
    Database Concepts 7th Edition David M. Kroenke • David J. Auer Online Appendix E SQL Views, SQL/PSM and Importing Data Database Concepts SQL Views, SQL/PSM and Importing Data Appendix E All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America. Appendix E — 10 9 8 7 6 5 4 3 2 1 E-2 Database Concepts SQL Views, SQL/PSM and Importing Data Appendix E Appendix Objectives • To understand the reasons for using SQL views • To use SQL statements to create and query SQL views • To understand SQL/Persistent Stored Modules (SQL/PSM) • To create and use SQL user-defined functions • To import Microsoft Excel worksheet data into a database What is the Purpose of this Appendix? In Chapter 3, we discussed SQL in depth. We discussed two basic categories of SQL statements: data definition language (DDL) statements, which are used for creating tables, relationships, and other structures, and data manipulation language (DML) statements, which are used for querying and modifying data. In this appendix, which should be studied immediately after Chapter 3, we: • Describe and illustrate SQL views, which extend the DML capabilities of SQL. • Describe and illustrate SQL Persistent Stored Modules (SQL/PSM), and create user-defined functions. • Describe and use DBMS data import techniques to import Microsoft Excel worksheet data into a database. E-3 Database Concepts SQL Views, SQL/PSM and Importing Data Appendix E Creating SQL Views An SQL view is a virtual table that is constructed from other tables or views.
    [Show full text]
  • SQL Standards Update 1
    2017-10-20 SQL Standards Update 1 SQL STANDARDS UPDATE Keith W. Hare SC32 WG3 Convenor JCC Consulting, Inc. October 20, 2017 2017-10-20 SQL Standards Update 2 Introduction • What is SQL? • Who Develops the SQL Standards • A brief history • SQL 2016 Published • SQL Technical Reports • What's next? • SQL/MDA • Streaming SQL • Property Graphs • Summary 2017-10-20 SQL Standards Update 3 Who am I? • Senior Consultant with JCC Consulting, Inc. since 1985 • High performance database systems • Replicating data between database systems • SQL Standards committees since 1988 • Convenor, ISO/IEC JTC1 SC32 WG3 since 2005 • Vice Chair, ANSI INCITS DM32.2 since 2003 • Vice Chair, INCITS Big Data Technical Committee since 2015 • Education • Muskingum College, 1980, BS in Biology and Computer Science • Ohio State, 1985, Masters in Computer & Information Science 2017-10-20 SQL Standards Update 4 What is SQL? • SQL is a language for defining databases and manipulating the data in those databases • SQL Standard uses SQL as a name, not an acronym • Might stand for SQL Query Language • SQL queries are independent of how the data is actually stored – specify what data you want, not how to get it 2017-10-20 SQL Standards Update 5 Who Develops the SQL Standards? In the international arena, the SQL Standard is developed by ISO/ IEC JTC1 SC32 WG3. • Officers: • Convenor – Keith W. Hare – USA • Editor – Jim Melton – USA • Active participants are: • Canada – Standards Council of Canada • China – Chinese Electronics Standardization Institute • Germany – DIN Deutsches
    [Show full text]
  • Simba Amazon Redshift ODBC Driver
    Simba Amazon Redshift ODBC Driver Installation and Configuration Guide Simba Technologies Inc. Version 1.4.3 September 14, 2018 Installation and Configuration Guide Copyright © 2018 Simba Technologies Inc. All Rights Reserved. Information in this document is subject to change without notice. Companies, names and data used in examples herein are fictitious unless otherwise noted. No part of this publication, or the software it describes, may be reproduced, transmitted, transcribed, stored in a retrieval system, decompiled, disassembled, reverse-engineered, or translated into any language in any form by any means for any purpose without the express written permission of Simba Technologies Inc. Trademarks Simba, the Simba logo, SimbaEngine, and Simba Technologies are registered trademarks of Simba Technologies Inc. in Canada, United States and/or other countries. All other trademarks and/or servicemarks are the property of their respective owners. Contact Us Simba Technologies Inc. 938 West 8th Avenue Vancouver, BC Canada V5Z 1E5 Tel: +1 (604) 633-0008 Fax: +1 (604) 633-0004 www.simba.com www.simba.com 2 Installation and Configuration Guide About This Guide Purpose The Simba Amazon Redshift ODBC Driver Installation and Configuration Guide explains how to install and configure the Simba Amazon Redshift ODBC Driver. The guide also provides details related to features of the driver. Audience The guide is intended for end users of the Simba Amazon Redshift ODBC Driver, as well as administrators and developers integrating the driver. Knowledge
    [Show full text]
  • Scalable Computation of Acyclic Joins
    Scalable Computation of Acyclic Joins (Extended Abstract) Anna Pagh Rasmus Pagh [email protected] [email protected] IT University of Copenhagen Rued Langgaards Vej 7, 2300 København S Denmark ABSTRACT 1. INTRODUCTION The join operation of relational algebra is a cornerstone of The relational model and relational algebra, due to Edgar relational database systems. Computing the join of several F. Codd [3] underlies the majority of today’s database man- relations is NP-hard in general, whereas special (and typi- agement systems. Essential to the ability to express queries cal) cases are tractable. This paper considers joins having in relational algebra is the natural join operation, and its an acyclic join graph, for which current methods initially variants. In a typical relational algebra expression there will apply a full reducer to efficiently eliminate tuples that will be a number of joins. Determining how to compute these not contribute to the result of the join. From a worst-case joins in a database management system is at the heart of perspective, previous algorithms for computing an acyclic query optimization, a long-standing and active field of de- join of k fully reduced relations, occupying a total of n ≥ k velopment in academic and industrial database research. A blocks on disk, use Ω((n + z)k) I/Os, where z is the size of very challenging case, and the topic of most database re- the join result in blocks. search, is when data is so large that it needs to reside on In this paper we show how to compute the join in a time secondary memory.
    [Show full text]
  • Secure Data Commons
    Secure Data Commons Data Analyst User Guide www.its.dot.gov/index.htm U.S. Department of Transportation Notice This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for its contents or use thereof. The U.S. Government is not endorsing any manufacturers, products, or services cited herein and any trade name that may appear in the work has been included only because it is essential to the contents of the work. Revision History # Name Version Revision Date Revision Description 1 REAN Cloud 1.0 08/02/2018 Initial Draft 2 REAN Cloud 2.0 09/13/2018 Added the Export Functionality 3 REAN Cloud 3.0 10/29/2019 Added the Manage Workstations chapter and updated the Guide per feedback comments 4 SDC Team 4.0 02/06/2020 • Added description of data ingestion and curation to Chapter 1 • Added description and link to instructions for setting up and logging in with Login.gov in Chapter 2 • Added note and example command to the “Download User Data from S3 Bucket through Portal” section under Chapter 2 • Added FMI section and link to GitLab page with more sample queries to Chapter 4 Sample Queries for SDC Datasets • Added note about printing documents in Chapter 5 • Added Chapter 6 Setting Up SDC with GitLab Repositories • Added more commands to the “AWS S3 CLI Commands” section under Chapter 7 5 SDC Team 5.0 04/13/2020 • Added “Connecting to Waze Data in Redshift Using Python” section to Chapter 3 Accessing and Launching Workstations
    [Show full text]
  • Sql Server to Aurora Postgresql Migration Playbook
    Microsoft SQL Server To Amazon Aurora with Post- greSQL Compatibility Migration Playbook 1.0 Preliminary September 2018 © 2018 Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document is provided for informational purposes only. It represents AWS’s current product offer- ings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without war- ranty of any kind, whether express or implied. This document does not create any warranties, rep- resentations, contractual commitments, conditions or assurances from AWS, its affiliates, suppliers or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agree- ments, and this document is not part of, nor does it modify, any agreement between AWS and its cus- tomers. - 2 - Table of Contents Introduction 9 Tables of Feature Compatibility 12 AWS Schema and Data Migration Tools 20 AWS Schema Conversion Tool (SCT) 21 Overview 21 Migrating a Database 21 SCT Action Code Index 31 Creating Tables 32 Data Types 32 Collations 33 PIVOT and UNPIVOT 33 TOP and FETCH 34 Cursors 34 Flow Control 35 Transaction Isolation 35 Stored Procedures 36 Triggers 36 MERGE 37 Query hints and plan guides 37 Full Text Search 38 Indexes 38 Partitioning 39 Backup 40 SQL Server Mail 40 SQL Server Agent 41 Service Broker 41 XML 42 Constraints
    [Show full text]
  • SQL Version Analysis
    Rory McGann SQL Version Analysis Structured Query Language, or SQL, is a powerful tool for interacting with and utilizing databases through the use of relational algebra and calculus, allowing for efficient and effective manipulation and analysis of data within databases. There have been many revisions of SQL, some minor and others major, since its standardization by ANSI in 1986, and in this paper I will discuss several of the changes that led to improved usefulness of the language. In 1970, Dr. E. F. Codd published a paper in the Association of Computer Machinery titled A Relational Model of Data for Large shared Data Banks, which detailed a model for Relational database Management systems (RDBMS) [1]. In order to make use of this model, a language was needed to manage the data stored in these RDBMSs. In the early 1970’s SQL was developed by Donald Chamberlin and Raymond Boyce at IBM, accomplishing this goal. In 1986 SQL was standardized by the American National Standards Institute as SQL-86 and also by The International Organization for Standardization in 1987. The structure of SQL-86 was largely similar to SQL as we know it today with functionality being implemented though Data Manipulation Language (DML), which defines verbs such as select, insert into, update, and delete that are used to query or change the contents of a database. SQL-86 defined two ways to process a DML, direct processing where actual SQL commands are used, and embedded SQL where SQL statements are embedded within programs written in other languages. SQL-86 supported Cobol, Fortran, Pascal and PL/1.
    [Show full text]
  • Sql Merge Performance on Very Large Tables
    Sql Merge Performance On Very Large Tables CosmoKlephtic prologuizes Tobie rationalised, his Poole. his Yanaton sloughing overexposing farrow kibble her pausingly. game steeply, Loth and bound schismatic and incoercible. Marcel never danced stagily when Used by Google Analytics to track your activity on a website. One problem is caused by the increased number of permutations that the optimizer must consider. Much to maintain for very large tables on sql merge performance! The real issue is how to write or remove files in such a way that it does not impact current running queries that are accessing the old files. Also, the performance of the MERGE statement greatly depends on the proper indexes being used to match both the source and the target tables. This is used when the join optimizer chooses to read the tables in an inefficient order. Once a table is created, its storage policy cannot be changed. Make sure that you have indexes on the fields that are in your WHERE statements and ON conditions, primary keys are indexed by default but you can also create indexes manually if you have to. It will allow the DBA to create them on a staging table before switching in into the master table. This means the engine must follow the join order you provided on the query, which might be better than the optimized one. Should I split up the data to load iit faster or use a different structure? Are individual queries faster than joins, or: Should I try to squeeze every info I want on the client side into one SELECT statement or just use as many as seems convenient? If a dashboard uses auto refresh, make sure it refreshes no faster than the ETL processes running behind the scenes.
    [Show full text]
  • “A Relational Model of Data for Large Shared Data Banks”
    “A RELATIONAL MODEL OF DATA FOR LARGE SHARED DATA BANKS” Through the internet, I find more information about Edgar F. Codd. He is a mathematician and computer scientist who laid the theoretical foundation for relational databases--the standard method by which information is organized in and retrieved from computers. In 1981, he received the A. M. Turing Award, the highest honor in the computer science field for his fundamental and continuing contributions to the theory and practice of database management systems. This paper is concerned with the application of elementary relation theory to systems which provide shared access to large banks of formatted data. It is divided into two sections. In section 1, a relational model of data is proposed as a basis for protecting users of formatted data systems from the potentially disruptive changes in data representation caused by growth in the data bank and changes in traffic. A normal form for the time-varying collection of relationships is introduced. In Section 2, certain operations on relations are discussed and applied to the problems of redundancy and consistency in the user's model. Relational model provides a means of describing data with its natural structure only--that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other. A further advantage of the relational view is that it forms a sound basis for treating derivability, redundancy, and consistency of relations.
    [Show full text]
  • Firebird SQL Best Practices
    Firebird SQL best practices Firebird SQL best practices Review of some SQL features available and that people often forget about Author: Philippe Makowski IBPhoenix Email: [email protected] Licence: Public Documentation License Date: 2016-09-29 Philippe Makowski - IBPhoenix - 2016-09-29 Firebird SQL best practices Common table expression Syntax WITH [RECURSIVE] -- new keywords CTE_A -- first table expression’s name [(a1, a2, ...)] -- fields aliases, optional AS ( SELECT ... ), -- table expression’s definition CTE_B -- second table expression [(b1, b2, ...)] AS ( SELECT ... ), ... SELECT ... -- main query, used both FROM CTE_A, CTE_B, -- table expressions TAB1, TAB2 -- and regular tables WHERE ... Philippe Makowski - IBPhoenix - 2016-09-29 Firebird SQL best practices Emulate loose index scan The term "loose indexscan" is used in some other databases for the operation of using a btree index to retrieve the distinct values of a column efficiently; rather than scanning all equal values of a key, as soon as a new value is found, restart the search by looking for a larger value. This is much faster when the index has many equal keys. A table with 10,000,000 rows, and only 3 differents values in row. CREATE TABLE HASH ( ID INTEGER NOT NULL, SMALLDISTINCT SMALLINT, PRIMARY KEY (ID) ); CREATE ASC INDEX SMALLDISTINCT_IDX ON HASH (SMALLDISTINCT); Philippe Makowski - IBPhoenix - 2016-09-29 Firebird SQL best practices Without CTE : SELECT DISTINCT SMALLDISTINCT FROM HASH SMALLDISTINCT ============= 0 1 2 PLAN SORT ((HASH NATURAL)) Prepared in
    [Show full text]
  • Join , Sub Queries and Set Operators Obtaining Data from Multiple Tables
    Join , Sub queries and set operators Obtaining Data from Multiple Tables EMPLOYEES DEPARTMENTS … … Cartesian Products – A Cartesian product is formed when: • A join condition is omitted • A join condition is invalid • All rows in the first table are joined to all rows in the second table – To avoid a Cartesian product, always include a valid join condition in a WHERE clause. Generating a Cartesian Product EMPLOYEES (20 rows) DEPARTMENTS (8 rows) … Cartesian product: 20 x 8 = 160 rows … Types of Oracle-Proprietary Joins – Equijoin – Nonequijoin – Outer join – Self-join Joining Tables Using Oracle Syntax • Use a join to query data from more than one table: SELECT table1.column, table2.column FROM table1, table2 WHERE table1.column1 = table2.column2; – Write the join condition in the WHERE clause. – Prefix the column name with the table name when the same column name appears in more than one table. Qualifying Ambiguous Column Names – Use table prefixes to qualify column names that are in multiple tables. – Use table prefixes to improve performance. – Instead of full table name prefixes, use table aliases. – Table aliases give a table a shorter name. • Keeps SQL code smaller, uses less memory – Use column aliases to distinguish columns that have identical names, but reside in different tables. Equijoins EMPLOYEES DEPARTMENTS Primary key … Foreign key Retrieving Records with Equijoins SELECT e.employee_id, e.last_name, e.department_id, d.department_id, d.location_id FROM employees e, departments d WHERE e.department_id = d.department_id; … Retrieving Records with Equijoins: Example SELECT d.department_id, d.department_name, d.location_id, l.city FROM departments d, locations l WHERE d.location_id = l.location_id; Additional Search Conditions Using the AND Operator SELECT d.department_id, d.department_name, l.city FROM departments d, locations l WHERE d.location_id = l.location_id AND d.department_id IN (20, 50); Joining More than Two Tables EMPLOYEES DEPARTMENTS LOCATIONS … • To join n tables together, you need a minimum of n–1 • join conditions.
    [Show full text]
  • SQL: Programming Introduction to Databases Compsci 316 Fall 2019 2 Announcements (Mon., Sep
    SQL: Programming Introduction to Databases CompSci 316 Fall 2019 2 Announcements (Mon., Sep. 30) • Please fill out the RATest survey (1 free pt on midterm) • Gradiance SQL Recursion exercise assigned • Homework 2 + Gradiance SQL Constraints due tonight! • Wednesday • Midterm in class • Open-book, open-notes • Same format as sample midterm (posted in Sakai) • Gradiance SQL Triggers/Views due • After fall break • Project milestone 1 due; remember members.txt • Gradiance SQL Recursion due 3 Motivation • Pros and cons of SQL • Very high-level, possible to optimize • Not intended for general-purpose computation • Solutions • Augment SQL with constructs from general-purpose programming languages • E.g.: SQL/PSM • Use SQL together with general-purpose programming languages: many possibilities • Through an API, e.g., Python psycopg2 • Embedded SQL, e.g., in C • Automatic obJect-relational mapping, e.g.: Python SQLAlchemy • Extending programming languages with SQL-like constructs, e.g.: LINQ 4 An “impedance mismatch” • SQL operates on a set of records at a time • Typical low-level general-purpose programming languages operate on one record at a time • Less of an issue for functional programming languages FSolution: cursor • Open (a result table): position the cursor before the first row • Get next: move the cursor to the next row and return that row; raise a flag if there is no such row • Close: clean up and release DBMS resources FFound in virtually every database language/API • With slightly different syntaxes FSome support more positioning and movement options, modification at the current position, etc. 5 Augmenting SQL: SQL/PSM • PSM = Persistent Stored Modules • CREATE PROCEDURE proc_name(param_decls) local_decls proc_body; • CREATE FUNCTION func_name(param_decls) RETURNS return_type local_decls func_body; • CALL proc_name(params); • Inside procedure body: SET variable = CALL func_name(params); 6 SQL/PSM example CREATE FUNCTION SetMaxPop(IN newMaxPop FLOAT) RETURNS INT -- Enforce newMaxPop; return # rows modified.
    [Show full text]