AAlliibbaabbaa CClloouudd Alibaba Cloud
DDaattaa O Onlninlein Mei gMraitgiornation MMigigrraatte ddaatat afr ofmro Gmoo Ggloeogle Cloud Storage to OSS Cloud Storage to OSS
Document Version: 20210729
Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Legal disclaimer
Legal disclaimer
Alibaba Cloud reminds you t o carefully read and fully underst and t he t erms and condit ions of t his legal disclaimer before you read or use t his document . If you have read or used t his document , it shall be deemed as your t ot al accept ance of t his legal disclaimer.
1. You shall download and obt ain t his document from t he Alibaba Cloud websit e or ot her Alibaba Cloud- aut horized channels, and use t his document for your own legal business act ivit ies only. The cont ent of t his document is considered confident ial informat ion of Alibaba Cloud. You shall st rict ly abide by t he confident ialit y obligat ions. No part of t his document shall be disclosed or provided t o any t hird part y for use wit hout t he prior writ t en consent of Alibaba Cloud.
2. No part of t his document shall be excerpt ed, t ranslat ed, reproduced, t ransmit t ed, or disseminat ed by any organizat ion, company or individual in any form or by any means wit hout t he prior writ t en consent of Alibaba Cloud.
3. The cont ent of t his document may be changed because of product version upgrade, adjust ment , or ot her reasons. Alibaba Cloud reserves t he right t o modify t he cont ent of t his document wit hout not ice and an updat ed version of t his document will be released t hrough Alibaba Cloud-aut horized channels from t ime t o t ime. You should pay at t ent ion t o t he version changes of t his document as t hey occur and download and obt ain t he most up-t o-dat e version of t his document from Alibaba Cloud-aut horized channels.
4. This document serves only as a reference guide for your use of Alibaba Cloud product s and services. Alibaba Cloud provides t his document based on t he "st at us quo", "being defect ive", and "exist ing funct ions" of it s product s and services. Alibaba Cloud makes every effort t o provide relevant operat ional guidance based on exist ing t echnologies. However, Alibaba Cloud hereby makes a clear st at ement t hat it in no way guarant ees t he accuracy, int egrit y, applicabilit y, and reliabilit y of t he cont ent of t his document , eit her explicit ly or implicit ly. Alibaba Cloud shall not t ake legal responsibilit y for any errors or lost profit s incurred by any organizat ion, company, or individual arising from download, use, or t rust in t his document . Alibaba Cloud shall not , under any circumst ances, t ake responsibilit y for any indirect , consequent ial, punit ive, cont ingent , special, or punit ive damages, including lost profit s arising from t he use or t rust in t his document (even if Alibaba Cloud has been not ified of t he possibilit y of such a loss).
5. By law, all t he cont ent s in Alibaba Cloud document s, including but not limit ed t o pict ures, archit ect ure design, page layout , and t ext descript ion, are int ellect ual propert y of Alibaba Cloud and/or it s affiliat es. This int ellect ual propert y includes, but is not limit ed t o, t rademark right s, pat ent right s, copyright s, and t rade secret s. No part of t his document shall be used, modified, reproduced, publicly t ransmit t ed, changed, disseminat ed, dist ribut ed, or published wit hout t he prior writ t en consent of Alibaba Cloud and/or it s affiliat es. The names owned by Alibaba Cloud shall not be used, published, or reproduced for market ing, advert ising, promot ion, or ot her purposes wit hout t he prior writ t en consent of Alibaba Cloud. The names owned by Alibaba Cloud include, but are not limit ed t o, "Alibaba Cloud", "Aliyun", "HiChina", and ot her brands of Alibaba Cloud and/or it s affiliat es, which appear separat ely or in combinat ion, as well as t he auxiliary signs and pat t erns of t he preceding brands, or anyt hing similar t o t he company names, t rade names, t rademarks, product or service names, domain names, pat t erns, logos, marks, signs, or special descript ions t hat t hird part ies ident ify as Alibaba Cloud and/or it s affiliat es.
6. Please direct ly cont act Alibaba Cloud for any errors of t his document .
> Document Version: 20210729 I Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Document convent ion s Document conventions
St yle Descript ion Example
A danger notice indicates a situation that Danger: Danger will cause major system changes, faults, Resetting will result in the loss of user physical injuries, and other adverse configuration data. results.
Warning: A warning notice indicates a situation Warning that may cause major system changes, Restarting will cause business faults, physical injuries, and other adverse interruption. About 10 minutes are results. required to restart an instance.
A caution notice indicates warning Not ice: Not ice information, supplementary instructions, If the weight is set to 0, the server no and other content that the user must longer receives new requests. understand.
A note indicates supplemental Not e: Not e instructions, best practices, tips, and You can use Ctrl + A to select all files. other content.
Closing angle brackets are used to Click Set t ings > Net work> Set net work > indicate a multi-level menu cascade. t ype.
Bold formatting is used for buttons , Bold menus, page names, and other UI Click OK. elements.
Run the cd /d C:/window command to Courier font Courier font is used for commands enter the Windows system folder.
Italic formatting is used for parameters bae log list --instanceid Italic and variables. Instance_ID
This format is used for an optional value, [] or [a|b] ipconfig [-all|-t] where only one item can be selected.
This format is used for a required value, {} or {a|b} switch {active|stand} where only one item can be selected.
> Document Version: 20210729 I Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Table of Cont ent s Table of Contents
1.Background information 05
2.Prepare for a migration 06
3.Implement migration 08
4.Manage migration jobs 15
> Document Version: 20210729 I Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Background informat i on 1.Background information This tutorial describes how to migrate data from Google Cloud Storage to Alibaba Cloud Object Storage Service (OSS). Alibaba Cloud Data Online Migration is a data channel between different storage services. You can use Data Online Migration to migrate data from third-party data stores to OSS or between OSS buckets. When you use Data Online Migration, you need only to log on to the console, specify a source data address and a destination OSS data address, and then create a migration job. After the migration job starts, you can manage the job in the console. For example, you can view the progress and traffic information of the job. You can also generate a migration report to check the list of migrated files and the list of files that fail to be migrated.
Not ice When you read data from the source data address during a migration job, a fee is incurred for the outbound Internet traffic. You are charged by the storage service provider of the source data address. By default, Data Online Migration does not support cross-border data migration. For example, you cannot migrate data from a bucket that is deployed in the China (Beijing) region to a bucket that is deployed in the US (Silicon Valley) region. Before you create a migration job to migrate data across borders, you must submit a ticket to apply for the permissions that are required for creating cross-border migration jobs. You must commit that your business is legitimate, data transit conforms to local rules and regulations, and your data does not contain illegal information.
This tutorial includes the following topics: Preparations Implement migration Subsequent operations
> Document Version: 20210729 5 Migrat e dat a from Google Cloud St orage t o OSS·Prepare for a migrat io Dat a Online Migrat ion n 2.Prepare for a migration This topic describes how to prepare for a migration.
Google Cloud Platform
Estimate the amount of data to be migrated Estimate the size and the number of files that you want to migrate. You can use the gsutil tool or storage logs to view the capacity of a bucket. For more information, see Getting bucket information.
Not e You must enter an appropriate size and number of files when you create a migration job.
Restore the data in the bucket To migrate data of the Archive storage class, you must restore the data before you create a migration job. Otherwise, the files that are not restored or being restored fail to be migrated. This is because Data Online Migration does not support the restore operation of archived files. For more information about how to restore archived data, see Solutions to migrate archived data.
Not e Before you create a source data address and a migration job, make sure the Archive data is restored. The time required to restore Archive data is based on the data volume. Therefore, you must estimate and specify a restoration duration so that data does not enter the archived state. You are charged when you restore archived files. For information about the billing methods, contact the service provider that offers the source data address.
Create a private key of a service account that is used for migration. i. Log on to the IAM & Admin console. ii. In the left-side navigation pane, click Service Account s. Click the Select a project drop-down list on the top navigation bar. In the dialog box that appears, select the project in which the bucket resides. iii. Click CREAT E SERVICE ACCOUNT . In the wizard that appears, create a service account and grant the account the read permissions on the bucket that stores the migrated data. In Step 3, click CREAT E KEY, and select JSON as the key type. Click CREAT E to save the JSON file to your local machine. Click CLOSE. Click DONE.
Not e If you already have a service account, click the service account name and choose EDIT > CREAT E KEY. Select JSON as the key type and choose CREAT E > SAVE.
Alibaba Cloud OSS
Create a destination bucket.
Create a destination bucket to store the migrated data. For more information, see Create buckets.
Create and authorize a Resource Access Management (RAM) user.
6 > Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Prepare for a migrat io n
i. Log on to the RAM console. ii. In the left-side navigation pane, choose Ident it ies > Users > Creat e User. iii. Select Console Access and Programmat ic Access, and then specify the required RAM user information. iv. Save the new account, password, AccessKey ID, and AccessKey secret. v. Find the RAM user, and click Add Permissions in the Actions column. Grant the RAM user the read and write permissions on OSS (AliyunOSSFullAccess) and the management permissions on Data Transport (AliyunMGWFullAccess). vi. In the left-side navigation pane, choose Overview > RAM user logon. On the page that appears, log on to the official website of Alibaba Cloud by using the credentials of the RAM user that you have created.
> Document Version: 20210729 7 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion 3.Implement migration This topic describes the notes and procedure for data migration.
Notes
When you perform an online migration, take note of the following issues: A migration job occupies the network resources of the source data address and the destination data address. To ensure business continuity, we recommend that you specify a rate limit for your migration job or run your migration job during an off-peak period. Before a migration job starts, the system checks the files at the source data address and the destination data address. If a source file has the same name as a destination file and the source file is updated later than the destination file, the destination file is overwritten. If the two files have different content, you must change the name of one file or back up the destination file. Data Online Migration allows you only to migrate data of a single bucket at a time. You cannot migrate all data that belongs to your account at the same time.
Step 1: Create a source data address
1. Log on to the Alibaba Cloud Data Transport console. 2. In the left-side navigation pane, choose Dat a Online Migrat ion > Dat a Address. On the page that appears, click Creat e Dat a Address. 3. In the Creat e Dat a Address panel, configure the following parameters and click OK.
Parameter Required Description
Dat a T ype Yes Select Google St orage.
Enter a name that is 3 to 63 characters in length. Dat a Name Yes Special characters are not supported, except for hyphens (-) and underscores (_).
Enter the name of the bucket that stores the data Bucket Yes to be migrated.
Migrate All Data: All the data in the bucket is migrated.
When you migrate all data, you do not need to specify a prefix.
Pref ix Yes Migrate Partial Data: Only the files in a specified directory (prefix) are migrated. A prefix must start with a forward slash (/) and end with a forward slash (/). For example, you can specify /data/to/oss/ or /docs/ as the prefix.
Upload the JSON file that you have obtained Key File Yes during preparation.
8 > Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
4. (Optional)If the system prompts that this feature is in the public preview, click Applicat ion to submit a whitelist permission application. 5. Enter the required information and submit the application for public preview. After the application is approved, the system sends a text message to notify you.
Step 2: Create a destination data address
1. Choose Dat a Online Migrat ion > Dat a Address, and click Creat e Dat a Address. 2. In the Creat e Dat a Address panel, configure the parameters and click OK.
Parameter Required Description
Dat a T ype Yes Select OSS.
Select the region where the destination data Dat a Region Yes address is located.
Enter a name that is 3 to 63 characters in length. Dat a Name Yes Special characters are not supported, except for hyphens (-) and underscores (_).
Select an endpoint based on the region where your data is located.
http://oss-cn-endpoint.aliyuncs.com: indicates that you use an HTTP-based endpoint to access Object Storage Service (OSS) over the Internet.
https://oss-cn-endpoint.aliyuncs.com: OSS Endpoint Yes indicates that you use an HTTPS-based endpoint to access OSS over the Internet. For more information, see Endpoint.
Not e When you migrate third-party data to OSS, you must access OSS from an Internet endpoint.
Specify the AccessKey pair that is used to migrate Access Key Id and Yes data. For more information, see Create an Access Key Secret AccessKey pair.
OSS Bucket Yes Select a bucket to store the migrated data.
> Document Version: 20210729 9 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
Parameter Required Description
An OSS prefix cannot start with a forward slash (/) and must end with a forward slash (/). For example, you can specify data/to/oss/ as the OSS prefix. Do not specify this parameter if you want to migrate data to the root directory of the specified bucket.
Not ice If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you configure the OSS Pref ix No destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file to be migrated is /test/test.png, you must specify an OSS prefix, such as oss/. After the migration job is completed, the name of the OSS file changes from /test/test.png to oss//test/test.png.
Step 3: Create a migration job
1. Choose Dat a Online Migrat ion > Migrat ion Jobs, and click Creat e Job. 2. In the Creat e Job panel, read the Terms of Migration Service and select I underst and t he above t erms and condit ions, and apply f or opening dat a migrat ion service. Click Next . Then, the Fee Reminder message that reminds you of the request fee appears. Click Yes, Go Ahead.
3. In the Job Conf ig step, configure the following parameters and click Next .
Parameter Required Description
Enter a job name. The name must be 3 to 63 characters in length and can contain only Job Name Yes lowercase letters, digits, and hyphens (-). The name cannot start with or end with a hyphen (-).
10 > Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
Parameter Required Description
Source Dat a Yes Select the source data address that you created. Address
Select the destination data address that you created.
Not ice The source data address can be located in a different country than the destination data address. In this case, before Dest inat ion Dat a you configure the migration job, you must Yes Address submit a ticket to request permissions to create cross-country migration jobs. You must ensure that your business is legitimate, the data transit process conforms to local rules and regulations, and your data does not contain illegal information.
Do not f ilt er: All data from the source data address is migrated. Exclude: The files and subdirectories in the excluded directories are not migrated. Cont ain: Only the files and subdirectories in the specified directories are migrated.
Not e A directory name cannot start with a forward slash (/) or a backslash (\) and cannot contain double slashes Specif ied Direct ory No (//), double periods (..),or double quotation marks ("). The character string that consists of the specified directory names cannot exceed 10 KB in size. A directory must end with a forward slash (/). For example, you can specify docs/ as the directory. You can specify a maximum of 20 directories of the Exclude or Contain type.
Full: The system runs a full migration job. When you run a full migration job, you must configure the St art T ime Point of File parameter. The system migrates the data whose last modification time is later than the specified st art t ime at one time. The migration job is closed after all data is
> Document Version: 20210729 11 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
migrated. You can submit the job again if the Parameter Required Description data at the source data address changes. In this case, only the data that is changed after the previous job is migrated. Increment al: The system runs the migration job based on the specified Migrat ion Int erval and Migrat ion T imes parameters. You must configure the St art T ime Point of File parameter. During the first migration, the system migrates the full data that is modified after the specified st art t ime. After the first migration is complete, incremental migrations are performed based on the specified Migrat ion Int erval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration started and before the current migration starts. If you set Migration Times to N, full migration is performed once and then incremental migration is performed (N - 1) times. For example, you set Migration Interval to 1 hour, Migration Times to 5, and St art T ime Point of File to 08:00, March 5, 2019. In this case, the current time is 08:00, Migrat ion T ype Yes March 10, 2019. When the first migration starts, the system migrates the files that are last modified between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the first migration requires 1 hour to complete and the second migration starts at 10:00, March 10, 2019. The two hours from 08:00 to 10:00 include the 1 hour that the first migration requires and the migration interval of 1 hour. During the second migration, the system migrates the files that are last modified between 08:00, March 10, 2019 and 10:00, March 10, 2019. The migration job consists of one full migration and four incremental migrations.
12 > Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
Parameter Required Description Not ice Before you start a migration jAolbl: oAfl lt fhiele Fsu allr eo rm Inigcraetmeden. tal type, the system compares files at the source data Assign: Only the files that are created or address with those at the destination data modified after the specified time are migrated. address. If a source file has the same name For example, if you set the start time to St art T ime Point of as a destination file, the destination file is Yes 08:00:00, November 01, 2018, only the files File overwritten when one of the following that are created or modified after 08:00:00, conditions is met: November 01, 2018 are migrated. Files that were creTahtee cdo onrt emnot dtyifpieeds boef ftohree stohuer scpe efciliefied time area nsdki pthpe dd.estination file are different. The source file is updated after the Yes (only for previous migration. The default value is 1 hour, and the maximum Migrat ion Int erval incremental value is 24T hhoeu srisz.e of the source file is different migrations) from that of the destination file.
Yes (only for The default value is 1, and the maximum value is Migrat ion T imes incremental 30. migrations)
The method of how to process a source file that has the same name as a destination file during migration. Valid values: LastModified Condition: If a source file is named the same as a destination file, the system compares the values of the LastModified, Size, and Content- Type attributes of the source file with those of the destination file.
If the values of all the LastModified, Size, and Content-Type attributes for the two files are the same, the source file is skipped File Overwrit e Yes Met hod and not migrated. If the values of at least one of the LastModified, Size, and Content-Type attributes for the two files are different, the destination file is overwritten with the source file.
All: If a source file has the same name as a destination file, the system directly overwrites the destination file with the source file. If a source file has the same name as a destination file, the system directly skips the source file.
4. On the Perf ormance tab, navigate to the Dat a Predict ion section and specify the Dat a Size and File Count parameters.
Not e To ensure a successful migration, estimate the amount of data to be migrated as accurately as possible. For more information, see Estimate the amount of data to be migrated.
> Document Version: 20210729 13 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Implement migrat ion
5. Optional. On the Perf ormance tab, navigate to the Flow Cont rol section, specify the T ime Range and Max Flow parameters, and then click Add.
Not e To ensure business continuity, we recommend that you specify the T ime Range and Max Flow parameters based on the workload peaks and troughs.
6. Click Creat e and wait until the migration job is completed.
14 > Document Version: 20210729 Migrat e dat a from Google Cloud St Dat a Online Migrat ion orage t o OSS·Manage migrat ion job s 4.Manage migration jobs This topic describes several subsequent operations that you can perform after you create a migration job.
View the status of a migration job
After you create a migration job, one of the following states is displayed for the job: Migrating: Data is being migrated. Wait until the migration job is completed. Create Failed: The migration job has failed to be created. You can view the cause of the failure and create another migration job. Finished: The migration job is completed. You can view the migration report. Failed: The migration job has failed. You can view the migration report and migrate the files that have failed to be migrated.
Modify the throttling settings
During migration, you can modify the throttling settings of a migration job based on your needs. 1. Log on to the Data Transport console. 2. Choose Dat a Online Migrat ion > Migrat ion Jobs. On the migration job list, find a migration job and click Manage. 3. Click St op and verify that the job is stopped. 4. On the Flow Control Time Schedule chart, click Reset . To add a throttling setting, select an appropriate time range and maximum flow and then click Add.
To delete a throttling setting, click next to the throttling setting.
To modify a throttling setting, delete the current setting and then add a new throttling setting. 5. Click Submit and then click St art to rerun the migration job.
View a migration report
1. On the migration job list, find the migration job and click Manage. 2. Click Generat e Migrat ion Report . After a report is generated, click Export to export the migration report. In a migration report, the following files appear in the File List s section:
The file whose name ends with _total_list . This file lists all files to be migrated. The file whose name ends with _completed_list . This file lists the files that have been migrated. The file whose name ends with _error_list . This file lists the files that have failed to be migrated. 3. In the Object Storage Service (OSS) console, find the automatically generated folder aliyun_mgw_i mport_report/. The folder includes the three files that appear in the migration report. You can download these files and view the detailed list that is included in each file. We recommend that you use the ossbrowser tool to view these files. Each list consists of the following columns:
> Document Version: 20210729 15 Migrat e dat a from Google Cloud St orage t o OSS·Manage migrat ion job Dat a Online Migrat ion s
The columns of the file whose name ends with _total_list include the source data address, file name, source version ID, and destination version ID. The columns also display whether the job version is the latest at the source data address and whether the job version is deleted at the source data address. The last two columns display the file size (measured in bytes) and the time when the file was last modified. The format of a source data address is
Not e Skip migration: indicates whether the file that has the same name at the source data address and the destination data address has been skipped during migration. Valid values: 1 and 0. The value 1 indicates that the file has been skipped during migration. The value 0 indicates that the file has not been skipped during migration. The value of this field depends on the value of the File Overwrit e Met hod parameter that you specify when you create the migration job.
The columns of the file whose name ends with _error_list include the file name, the source version ID, and the destination version ID. The columns also display whether the job version is the latest at the source data address and whether the job version is deleted at the source data address. The last three columns display the time when the migration job started, the time when the migration job ended, and the error description.
Not e The following four fields apply only if multi-version migration is enabled: Source version ID: the version ID of the file at the source data address. Destination version ID: the version ID of the file at the destination data address. Latest at source data address: indicates whether the current version is the latest version of the file at the source data address. Valid values: true and false. Deleted at source data address: indicates whether the current version is deleted at the source data address. Valid values: true and false.
Rerun a failed migration job
If a migration job fails, you can check the generated file whose name ends with _error_list to identify the cause of the failure and resolve the issue. Then, on the migration job list, find the failed migration job, click Manage, and click Ret ry to migrate the failed files.
16 > Document Version: 20210729