Data Migration

Home , Data migration

Data Migration DISSECTING THE WHAT, THE WHY, AND THE HOW 02 Table of Contents The What 04 What Is Data Migration? 05 Types of Data Migration 05 Database Migration 05 Application Migration 05 Storage Migration 06 Cloud Migration 06 The Why 07 Situations That Prompt Data Migration 08 Challenges in Data Migration 08 1. The Complexity of Source Data 08 2. Loss of Data or Corrupt Data 08 3. Need for In-Depth Testing and Validation 08 Factors that Impact the Success of a Data Migration Process 09 Is Your Migration Project Getting the Attention It Needs? 09 Thoroughly Understand the Design Requirements 09 Budget for the Field Expert 10 Collaborate with the End Users 10 Migration Isn’t Done in OneGo 10 Backup Source Data 10 Migration Doesn’t Make Old Systems Useless 11 Plan for the Future 11 The How 12 Data Migration Techniques 13 Extract, Load, Transform (ETL) 14 The 7 R’s of Data Migration 14 Data Migration Tools 14 Finding the Right Migration Software – Features to Consider 15 Easy Data Mapping 15 Advanced Data Integration and Transformation Capabilities 15 Enhanced Connectivity 15 Automated Data Migration 15 Planning to Migrate? Steps to A Successful Enterprise Data Migration 16 1. Design a Strategy 16 2. Assess and Analyze 16 3. Collect and Cleanse Data 16 4. Sort Data 17 5. Validate Data 18 6. Migrate 19 Conclusion 20 Astera Centerprise – Making the Data Migration Process Painless 21 About Astera Software 22 03 Summary

With data of varying formats pouring in from diﬀerent systems, the existing system may require an upgrade to a larger server. Data migration enables organizations to expand their data store and management capabilities and make full use of data to drive business decisions. It is a widely used process of moving data, and almost every organization undertakes one at some point.

This eBook will give you a deeper insight into the data migration process. It will further discuss why this process is essential for organizations, the challenges they might encounter during the process, and ﬁnally list down the techniques and tools that can simplify and streamline data migration.

www.astera.com 04

The what Know the Basics

www.astera.com 05

What Is Data Migration?

According to the International Data Corporation (IDC), data migration represents 60% of any large enterprise IT project. But what is it?

Simply put, it refers to the data transfer between diﬀerent types of ﬁle formats, databases, and storage systems. However, ‘transfer’ is not the only aspect of data migration methodology. If the data is diverse, the migration process includes mappings and transformations between source and target data. Above all, data quality must be assessed before migration to ensure a successful implementation. The success rate of any data migration project is directly dependent on the diversity, volume, and quality of data being transferred. Types of Data Migration

Data migration can be broadly categorized into four types: Database Migration This type of migration involves moving data between two database engines. However, when the original data changes, it simultaneously affects the data language or protocol. The change is reflected in the application layer as well. Database migration modifies data without modifying the schema. The circumstances that demand this type of migration include: • When the database software requires an update • To migrate a database to the cloud • In case the organization needs to change database vendors It requires careful planning and testing as there are several small tasks involved in the process, such as determining the storage capacity of the target database, testing applications, and ensuring data confidentiality. Application Migration This type of migration occurs when an organization switches from one platform or vendor application to another. Each application has a unique data model. Moreover, applications are not portable. There- fore, the operating systems, virtual machine configurations, and management tools of each application can be different in the development and deployment environment. These factors can contribute to the complexity of the migration process. Introducing middleware products in the process simplifies it; as they help bridge the technology gap.

www.astera.com 06

Storage Migration The process deals with moving data from one storage system to another, such as a hard disk or the cloud. The need for storage migration arises due to technology upgrades. It is easy to implement data management features like data validation, cloning, reducing corrupt or old information, etc. during this process. In addition, storage migration enables cost-effective scaling and faster performance. Cloud Migration Moving to the cloud ensures scalability, requires fewer storage resources, and is cost-effective and therefore, cloud migration has become one of the latest trends in the data management industry. In cloud migration, an organization’s complete or partial information assets, applications, or services are deployed to the cloud. The cloud’s firewall protects the migrated data.

www.astera.com 01 07

The why Reasons for Migrating Data

www.astera.com 08

Situations That Prompt Data Migration

The most common reason for migration is the need to move data to a new system to scale up and accommodate volume growth. However, other circumstances may also prompt organizations to opt for a data migration project. The reasons include: • To replace legacy systems that are unable to keep up with the evolving performance requirements of the organization. • To reduce storage footprint by migrating to a system that consumes less power and space. Consequently, cutting down the operational costs. • To stay competitive by adopting state-of-the-art technologies, paving the path for data migration. • To prepare a disaster recovery plan; as data migration can help create a secure. • To transfer data to the cloud, eliminating the cost of on-premise IT infrastructure.

Challenges in Data Migration

Migrating enterprise data from one system to another is always challenging, especially when you have to move millions of records across systems. To help simplify your data migration process, we’ve identiﬁed the following most common data migration challenges you could face and how to avoid them: 1. The Complexity of Source Data One of the main challenges in data migration is the complexity of source data. Often, you’ll have to perform data integration on the source data to work through any intricacies. For example, if you were using a legacy data storage system, then you will have to perform data transformation to make the data compatible with the target system. 2. Loss of Data or Corrupt Data While migrating data from one system to another, you might face data loss or corruption. To tackle this issue, make sure that the number of source records matches with the number of migrated records. Various tools are available in the market that can help you authenticate the migrated data. 3. Need for In-Depth Testing and Validation For complex data migrations that involve thousands of records, the cost of failure is simply too high. Therefore, the need to perform in-depth data validation and testing cannot be ignored. Keep in mind any events that could have aﬀected data quality. Also, test a large data volume to assure quality. It’s recommended to pull at least 10 to 20 percent of your data to make sure you include an extensive range.

www.astera.com 09

Factors that Impact the Success of a Data Migration Process

A fully functional data migration process must include the following considerations:

Is Your Migration Project Getting the Attention It Needs? Selecting a new enterprise-wide system is a strategic business undertaking that deals with new technologies and professionals. Migration is usually a small part of a much bigger project. An average business typically focuses on system conﬁguration and other technicalities rather than ensuring the data that will populate the new system is ﬁt for purpose.

Holistically speaking, out of the plethora of tasks, data migration falls low on the list of priorities by a management that consider it to be a simple task of transferring data, and does not find the high costs and administrative burden justifiable. Thus, data migration, the resources it requires as well as the difficulty of the task is often underestimated and put on the backburner until it’s too late. Make sure that doesn’t happen with you.

Holistically speaking, out of the plethora of tasks, data migration falls low on the list of priorities by a management that consider it to be a simple task of transferring data, and does not find the high costs and administrative burden justifiable. Thus, data migration, the resources it requires as well as the difficulty of the task is often underestimated and put on the backburner until it’s too late. Make sure that doesn’t happen with you.

Thoroughly Understand the Design Requirements Firm grasp of the design requirements is critical, namely migration priorities and schedules, replication and backup settings, and capacity planning. A simple miscalculation in this area can have lasting repercussions that may impact the cost. This is also the stage where the IT department decides which migration strategy is suitable for the project – Trickle or Big Bang. Selecting one that suits you best is highly subjective to the ﬁrm’s existing data and future requirements. Let’s have a look.

Big Bang – This type of migration initiates and completes full transfer within a limited time frame. Expect some downtime as the data moves and processes, however the project execution is rather quick. Trickle – This migration strategy conducts the project in multiple phases. The target and source systems run simultaneously, which keeps the migration running in real-time. This type of migration is more complex and time-consuming as compared to Big Bang, however it takes lesser downtime and eliminates operational eﬃciencies.

www.astera.com 09

Budget for the Field Expert Quite a few tech-ﬁrms prefer a hands-on approach and migration budgets simply don’t allow for an expert’s input. Nevertheless, unless the ﬁrm happens to have an in-house migration specialist, they will need to spend money to save money by hiring a data migration specialist.

Collaborate with the End Users The data migration process should be considered a business project, rather than a technical set of steps, that involves end-users. Your staﬀ or customers will have a stake in the migration and understandable anxiety over its success.

Involve them: Depending on the specific data rules you plan to implement, consider which data should receive priority to migrate first. In addition, try to understand and implement what your end-users are hoping from the migration: Better performance? Analytics? A simpler way to issue requests? By following this approach, you will experience a far unified and comprehensive migration project that takes into account the concern of the stake holders, and saves considerable time and cost in the long run.

Migration Isn’t Done in One Go It’s quite common for a data migration project to have multiple phases. This is a known engineering practice which breaks down the entire process of migration into manageable chunks. Instead of attempting to resolve migration, risk errors, and redundancies in one go. In some migration cases, the ﬁrst attempt fails and needs to be restarted. Multiple phases provide checkpoints allowing you to perform integrity checks as the migration progresses.

When compared to data migration, storage migration is far simpler as you don’t need to update old storage and map it to new. However, migrating data between vastly diﬀerent storage systems is the real challenge. Using software that have a code-free environment with built-in data quality features, advanced transformations, and support for a wide variety of data formats simplify storage migrations.

Backup Source Data With massive amounts of data and restricted storage space, some data can’t be backed up. Should the worst happen, you’ll lose invaluable data during migration. Ideally you must be well prepared with back- ups to restore data and try again. Since most databases usually have terabytes upon terabytes of data it is understandable that you may run out of storage space. In such a case, arrange for more backup space without hesitating. You cannot put a price on data you’ve spent years accumulating.

www.astera.com 10

Migration Doesn’t Make Old Systems Useless It’s quite likely that your old system will continue to be in use alongside your brand-new replacement. Taking the old system oﬄine immediately can prove to be counterproductive as the new rollout may experience some hiccups until it’s completely operational for the end user.

Plan for the Future Once all the data has migrated successfully, attempt to test the migration again using a mirror of the production environment. Once it checks out, you may go-live carefully and carry out ﬁnal tests. When the environment is operating smoothly, you can shut oﬀ the old systems.

You can also take initiatives to make your life easier post-migration. Rather than spending costly resources for updating source data before migration, you can establish analytics and governance con- trols in this new environment. Continuously monitoring migrated data to look for unusual access patterns, orphaned work sets, and security can also help you in the long run. This will ensure that your enterprise data is utilized to the fullest.

www.astera.com 12

The how Perform Successful Data Migrations Easily

www.astera.com 13

Data Migration Techniques

Several factors determine the right migration technique for an organization, such as the available resources, data volume, data sensitivity, and business requirements. The method that oﬀers a blend of reliability, migration eﬃciency, and minimum impact on users and business processes is the one that can best serve the organization’s needs.

TDWI conducted a survey to identify the technologies organizations use to migrate or consolidate databases. ETL emerged as the preferred technology with 41% respondents voting in its favor. Other answers included: manual coding (27%), database replication (11%), and Enterprise Application Integra- tion (3.5%).

Fig. 1: ETL Emerges as the Preferred Choice for Data Migration

www.astera.com 14

Extract, Load, Transform (ETL) ETL tools can handle the complex requirements of the data migration process. These include processing big data sets, in-depth data proﬁling, and integration between multiple platforms. Some ETL tools even provide the capability of automating the standard ETL tasks, such as retrieving data from operational systems, transforming it into a uniﬁed format, and loading it into a destination database.

The 7 R’s of Data Migration

When planning your data migration strategy, it’s imperative to keep in mind these seven critical characteristics: • Robust: Eﬃciently manage all phases of data migration ranging from data extraction and transformation to authentication and loading into the destination system, even when data volume is high • Resilient: Easy to modify and handle problems in the operating environment • Rapid: Eﬃcient process execution, leveraging the present source or destination facilities to support quick processing • Reporting: Progress pointers during the migration process should be available • Reconciliation: Reconcile the completed process • Recoverable: The data should be retrievable from the point of failure when needed • Reusable: Should have reusable process modules, including transformation utilities, error control, and data cleaning procedures that can be used in other tasks without impacting the already migrated data Data Migration Tools Data migration tools are broadly divided into three types:

On-premise: Installed on-premise to enable data transfer within the organization.

Open-source: Community-developed tools available at a low price or free of charge.

Cloud-based: Available on the cloud, these tools move data from diﬀerent systems (including applications, cloud and on-premise data stores, services) to the cloud.

www.astera.com 15 Finding the Right Migration Software – Features to Consider The key to selecting the right data migration tool is research. Browsing through review websites like Capterra, G2 Crowd, etc. will give you a fair idea of what options are available in the industry.

The next step is to know about the basic features that will help you perform successful enterprise data migration.

Here are the important features that you should consider when selecting a data migration tool:

Easy Data Mapping To carry out a successful data migration process, it’s essential to correctly map data from source to destination. To enable business personnel and data professionals use these processes easily, it’s best to go for a tool that oﬀers a code-free, drag-and-drop, graphical user interface.

Advanced Data Integration and Transformation Capabilities Executing a successful migration project involves extracting data from the desired source, identifying quality issues and errors through proﬁling, and transforming it to follow the destination schema. A data management solution that oﬀers connectivity to a wide range of heterogeneous sources, ETL, and data quality and transform features restructures data for targeted delivery and ensures that the enterprises can execute complex migration projects and get maximum value from the migrated data.

Enhanced Connectivity A data migration tool should seamlessly connect with disparate source and destination data structures, such as XML, JSON, EDI, etc. If your business requires connectivity to business applications (Microsoft Dynamics CRM, Salesforce, etc.) or databases (IBM DB2, Teradata), look for a tool that oﬀers pre-built connectors for these applications.

Automated Data Migration Features like workflow orchestration and job scheduling eliminate manual work, streamline data processes, and enable faster data migration. Especially if your migration project involves large data sets, automating the process offers additional benefits such as reduced implementation time, lower project costs, and improved ROI.

www.astera.com 16 Planning to Migrate? Steps to A Successful Enterprise Data Migration Data migration is more than just moving data for one repository to another. Executing a successful transfer includes several steps. These include

1. Design a Strategy Selecting a strategy depends on the need for data migration. Is it because the organization is consoli- dating systems following an acquisition or a merger or is it because of data overload? Solidifying the business objectives will lay the groundwork for the data migration plan.

Next, the organization needs to decide whether to go for data migration services or enterprise-grade ETL tools designed to facilitate data migration. There are several service providers that will identify the source and destination systems, generate workﬂows, and migrate the content. Alternatively, purchasing a code-free data migration solution gives you the liberty to carry out the transfer at your convenience, speeds up the project, and reduces risks. Simultaneously, it can lower the overall project cost.

By strategizing beforehand, the organization can inform business and IT users regarding the goals of the migration project and ensure enterprise-wide successful adoption and implementation.

2. Assess and Analyze The next step is to analyze enterprise data. Answering the following questions can help with this assessment:

• How much of the enterprise data requires migration? • Is the data structured or unstructured? • Is data redundancy going to be an issue? • How much of the data is ROT (redundant, obsolete, or trivial)? • Is the information old or recent? Comb through all the workﬂows and shared drives that store data to ﬁgure out its use throughout the organization.

3. Collect and Cleanse Data This step involves removing ROT data, compressing the remaining content, and converting enterprise-wide data into a single format. For example, in Figure. 2 data proﬁling transformation is applied on the order data of a company to examine the available information and remove duplicate/redundant records. Figure. 3 shows the details of the data set.

www.astera.com 17

Fig.2. Data Proﬁling

Fig. 3: Data Proﬁling Result

Manual execution of these tasks can be time and resource intensive for large datasets. Therefore, ﬁnding a solution that automates these tasks can make the migration process eﬃcient and error-free.

4. Sort Data Once you have proﬁled the data into a high-quality and usable form, the next phase is to categorize it according to the migration requirements. The categorization can be done on the basis of product type, ID, or any other criteria. It makes easier to route data into the right buckets.

Continuing the above-mentioned example, ﬁgure. 4 shows the proﬁled data sorted with respect to the Product ID.

www.astera.com 18

Fig. 4: Applying Sort Transformation

5. Validate Data This step involves reviewing the execution process. Assess the data rules and check if they are working the way they are supposed to, and map out any exceptions in your dataflow. Even if the process is automated, it’s best to keep a check on it to ensure that you don’t encounter any unexpected challenges when implementing the process. After sorting the data, we will apply the data quality rules to determine the errors based on defined rules. In this scenario, we are using data quality rules to identify and eliminate the records in which the field ‘Quantity’ has a zero value.

www.astera.com 19

Fig.5: Applying Data Quality Rules

6. Migrate This is the ﬁnal step in which everything falls into place. The steps mentioned above will give you well-organized and clean datasets; all you have to do is migrate them from one system to another. Once the data is migrated to the targeted destination, which can be a database or a data repository, such as a data mart or a data warehouse. The employees can then access it to fuel data analytics, streamline workﬂows, enhance data security, and more.

In the example below, the data is migrated to SQL server after being proﬁled, sorted, and checked for errors.

www.astera.com 20

Fig. 6: Data Migrated to the Destination Database

Conclusion

Migrations that are not completed on time can prove to be costly, both in terms of revenue and reputation. To expedite data migration projects and minimize the risk of failure, your business needs an enterprise-grade tool that can automate the repetitive tasks associated with data migration while ensuring data quality and minimum downtime. Astera Centerprise is an all-inclusive data management solution that enables businesses to accomplish faster data migrations in an easy-to-use, code-free environment. With the capability to process large data volumes and complex hierarchal structures like EDI, XML, JSON, and more, Astera Centerprise has successfully carried out complex data migration projects for several Fortune 1000 companies.

www.astera.com 21 Astera Centerprise – Making the Data Migration Process Painless

Astera Centerprise is an end-to-end data management solution that is designed to cater to the complex data needs of businesses. When you need to migrate data between two diﬀerent versions of a database engine or perform a heterogeneous migration, the software can help you with:

• Parallel processing engine and high availability feature that promise you optimal performance with minimal downtime • Data synchronization capability that helps you capture changed data and save the time and processing power spent on bulk data loads • Advanced data proﬁling and quality features that allow you to validate data against custom business rules to minimize data errors and inconsistencies • Support for a range of on-premise and cloud-based databases to cater to any data migration use-case • Drag-and-drop mapping to perform complex data transformations in a code-free manner

Interested in learning more? Watch this quick demo, download a Astera Centerprise trial, or contact us to discuss your data migration use-case.

www.astera.com 22

About Astera Software Astera Software is a rapidly-growing provider of enterprise-ready data solutions. We help business users bridge the data-to-insight gap with our suite of user-friendly yet high- performance data extraction, data quality, data integration, data warehousing & electronic data interchange solutions, which are used by both midsize and Fortune 500 companies across a range of industries. Acclaimed for their intuitive interface and advance functionality, our products oﬀer the same level of usability to both developers and non-developers, allowing business users to spend less time managing data and more time using it.