Data migration is the process of moving data from one system to another. In this data-centric world, most businesses will inevitably need to do it at some point, whether to upgrade old systems, embrace the cloud, or to otherwise keep data robust and accessible.
Let’s discuss what exactly qualifies as data migration, some real-world examples of organizations that have successfully executed a variety of migration workflows, and what tools and software are involved. Since we love spatial data so much here at Safe Software, we’ll focus on examples that use spatial data, such as utility networks, telecommunications, and other infrastructure.
Photo by ThisIsEngineering: https://www.pexels.com/photo/code-projected-over-woman-3861969/
What is Data Migration?
Data migration comes in many forms. It can mean:
- Moving to the cloud, e.g. moving on-premises data to Amazon Web Services
- Upgrading a database, e.g. upgrading a DBMS to the latest version
- Switching applications or formats, e.g. converting CAD data to GIS
- Updating legacy systems, e.g. modernizing a collection of documents or old software
- Embracing specialized software, e.g. moving utility data to ArcGIS Utility Network
- Adopting big data systems, e.g. loading billions of records into SAP HANA
Achieving these tasks might seem straightforward—after all, you’re just moving data from A to B. An ETL (extract-transform-load) process should be sufficient. But moving data without losing data integrity can be challenging, so data validation should always be part of any migration workflow. Before and after the migration, it’s important to check that the data is correct, complete, and free of redundancies. Further, the data usually needs to be prepared and transformed for the new system, which is where data migration tools come in handy.
- See also: Learn how to perform data migration using APIs in The Ultimate Guide to APIs
Photo by Markus Spiske: https://www.pexels.com/photo/black-laptop-computer-turned-on-showing-computer-codes-177598/
8 Data Migration Examples
Migrating CAD to GIS is a common task for GIS departments across many industries, and a challenge because CAD and GIS applications store data in different ways. The City of Oak Ridge, Tennessee, migrated their Electric Department’s data from Autodesk AutoCAD to Esri ArcGIS to transition their mapping data to a more suitable environment.
When millions of records are stored in ordinary documents or other legacy systems, it can be overwhelming to think of migrating all of this data to a new, more accessible and useful system. Barnsley Metropolitan Borough Council successfully migrated over 10 years of data (over 5 million documents!) from its legacy document management system (DMS) to Microsoft SharePoint.
Converting data models is another form of migration. Central Contra Costa Sanitary District migrated sewer pipe models and attributes to a new data model without losing any information.
Big data migrations can be daunting, and choosing the right tool for the job is essential to performing these gargantuan tasks in a quick, efficient, and affordable way. IHS Markit moved 1.5 billion rows from Denodo into Snowflake. That’s terabytes of data. By automating and scaling this task using FME Server, they achieved this in just 5 hours.
Sometimes, a data migration includes integrating data from multiple sources into a central repository. Comporium migrated telecommunications data from ArcGIS and Microstation into Smallworld.
- See also: Migrating into and out of Oracle Spatial
More companies are moving on-premises data to the cloud for better security, cost savings (in hardware, software, and maintenance), better performance, stability, and accessibility. The Arkansas Geographic Information Systems Office moved over 7 terabytes of on-premises data into Amazon S3, and enjoyed such benefits after the migration.
Spatial data migrations come in many forms, and usually include transforming geometry, attributes, and coordinate systems. Consortech created an AutoCAD-based tool for infrastructure management, which helps get data into GIS. Their workflow included migrating geometry and attributes, translating from English to French, schema mapping, and data validation.
Migrating to the ArcGIS Utility Network is one example of leveraging a specialized application. It helps users align utility assets in a standard industry model. For example, AltaGas migrated from the ArcGIS Pipeline Data Model to the Utility Network to harness its powerful network management. In this presentation, Esri also demonstrated this migration by creating a drone workflow to read data from an Esri ArcGIS Portal Feature Service, prepare and validate it, and migrate it into the Utility Network.
This workspace shows how to migrate data to the Esri utility network with FME
Data Migration Tools: What is FME?
All of the above organizations used FME to migrate their data. FME is the #1 platform for geospatial data integration, offering a powerful environment for building workflows to conquer any data challenges. When you build an FME workflow to move data from A to B, you can include validation steps to check and repair the data, as well as complex transformations to prepare it for the new system. FME’s built-in format support will ensure the migration happens without losing data integrity. Finally, Automations can be set up to run migrations in a scalable, robust way.
Get started on your data migration workflow: try FME for free.
Tiana Warner
Tiana is a Senior Marketing Specialist at Safe Software. Her background in computer programming and creative hobbies led her to be one of the main producers of creative content for Safe Software. Tiana spends her free time writing fantasy novels, riding her horse, and exploring nature with her rescue pup, Joey.
