"We use FME daily to help automate our business. It has helped me accomplish many things that I would not otherwise be able to do."
Kenneth Dunn, Georgia PowerMore and more data is collected every day, and it can be challenging to keep up with the pace of change. Big data is a term used to describe these datasets that are high in volume, have a variety of data types, and are processed at a high velocity. It’s safe to say that if you’re dealing with big data, you’ll be dealing with data integration, too.
With FME, not only can you align your data to help create a cohesive big data system — you can also validate, simplify, and transform your data exactly the way you want to. Set up schedules or trigger-events that run your workflows automatically so your data is always available to those that need it.
Why FME for Big Data?
Connect to Big Data Systems
Ecosystems like Hadoop and Apache Kafka are popular due to their scalability and capacity to manage large datasets. Connect to these systems in your FME workflows to clean and process your data.
Unstructured and Structured Data Support
Create workflows that transform and convert your data, no matter its structure, for optimal use. Whether it’s natural language processing you’re looking for or you’re using spatial data, FME can help!
Database and Data Warehouse Support
With more and more data being sent your way, figuring out how and where to store it becomes an essential consideration. Connect to the database or data warehouse of your choice with your unique workflows.
Custom Data Validation
Modify and validate your data by applying your own rules with any of the FME transformers. The power is in your hands! Building your validation workflows once, and use them over again any time you need.
Run Workflows Automatically
With big data, you’ll have a lot of workflows to run. So why not automate them? Use FME Server or FME Cloud to run your workflows on a schedule or using event-based triggers to keep all your data in line and ready for use.
Free Support
No matter where you are in your FME journey, we are here to help! Visit the FME Community to discuss your data journey with other FME users and our experts, or reach out to us via Live Chat for further assistance.
FME for Data Warehouses
Having proper storage architecture in place is essential for managing big data. This is why data warehouse systems like Google BigQuery, Teradata, and Amazon Redshift exist! Create workflows in FME that read data from, or write data to any of the supported data warehouses.
Eliminate data silos and integrate all your data, no matter if it’s legacy data, collected from IoT or from smartphones using FME. No longer will your enterprise data be left inaccessible to users, or worst yet, unused. With FME, you and your team will be able to make better decisions based on your data to help power your organization.
FME for Automation
When you’re facing big data, tasks will inevitably start piling up, and time will be your biggest constraint. But don’t worry! With the automation capabilities of FME, you can use your time effectively and efficiently while letting FME transform, convert, and validate in the background.
Create your workflows in FME Desktop and then use FME Server or FME Cloud to set up a schedule or event-based triggers to run your workflows in real-time. Use the Automations feature to watch for changes in directories or web services to trigger a data download or upload to the location of your choice. Even send out notifications to users when events occur, so everyone is aware of the status of your data.
FME for Application Integration
Whether you want to gather data from applications or connect to an app that will help you better understand your data, FME has the tools to help you. From Amazon S3 to Cityworks, FME can help you transform and connect your data the way you need to.
Don’t see what you’re looking for? No problem. Use the HTTPCaller to connect to any API you need. This transformer gives you the option to connect to applications securely so that you can add more to your big data collection with ease.
-
"When you combined our old rack space costs ($3,200 per month) with our 3-5yr hardware costs ($170k), migrating to Amazon and FME Cloud became an easy decision."
Seth LeMaster, Arkansas GIS Office -
"I love FME because it just makes my job easier every day."
Carlos Silva, Vancouver International Airport -
"Not only does FME enable us to provide unprecedented access to our data, both to staff and the public, but it also gives us the ability to automate data processing tasks releasing time to focus on developing ways to use data effectively."
David McDermott, Surrey Heath Borough Council -
"FME has intuitively, and instrumentally, aided us in collecting data from myriad sources and through the wealth of transformers and writers to then distribute quality data. What would have taken weeks we were able to accomplish in days. It was incredibly useful in accomplish our goal to promote greater police data visibility."
Peter Moore, Socrata -
"FME dissolves the boundaries between software systems, provides the tools to efficiently perform almost every data transformation task, and makes it easy for me to build solutions."
Steve Grise, Vertex3
Related Resources
| Webinar | Big Data - Tap Into Cloud Infrastructure | Watch and Learn |
| Presentation | Big Cities, Big Dreams, Big Data Problems | Watch and Learn |
| Presentation | Performance and Application of GIS and Big Data ETL Processes | View Slides |
| Blog | FME and the 4 V's of Big Data | Read the Post |
| Blog | 10 Ways to Automate All Your Data Workflows | Read the Post |