Technology lets people automate pretty much everything—from manufacturing to tweets to space exploration. We made FME so people can automate their data-related tasks. Here are a few real-world FME Server scenarios to inspire you, along with tips for your own data workflows.
Why do we need data automation? Allow me to explain the motivation behind FME with a flow chart.
Ok, now that’s cleared up… the point of FME and FME Server is to do all your workflows automatically so you don’t have to.
|
“FME Server does all my work for me while I sleep at night.” – Jerrod, Devon Energy
|
Data automation can mean a few things:
- Delivering data anywhere, anytime—as zipped files, URLs, visual reports, web maps, or otherwise.
- Scheduling a workflow to run whenever it’s needed.
- Triggering an event. Triggers can be systems, sensors, or human actions like dropping files in directories or submitting requests on a website.
Let’s get into some examples & tips.
Loading data on demand, on a schedule, and in real time
Validate data before it’s loaded
Example: The Swedish EPA must load data that’s collected in the field while ensuring it meets quality standards. First, the person who collected the data uploads the files using a web form. Then, FME Server then analyzes the data quality and emails a report to that person.
Tip: Data validation is a best practice. Allowing providers to immediately know the quality of their submission saves time, as bad data can be caught and fixed without involving anyone else.
Replicate data on a schedule
Example: San Jose Water Company must keep their data synchronized between different systems. Their FME workspaces transform Oracle Spatial data and load it into PostGIS. FME Server runs these workspaces at regular intervals.
|
|
“FME Server is the caretaker that lets us pay attention to other things.” – San Jose Water Company
|
Tip: For duplicate systems, nightly replication avoids disruptions during the day.
Send a notification when a system is updated
Example: When data in one of San Jose Water Company’s systems is updated, another department must receive a request to review the update. When their database is modified, FME Server triggers a workflow that creates a notification in PivotalTracker the other department.
Tip: Connect systems in real time if you need to initiate the next action as soon as possible. Time-to-insight is an increasingly important metric.
Working with big data volumes
Query colossal networks
Example: The London Fire Brigade must determine travel times between all their fire stations, across their entire road network, and must reflect changing traffic conditions. This requires over 10,000 route queries, which translates to over 40 hours of overtime work to complete. Automating the route queries with FME Server cuts that to just 2 hours (and it’s done by a computer, not a human).
Tip: Estimate the total required time for a manual task (duration x frequency) and decide if automating it will help.
Generate giant maps
Example: Ordnance Survey Ireland must regularly generate national cartographic maps for emergency services. Their FME workspace restructures and transforms 14,000+ vector datasets and generates map tiles. The data was so huge that running the workflow on a single desktop machine wasn’t enough, so they moved it to FME Server to leverage multiple FME Engines in parallel.
|
|
“The volume of data required to be processed needed an automation workspace run on FME Server.” – Ordnance Survey Ireland
|
Tip: When working with big data, look for ways to improve performance in your FME workspaces. Use 64-bit FME for full access to memory resources, follow best practices, use parallel processing within transformers, and consider leveraging FME’s superfast point cloud transformers. Use dedicated FME Server machine(s) and consider parallelizing by using multiple FME Engines.
Generate even gianter maps
Example: Pelmorex must generate an 880,000-tile global meteorological web map every 12 hours. This is provided from AWS S3 and requires 80 hours of processing time (in just 12 hours!). The solution: Five 16-core FME Cloud instances are started on demand to process the data in 1 hour, and then shut down.
Tip: The cloud allows you to provision processing power as needed (e.g. you can schedule instances to run only when you need them). If your data is already in the cloud and you need to process a lot of data in a short time, FME Cloud will definitely be cheaper and more effective than on-premises FME Server instances.
Show data in real time
Example: A certain railway company must generate a web map with the real-time location and status of thousands of assets. This means over 3 million messages per day coming in via JMS. FME writes the messages to a database, does some GIS transformations, and streams it to the web map using WebSockets. FME Engines are added as processing capacity and fault-tolerance is needed.
Tip: If you need to process more than 1 message per second, this is “Fast Data”. Fast data could be weather, social media, GPS, IoT, or anything else that sends messages at such a rate. For this, use message streams (e.g. JMS, SQS) and keep multiple FME Engines running at all times.
Making data available
Create a map-focused data downloader
Example: The State of Montana allows users to select an area of interest on a map, and automatically delivers it to them in their desired format and coordinate system.
Tip: A data portal with a map focus lets users quickly narrow in on their geographic area of interest.
Create a metadata-focused data downloader
Example: York Region allows users to select a category of interest and a layer, and automatically delivers it to them in their desired format and coordinate system.
Tip: A data portal with a metadata focus lets users quickly narrow in on the type of data they’re interested in.
Offer analytics in addition to the data
Example: Galway County Council lets users click a map to indicate the site they want, and FME analyzes the data (spatial overlays, KPI calculations, etc.) and delivers a report.
Tip: Providing online decision support tools rather than just raw data can help streamline and spur economic activity.
*
Are you inspired yet? Data automation can mean event triggers, running a workflow on a schedule, or making data available on demand. With FME, you can also automate wildfire threat assessments and so many other use cases. This is the quintessence of FME Server. Let us know what kind of automations you’re doing. We love to hear about it!
You might also want to read:
Automating an Auction with FME Server
Tiana Warner
Tiana is a Senior Marketing Specialist at Safe Software. Her background in computer programming and creative hobbies led her to be one of the main producers of creative content for Safe Software. Tiana spends her free time writing fantasy novels, riding her horse, and exploring nature with her rescue pup, Joey.
