Technology lets people automate pretty much everything—from manufacturing to tweets to space exploration. We made FME so people can automate their data-related tasks. Here are a few real-world FME Server scenarios to inspire you, along with tips for your own data workflows.

Why do we need data automation? Allow me to explain the motivation behind FME with a flow chart.

FME Flowchart

Ok, now that’s cleared up… the point of FME and FME Server is to do all your workflows automatically so you don’t have to.

quotes

 

“FME Server does all my work for me while I sleep at night.” – Jerrod, Devon Energy

 

Data automation can mean a few things:

Let’s get into some examples & tips.

Loading data on demand, on a schedule, and in real time

EvangelistNumber1Validate data before it’s loaded

Example: The Swedish EPA must load data that’s collected in the field while ensuring it meets quality standards. First, the person who collected the data uploads the files using a web form. Then, FME Server then analyzes the data quality and emails a report to that person.

Validate before loading

Tip: Data validation is a best practice. Allowing providers to immediately know the quality of their submission saves time, as bad data can be caught and fixed without involving anyone else.

EvangelistNumber2

Replicate data on a schedule

Example: San Jose Water Company must keep their data synchronized between different systems. Their FME workspaces transform Oracle Spatial data and load it into PostGIS. FME Server runs these workspaces at regular intervals.

quotes

 

“FME Server is the caretaker that lets us pay attention to other things.” – San Jose Water Company

 

Tip: For duplicate systems, nightly replication avoids disruptions during the day.

EvangelistNumber3

Send a notification when a system is updated

Example: When data in one of San Jose Water Company’s systems is updated, another department must receive a request to review the update. When their database is modified, FME Server triggers a workflow that creates a notification in PivotalTracker the other department.

Tip: Connect systems in real time if you need to initiate the next action as soon as possible. Time-to-insight is an increasingly important metric.

Working with big data volumes

EvangelistNumber4Query colossal networks

Example: The London Fire Brigade must determine travel times between all their fire stations, across their entire road network, and must reflect changing traffic conditions. This requires over 10,000 route queries, which translates to over 40 hours of overtime work to complete. Automating the route queries with FME Server cuts that to just 2 hours (and it’s done by a computer, not a human).

Big networks

Tip: Estimate the total required time for a manual task (duration x frequency) and decide if automating it will help.

EvangelistNumber5

Generate giant maps

Example: Ordnance Survey Ireland must regularly generate national cartographic maps for emergency services. Their FME workspace restructures and transforms 14,000+ vector datasets and generates map tiles. The data was so huge that running the workflow on a single desktop machine wasn’t enough, so they moved it to FME Server to leverage multiple FME Engines in parallel.

quotes

 

“The volume of data required to be processed needed an automation workspace run on FME Server.” – Ordnance Survey Ireland

 

Tip: When working with big data, look for ways to improve performance in your FME workspaces. Use 64-bit FME for full access to memory resources, follow best practices, use parallel processing within transformers, and consider leveraging FME’s superfast point cloud transformers. Use dedicated FME Server machine(s) and consider parallelizing by using multiple FME Engines.

EvangelistNumber6Generate even gianter maps

Example: Pelmorex must generate an 880,000-tile global meteorological web map every 12 hours. This is provided from AWS S3 and requires 80 hours of processing time (in just 12 hours!). The solution: Five 16-core FME Cloud instances are started on demand to process the data in 1 hour, and then shut down.

Tip: The cloud allows you to provision processing power as needed (e.g. you can schedule instances to run only when you need them). If your data is already in the cloud and you need to process a lot of data in a short time, FME Cloud will definitely be cheaper and more effective than on-premises FME Server instances.

EvangelistNumber7Show data in real time

Example: A certain railway company must generate a web map with the real-time location and status of thousands of assets. This means over 3 million messages per day coming in via JMS. FME writes the messages to a database, does some GIS transformations, and streams it to the web map using WebSockets. FME Engines are added as processing capacity and fault-tolerance is needed.

Train JMS queue

Tip: If you need to process more than 1 message per second, this is “Fast Data”. Fast data could be weather, social media, GPS, IoT, or anything else that sends messages at such a rate. For this, use message streams (e.g. JMS, SQS) and keep multiple FME Engines running at all times.

Making data available

EvangelistNumber8Create a map-focused data downloader

Example: The State of Montana allows users to select an area of interest on a map, and automatically delivers it to them in their desired format and coordinate system.

Montana

Tip: A data portal with a map focus lets users quickly narrow in on their geographic area of interest.

EvangelistNumber9Create a metadata-focused data downloader

Example: York Region allows users to select a category of interest and a layer, and automatically delivers it to them in their desired format and coordinate system.

York

Tip: A data portal with a metadata focus lets users quickly narrow in on the type of data they’re interested in.

EvangelistNumber10Offer analytics in addition to the data

Example: Galway County Council lets users click a map to indicate the site they want, and FME analyzes the data (spatial overlays, KPI calculations, etc.) and delivers a report.

Tip: Providing online decision support tools rather than just raw data can help streamline and spur economic activity.

*

Are you inspired yet? Data automation can mean event triggers, running a workflow on a schedule, or making data available on demand. This is the quintessence of FME Server. Let us know what kind of automations you’re doing. We love to hear about it!

About Data Automation Big Data Data Analytics Data Download Data Transformation FME Cloud FME Server

Tiana Warner

Tiana is a product marketing manager at Safe Software. Her background in computer programming and creative hobbies led her to be one of the main producers of creative content for Safe Software. Tiana spends her free time writing fantasy novels and riding her horse, Bailey.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts