Deploying a distributed FME Server can be a complex and time-consuming task. This is why I am particularly excited about this new FME Server deployment, as it enables a distributed Windows deployment of FME Server that is optimized for the Azure Cloud in less than five minutes.
This offering, available in the Azure marketplace, defines not only the virtual machine (VM) configuration but also the file storage, load balancers and databases. So how does it work?
Overview of Distributed Windows Deployment
This deployment is not a custom version of FME Server, it is based upon a distributed FME Server Windows installation and has been optimized for the Azure cloud. An overview of the architecture is shown below, here are the highlights:
- The deployment runs in your Azure account.
- The deployment is highly available by default with multiple VMs deployed for both the FME Server Core and FME Engines.
- All of the Virtual Machines are Windows. Since you own the VMs you can also install 3rd party libraries, as a result you can read/write to any Windows-based format.
- Normal FME Server licensing applies so you can use Standard or Dynamic Engines.
- The deployment we have created can be used as a starting point. The marketplace listing is an Azure Resource Manager (ARM) template that you can download after launch and modify.
Scaling
Another exciting aspect of this deployment is it leverages Azure virtual machine scale sets. Scale sets provide both a management and automation layer to make it easier to run highly available and scalable applications on Azure:
- Management Layer – Scale sets utilize Azure Resource Manager templates which ensures your configuration is consistent across all VMs making it much easier to configure and update VMs at scale.
- Automation Layer – The number of VMs can be increased or decreased in response to demand or a defined schedule.
As you can see from the architecture diagram, the deployment has two scale sets one for the FME Server Core and one for the FME Engines.
Automated scaling of FME Engines has become possible in FME Server 2020 because of the new Dynamic Engines which allow you to pay for FME Server based upon the number of proceeding hours.
FME Server Core Scale Set
For the FME Server Core scale set, running multiple cores will ensure application resiliency, however, the number of VMs you have running will likely be relatively static. The value the scale set will bring in this scenario is simplifying the management of the VMs. For example, if you have more than one VM in this scale set, you can increase or decrease the size of a VM, or perform maintenance on a VM without downtime.
FME Engine Scale Set
For the FME Engine scale set, the management layer is important but scaling is equally important as you may want to scale the number of VMs based upon demand. There are two ways to scale, scaling on a schedule or scaling based upon data.
Scaling on a Schedule
You can define a policy that will scale the number of virtual machines at the same time every week.
For example, by default, your FME Engine scale set might default to 1 VM. A scale condition can then be created that scales the FME Engine Scale set up to 3 VMs Monday through Friday, from 09:00 PST to 18:00 PST. Once applied, Azure will automatically scale the number of VMs up to 3 during these hours, the FME Engines configured on these instances will automatically connect to the FME Server Core without any user intervention.
Scaling based upon data
A policy can be defined that will scale the number of VMs based upon a metric. Azure provides the following metrics related to the host that you can create autoscale rules on: percentage CPU, network traffic, and disk I/O.
Because of the variable workflows that you can run on FME Server, which metric you use to scale will depend upon the workflows you are running. Percentage CPU is probably a good place to start.
Coming in FME Server 2021 are new metric endpoints that will enable you to scale much more intelligently. Rather than scaling based on what is currently happening, you can scale on the amount of work that will happen. For example, you’ll be able to get information on the estimated amount of job runtime that exists in a queue. Watch out for that in FME Server 2021.
Conclusion
Further details on how to launch and configure the FME Server deployment are available here. If you are looking to deploy FME Server on cloud providers other than Azure there are many options for deploying FME Server.
Unsure if you want to host FME Server yourself on the cloud, or use FME Cloud and have Safe Software host and manage your FME Server? Have a read of the FME Server vs FME Cloud feature comparison.
Resources
Stewart Harper
Stewart is the Technical Director of Cloud Applications and Infrastructure at Safe. When he isn’t building location-based tools for the web, he’s probably skiing or mountain biking.
