My colleague once referred to FME Server as “web glue”, and he’s not wrong to call it that. In fact, I spend a fair bit of my time experimenting with methods for moving data over the web.
In this post I’m going to focus on three of the most popular methods: AJAX, HTML5 WebSockets, and Webhooks. While I’ll primarily examine how to connect to data over the web, it’s worth noting that it’s also relatively easy to transform data and automate web processes at the same time, especially where spatial data or a map is involved.
Moving Data in a Client-Server Architecture
AJAX is great if you aren’t in a hurry, but if you’re moving a high volume of data then the overhead of creating an HTTP connection every time is going to be a bottleneck. You need a persistent connection instead. In addition, AJAX always has to poll the server for data rather than receive it via push from the server. If you want speed and efficiency you need WebSockets.
So, if you are dealing with real-time data you might need to upgrade to WebSockets. I say ‘upgrade’, because you can initialize a HTTP request on port 80 and request that it be upgraded to WebSockets in order to switch protocols. The benefit of using the standard web server port 80 is that many firewalls and organizations allow open access to this port.
Another powerful aspect of WebSockets is a capability called full duplex, which is a fancy way of saying that both the Client and Server can send each other messages. Contrast that with AJAX where the server has no method for pushing messages to the client. This two-way message sending allows web applications like Facebook and Gmail to provide the feel of a desktop application. Closer to home, it enables the creation of web applications like this Real-Time Map Editor that updates each users web map immediately with any additions, updates or deletions that are made.
Moving Data in a Server-Server Architecture
Strictly speaking, there are a number of ways of pushing data from one server to another server, including: Java Messaging Service (JMS), Amazon Simple Notification Service (SNS) and even HTML5 WebSockets. In many situations these are great solutions for robust, enterprise integration. However, the overhead of standing up a specialized server or paying for a service and also implementing the API is a barrier to system integration. This is especially the case with many of the newer cloud-based services coming online.
So how can we offer an easier method for pushing data from one server/service to another? The answer is remarkably simple and brilliant: Webhooks.
A Webhook is really just a URL that accepts a HTTP POST (or GET, PUT, DELETE). One server/service will provide another with a callback URL that it should push data to when it has some data to send. Brilliant! Any old web server can be used really. In fact, there are services available online that provide the web server, script editor and URLs for you.
One of our favorite web automation services, Zapier, supports Webhooks, which includes sending to another server/service’s callback URL and receiving data sent to their own callback URL. As it turns out, FME Server also support Webhooks. In the recent Developing for Data Transformation webinar, I utilized both Zapier and FME Server Webhooks to process a new file dropped into Google Drive, extract some spatial information with FME and then create a set of required actions in Google Tasks. (just thought of an idea: data validation that creates a task list of problems to fix – ™, all rights reserved).
In our recent webinar, we found that 59% of folks were familiar with AJAX, 44% with WebSockets, but only 7% with Webhooks. How about you? Each have their pros and cons and scenarios in which they’re best suited. Which methods are you employing? What’s been your experience?
Thinking about using FME Server to build an awesome web app or integration? It’s worth watching the aforementioned webinar, and please drop me a comment on what you’re up to or if you need assistance.