Binary web

My colleague once referred to FME Server as “web glue”, and he’s not wrong to call it that. In fact, I spend a fair bit of my time experimenting with methods for moving data over the web.

In this post I’m going to focus on three of the most popular methods: AJAX, HTML5 WebSockets, and Webhooks. While I’ll primarily examine how to connect to data over the web, it’s worth noting that it’s also relatively easy to transform data and automate web processes at the same time, especially where spatial data or a map is involved.

Moving Data in a Client-Server Architecture

When the client is JavaScript running on an end-user’s computer you end up with two basic options: AJAX or WebSockets.

AJAX (Asynchronous JavaScript and XML)

ajax-logo

Like the greek hero of the same name,                                                                                               AJAX leads and does not follow

AJAX allows your client-side JavaScript application to make a request to access a server-side resource. One nice thing about AJAX is that it won’t hold up your app while waiting for the response (i.e. it’s non-blocking). Instead, when the response is ready, a ‘callback’ function will be invoked to process the request. This was pretty awesome a decade ago and is still very relevant today. Here is an example of a simple coordinate-pair reprojection service that uses AJAX to send requests to FME Server’s Data Streaming service.

A decade ago you had to write your own AJAX request functions, but now there are great libraries like jQuery that provide cross-browser compatible AJAX request functions. jQuery is used on a lot of websites nowadays and I would call it the defacto standard for common JavaScript operations. In fact, it is a key component of the new FME Server JavaScript API that wraps up the most common FME Server REST API methods.

AJAX is great if you aren’t in a hurry, but if you’re moving a high volume of data then the overhead of creating an HTTP connection every time is going to be a bottleneck. You need a persistent connection instead. In addition, AJAX always has to poll the server for data rather than receive it via push from the server. If you want speed and efficiency you need WebSockets.

WebSockets

Lemme upgrade ya, make it all messages, give you full duplex, upgrade ya (image credit: pubnub.com)

HTML5 WebSockets

WebSockets allow your client-side JavaScript to open and persist a connection to a server. With WebSockets, data is exchanged as messages, which can happen very quickly due to the persistent connection.  This makes it possible to develop some really cool web apps like this live spatial dashboard showing ship, bus and plane locations.

So, if you are dealing with real-time data you might need to upgrade to WebSockets. I say ‘upgrade’, because you can initialize a HTTP request on port 80 and request that it be upgraded to WebSockets in order to switch protocols. The benefit of using the standard web server port 80 is that many firewalls and organizations allow open access to this port.

Another powerful aspect of WebSockets is a capability called full duplex, which is a fancy way of saying that both the Client and Server can send each other messages. Contrast that with AJAX where the server has no method for pushing messages to the client. This two-way message sending allows web applications like Facebook and Gmail to provide the feel of a desktop application. Closer to home, it enables the creation of web applications like this Real-Time Map Editor that updates each users web map immediately with any additions, updates or deletions that are made.

Moving Data in a Server-Server Architecture

Strictly speaking, there are a number of ways of pushing data from one server to another server, including: Java Messaging Service (JMS), Amazon Simple Notification Service (SNS) and even HTML5 WebSockets. In many situations these are great solutions for robust, enterprise integration. However, the overhead of standing up a specialized server or paying for a service and also implementing the API is a barrier to system integration. This is especially the case with many of the newer cloud-based services coming online.

So how can we offer an easier method for pushing data from one server/service to another? The answer is remarkably simple and brilliant: Webhooks.

Webhooks

aaarr-webhooks-pirate

No need to plunder for data since you’re going to push it to me anyway (image credit: mailchimp.com)

A Webhook is really just a URL that accepts a HTTP POST (or GET, PUT, DELETE). One server/service will provide another with a callback URL that it should push data to when it has some data to send. Brilliant! Any old web server can be used really. In fact, there are services available online that provide the web server, script editor and URLs for you.

One of our favorite web automation services, Zapier, supports Webhooks, which includes sending to another server/service’s callback URL and receiving data sent to their own callback URL. As it turns out, FME Server also support Webhooks. In the recent Developing for Data Transformation webinar, I utilized both Zapier and FME Server Webhooks to process a new file dropped into Google Drive, extract some spatial information with FME and then create a set of required actions in Google Tasks. (just thought of an idea: data validation that creates a task list of problems to fix – ™, all rights reserved).


In our recent webinar, we found that 59% of folks were familiar with AJAX, 44% with WebSockets, but only 7% with Webhooks. How about you? Each have their pros and cons and scenarios in which they’re best suited. Which methods are you employing? What’s been your experience?

Thinking about using FME Server to build an awesome web app or integration? It’s worth watching the aforementioned webinar, and please drop me a comment on what you’re up to or if you need assistance.

About Data Ajax FME Server HTML5 JMS SNS Web Webhooks WebSockets

Aaron Koning

Comments

4 Responses to “Moving Data over the Web: AJAX vs. WebSockets vs. Webhooks”

  1. Thorsten says:

    The Webhook support will be very helpful, thanks.

    “just thought of an idea: data validation that creates a task list of problems to fix – ™, all rights reserved”

    Too late for the ™, HALE (www.wetransform.to/products/hale) does that since 2009 ;). If you want to bring this to FME, there is a FME plugin (http://community.esdi-humboldt.eu/projects/hale/wiki/HALE_Schema_Transformation_Plug-in_for_FME).

    All the best,

    Thorsten

  2. Excellent, thanks for sharing with us. It made it much more understandable for me.
    Also, thanks for examples provided on the website, such as http://demos.fmeserver.com/spatialdashboard/index.html, that demoes use of websockets.

  3. Ismael Ortiz says:

    Hi, I’m using Ajax in which I can set up a TimeOut Response and Catch an Error like 400 Bad Request, etc;

    ¿ If I change to WebHook CAN I use TimeOut Response or Receive/Catch Errors like 400 Bad Request ?

    Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts