Increasingly we have come to expect the real-time delivery of data to our web browser/devices. Real-time is important, as having up-to-date information helps us make informed decisions. It’s particularly important to spatial data, and the application of HTML5 Web Sockets shows great potential in making it far easier to use and leverage real-time spatial data.

Why Real-Time Spatial Data is Becoming Increasingly Important

As we start to understand how to harness information from the social components of the web, one of the best ways to filter through the vast sums of information is recency. If we search for something within social media it is a fair assumption (as long as the source is trusted) that the more recent the information we obtain, the more reliable it is. For spatial this also holds true, in fact I would argue it is even more important, as a lot of information might only update minute-by-minute or hourly, but if we are tracking a moving object, the data stream has the potential to be continuous.

When I talk about real-time in a web browser/app, I mean that you should be able to sit with a web page open and watch information update on the page without any page refresh or interaction from the user (GMail chat is an example).

The Old Approach to Displaying Real-Time Spatial Data

Previously to show real-time information in a web browser several techniques could be adopted, all of them had their drawbacks. Long polling (or Comet) was probably the most popular and set up a persistent connection between the client and the server.

Let’s say we want to create a map that details the exact location of delivery vehicles in our fleet. This is a rough sketch on how it works using Comet. The client makes a connection back to the server and then once the server has an update it sends a response, as soon as the response is received by the client it makes another request to the server.

Polling Diagram

Long-polling works but it is complex to configure and maintain, produces a large amount of network traffic, and there is still a degree of lag (read more here). What it comes down to is that HTTP was never designed with this in mind so you are hacking.

Why HTML5 Web Sockets is the Solution

HTML5The solution is HTML5 Web Sockets. This is a new HTML5 standard which enables you to set up a bi-directional channel between the client and the server. Once setup you can send data either way with very low latency. What this means is that an event happening on the server can be captured (such as an update on a geometry object in the database) and pushed to the client instantly. Welcome to the world of push!

So back to our moving assets example now applied to Web Sockets. When you first load the web application, it creates a Web Socket connection to the server. Once connected, whenever an asset moves the server pushes the new data to the web application, and using JavaScript we update the map.

Web Sockets Diagram

Web Sockets obviously offer a huge amount of potential outside of the spatial world too, and much has already been written on it (here is a good overview of Web Sockets). But here’s why Web Sockets are particularly useful for delivering spatial data to the client:

That is a brief introduction to Web Sockets, the browser support is already quite good so it’s definitely a technology you can start to think about for production systems. Do you have any good use cases that you can think of?

Note: For the more technically minded I am going to look into setting up a test-case, I think I am going to use node.js and socket.io using FME Server’s notification service to trigger the push events on the server. Is this something you might be interested in?

About Data FME Server HTML Real Time Spatial Data Web Mapping

Stewart Harper

Stewart is the Technical Director of Cloud Applications and Infrastructure at Safe. When he isn’t building location-based tools for the web, he’s probably skiing or mountain biking.

Comments

13 Responses to “Bringing Real-Time Spatial Data to the Web Using HTML5 Web Sockets”

  1. Node.js was all the rage at the FOSS4G conference yet for many of us it’s still a bit…opaque.

    So any real-world scenario with FME Server notification would be very cool.

    Brian

  2. Great to see this sort of technology being linked with spatial data and very interested to see you write:

    the more recent the information we obtain, the more reliable it is. For spatial this also holds true, in fact I would argue it is even more important

    There are so many benefits from receiving information within a timely manner and that’s why it can be classed as “realtime”. I was recently on a realtime web panel discussing “The Right Place At The Right Time: How The Real-Time Web Influences The “local” World” at the Local Social Summit and we covered the frequently discussed question of “When does realtime matter?” You can see my write-up here. We covered realtime offering convenience and also opportunity, but not reliability. As you say, as long as you can trust the source then the sooner you get the information then the more accurate it is.

    A few other points/observations:

    Comet is actually an umbrella term. It tends to be incorrectly (IMO) synonymised with long-polling yet Comet servers actually use traditional HTTP polling, HTTP long-polling, HTTP streaming and WebSockets as their transport mechanism. Here are some examples.

    An HTTP Streaming solution removes the request/response process (well, significantly reduces the frequency of requests) so beyond the initial request the lag is negligible.

    There’s are fallback technique for older browsers that don’t support WebSockets (the technique is know as a polyfill). The most common one uses web-socket-js which uses Flash to add the WebSocket object to the runtime.

    I used to work for ESRI and now work for a hosted realtime service called Pusher so am very interested in seeing some demos and examples. In fact I wonder if you’d consider using Pusher and maybe we could work together on building a few examples? Feel free to drop me an email (phil@leggetter.co.uk/phil@pusher.com) if you are interested or fancy a chat.

  3. @Brian

    Not surprised it was talked about at the FOSS4G, seems to be one of those fashionable cool technologies at the moment, I wonder if it will last. I know the scalability of it is a big draw but it interests me just because it is so easy to setup and I don’t have to learn another language. I haven’t really played much so can’t really comment but I will look into setting something up.

  4. @Phil

    Thanks for the comments and the links, if you work for Pusher you sure know more about this than me!

    Real-time information is interesting because I think many people just assume that the closer to reality we can receive the data the better. This certainly holds true for some things (why wouldn’t you want stock information as up to date as possible?) but for other sources we likely need to filter/sort /clean the data and turn it into information before we push it out. Spatial data definitely leans towards the quantitative side of things which means it sits nicely alongside real-time delivery. Spatial data may need processing (which could likely be automated) but often we have a series of X, Y coordinates we need to push out and the sooner we do it the better. It is the information that goes along with the X, Y component which likely needs filtering.

    You mention mobile in your post too and these devices are now generating huge amounts of spatial data, most of it at the moment is not getting used intelligently. Saying that I don`t think we have even seen the start of truly harnessing the power of location on smartphones. Having things pushed to you based on where you are still in its infancy and something where real-time is going to be critical. If you are driving along the highway a 1 minute lag can make a huge difference as to whether a piece of information is of value to you.

    I would be interested in chatting to you to get some demos online too, I will drop you a line.

  5. Bala K. says:

    For the more technically minded I am going to look into setting up a test-case, I think I am going to use node.js and socket.io using FME Server’s notification service to trigger the push events on the server. Is this something you might be interested in?

    1) A push service which writes spatial data (in JSON) to a Canvas can work for us. Also, we may want to be able to paint some controls on that Canvas.

    2) How does file-downloads work via such push service?
    eg: If a user needs the shapefile on his workstation folder to be updated (using Web Socket protocol) by such notification service, will the technology you are using be sufficient? In other words, can an FME process running on a webserver write a shapefile to users workstation folder with such real-time protocol?

  6. @Bala

    Thanks for the feedback. Point 1 is definitely possible, not sure about writing it to canvas but pushing JSON is easy.

    With regards to point 2, you can’t really use the notification service or web sockets in this way. Streaming data directly to a workstation folder when using web based technology isn’t going to happen because of security, the user will need to be prompted to choose a location. I think you could use FME Server’s data streaming service to achieve what you are looking for though so feel free to contact me directly to discuss further.

  7. Bala K. says:

    I will check with my client and team what the exact need is. FME has capability to output to a server. Was trying to see if Html5 technology can be used to extend this server process to user workstation.

  8. @Stewart – great. Look forward to hearing from you and I’ll keep my eyes peeled for future posts.

  9. Bala K. says:

    Phil,
    There are 2 major HTML5/Javascript technologies I can think of.
    1) WebWorkers : To run the background process (like I/O) to write to a the user’s disk. This can be thought of as similar to the way “FME Writer” writes to the disk.
    2) Javascript(Eval)/Canvas : This can paint the JSON stream on a Canvas. This can be thought of as similar to the way “FME Visualizer” paints a screen.

    The advantage is we can have FME setup on a WebServer and that handles lot of user requests by streaming out the translation. I do not know if we have business use cases on this yet. Lot of people are on Thanksgiving Vacation. I’m putting out what I know.

    Thanks,
    Bala K.

  10. Hi Bala,

    These two scenarios are both feasible. The canvas scenario is already possible and would just require you to write a web application using the canvas technology and I talk about it here http://blog.safe.com/2011/10/the-future-of-styling-rendering-geographic-data-on-the-web/.

    The first one would require more thought and a specific set of requirements and I think you mean web sockets not web workers? At the end of the day FME Server has a REST API, how you call this and run translations receiving data back is up to you so I would imagine the flexibility of the product would allow you to do what you want.

  11. Bala K. says:

    No, I meant “WebWorkers” that use “WebSocket” protocol. WebWorkers are used to spawn background client-side jobs. There is an example of updating client-side database using that protocol. This can be modified to write to a storage. But I do not know if any browsers support that functionality.

    Anyway, please try the original testcase which should be a huge task given “node.js” is still beta. We are not using HTML5 yet given most of our users are still with XP. It would be nice to have such test-cases by the time we are ready to implement that technology.

  12. […] people expect to be able to easily use the most up-to-date data, all the time. (Stewart recently touched on the growing expectation of real-time delivery of data and how new web-based HTML technology can […]

  13. […] previously talked about bringing real-time data to the web using WebSockets. WebSockets make up one piece of the […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts