Address_downloaderToday we welcome guest blogger James Rutter to share how Surrey Heath Borough Council built a custom data downloader service. Surrey Heath already used FME Desktop for their workflows and wanted to improve their infrastructure by leveraging the cloud. This is a great example of what our users are accomplishing with FME Cloud. Plus if you’re interested in creating your own data downloader with Leaflet, James offers an excellent step-by-step guide to building the webpage, followed by a look at his FME Workspace and how he configured FME Cloud.


Recently at Surrey Heath we took the plunge and opted to implement FME Cloud to support what we’re doing with FME Desktop and our aspirations to build out more of our data and spatial capability on cloud services. This is in contrast to the old paradigm for local government of running and supporting everything on our own internal ICT infrastructure.

I should comment here that embarking on HTML coding was a learning curve for me as I’m not a developer. Every step was a result of a lot of Googling… which is why I thought it would serve some use to other FME’ers to provide an insight in how to do this. If you’re a coder, this is simple stuff… if not, read on!

Objective: Customized Data Download

I wanted a simple, reusable webpage that I could use to draw a polygon on a map, configure a couple of additional data options using checkboxes or other form input elements, and POST that to an FME Cloud data download service to return some data. My first attempt at this is letting me download geocoded mailing list address data in CSV format.

Infrastructure Overview

While we operate FME Desktop on our internal network, for the purpose of the post I’ll be talking mainly about our cloud services. Our HTML page / app is stored on a web-enabled folder in our Amazon S3 store. FME Cloud is… well, FME Cloud. The address data FME will retrieve is also in the cloud, stored in a Postgres database running on Amazon’s RDS service (and we used FME Desktop to get the data there in the first place). I began developing the HTML page as a standalone file on my desktop. I soon ran into CORS issues when trying to POST stuff to FME, so you will need to ensure from the outset that you host your HTML file on a ‘web-enabled’ location (in our case it was an Amazon S3 folder).

Steps

Building the webpage

Configuring FME

In Detail: Build Your Webpage

Get Sample Code from GitHub.

Create an Initial Blank Template

Begin by adding some pointers to javascript, css styling and frameworks needed to make this work in the <head> tags of the file and at the beginning of the <body> of the page. These include jquery, Bootstrap, Leaflet.js, Draw.js, Geodesy.js and OSMGeocoder.js.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

<!doctype html>
<html lang="en">
 <meta charset="utf-8" />
 <title>SHBC Address Downloader</title>
 <head>
 <link rel="stylesheet" href="http://code.jquery.com/ui/1.9.2/themes/base/jquery-ui.css" />
 <script src="http://code.jquery.com/jquery-1.8.3.js"></script>
 <script src="http://code.jquery.com/ui/1.9.2/jquery-ui.js"></script>
 <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
 <!-- Latest compiled and minified CSS -->
 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" crossorigin="anonymous">
 <!-- Optional theme -->
 <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap-theme.min.css" integrity="sha384-fLW2N01lMqjakBkx3l/M9EahuwpSfeNvV63J5ezn3uZzapT0u7EYsXMjQV+0En5r" crossorigin="anonymous">
 <!-- Latest compiled and minified JavaScript -->
 <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" crossorigin="anonymous"></script>
 <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet/v0.7.7/leaflet.css" />
 <link rel="stylesheet" href="http://leaflet.github.io/Leaflet.draw/leaflet.draw.css" />
 <link rel="stylesheet" href="http://k4r573n.github.io/leaflet-control-osm-geocoder/Control.OSMGeocoder.css"/>
 <style>
  <!-- html, body, #map {
  height:100%;
  width:100%;
  padding:0px;
  margin:0px;
  } -->
 </style>
 </head>

[/accordion]
[/accordions]

 

You can see some style tags in here referring to a div called #map in our html document. This styling controls the size of the Leaflet map on the page.

Initialize the App

Next you need to start your main <script> which is where all the action happens. Add the Leaflet map in first. In this instance it points to OpenStreetMap.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

<script>
 var map = L.map('map').setView([51.352915, -0.599373], 17);
 L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', {attribution: '&copy; <a href="http://osm.org/copyright">OpenStreetMap</a> contributors'
 }).addTo(map);

[/accordion][/accordions]

 

Add in the OSM Geocoder control which will let users search and navigate to locations.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

var osmGeocoder = new L.Control.OSMGeocoder({
 collapsed: false,
 position: 'bottomright',
 text: 'Find!',
 });
 map.addControl(osmGeocoder);

[/accordion]
[/accordions]

 

Then add in the Leaflet Draw plugin and do some configuration on it. Here we are essentially saying add new drawn features into the layer called drawnItems and we’re configuring the draw plugin to only show the button in the toolbar which lets users digitize a polygon. The rectangle, line and marker buttons will not appear. Here, allow Intersection is set to false so we remove the problem of users drawing self intersecting polygons.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

var drawnItems = new L.FeatureGroup();
 map.addLayer(drawnItems);
  
 var drawControlFull = new L.Control.Draw({
 edit: {
  featureGroup: drawnItems
 },
 draw: {
  polyline: false,
  rect:   false,
  circle: false,
  marker: false,
 polygon: {allowIntersection: false,
 drawError: {
   color: 'orange',
   timeout: 1000
  },
 showArea:true,
 }
 }
 });

[/accordion]
[/accordions]

 

Add Draw and Edit Logic

Once the Draw control is added, we then need to let users edit their polygon so we add an edit control in. Draw:false means the edit tool is greyed out until the polygon is digitized.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

var drawControlEditOnly = new L.Control.Draw({
 edit: {
 featureGroup: drawnItems
 },
 draw: false
});

[/accordion]
[/accordions]

 

So now we’ve got a map and some basic controls we need to make it do something. We are using the draw:created function to detect when a polygon has been drawn. When this happens we do drawControlFull.removeFrom(map); This will remove the drawing toolbar thus preventing further polygons being digitized. The edit control will become active. We’re using a function to capture the drawn polygon to GeoJSON (which is what we’ll be sending to FME) and then we’re going to write the feature to the developers console (use the menu options in Chrome or Firefox to open the developer tools) to see what we have.

Following on from that we’re going to create a variable called polygonarea which we’ll use a Leaflet Geodesy function called Lgeo.area to calculate its value. We’ll print this to the console to make sure we’ve got something useful (and you can use the measurement value in the console to roughly gauge how big you will allow users to draw a polygon) and then we’ll do a calculation on it. In the screenshot below you can see I’m checking it’s not over 250,000 sq m. If it is I’m going to pop up a warning notice which tells the user the polygon is too large. The warning notice contains a button which will reset the page (ie remove the polygon captured into memory) so the user can draw a new smaller one.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

map.on("draw:created", function (e) {
 var layer = e.layer;
 layer.addTo(drawnItems);
 drawControlFull.removeFrom(map);
 drawControlEditOnly.addTo(map)
 //Capture the drawn items to geojson
 var feature = e.layer.toGeoJSON();
 //write the geojson to the console to see what we have got
 console.log(feature)
 //get the area of the polygon drawn using Leaflet Geodesy plugin
 var polygonarea = (LGeo.area(e.layer));
 console.log(polygonarea)
 //Calculate if the polygon is over a certain size in sq metres and if it is put a
 //bootstrap warning up with a page refresh button
 //The javascript:history.go(0) resets javascript objects to zero (ie clears out the old polygon)
 if (polygonarea > 250000) {$('#submit2').after('<button type="button" class="btn btn-warning btn-lg btn-block"><A HREF="javascript:history.go(0)">Your Polygon is too large.</br>Click here to draw a new one</a></button>');}

[/accordion]
[/accordions]

 

Here we can see the polygon feature in the developers console and at the bottom we can see the area that has been calculated; in this case 3134.4 sq m.

Data Downloader Sample

Just before we look at the bit of the script that actually fires the stuff off to FME Cloud, we need to add a simple form into the page. Put this in below the script code. Here you can see I’ve added a couple of checkboxes as I want users to be able to choose between residential or commercial addresses in this case. It’s important you name your form input elements as these names get added to the query string in the URL that we fire off to FME (and it turn FME Cloud maps these to correspondingly named published parameters).

This form could contain a selector to tell FME what projection you want data in if you’re downloading spatial data or it could tell FME what dataset you want to download. It’s up to you how this is configured. The point of this form is to allow you to send published parameters to the FME workspace and it’s these parameters which will get tacked onto the URL query string when we POST our request to FME.

Here you might want to check out the sample code for the html file so you can see where in the file my form is. Below the form you can see I’m adding a small ‘data loading’ gif but currently it’s not displayed on the page. (We’ll display this gif when we hit the ‘submit’ button). Below that is our big ‘submit’ button which kicks the last part of our script into action. This is the ajax jQuery POST command.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

 <form id="userselection">
 <h4>Data Download</h4>
 <p class="help-block"> Please select the type of data you requre.</p>
 <div class="form-group">
 <div class="checkbox">
 <label>
 <input type="checkbox" id="comm_prop" name="commercial">Commercial Properties
 </label>
 </div>
 <div class="checkbox">
 <label>
 <input type="checkbox" id="res_prop" name="Residential">Residential Properties
 </label>
 </div>
 </div>
 </form>
 <div id='loadingmessage' style='display:none'>
 <img src='path_to_the_gif_you_are_using.gif' class="img-responsive" />
 </div>
 <button type="button" class="btn btn-primary" id="submit2" >Get your addresses!</button>
 <div class="bootstrapalert"></div>

[/accordion]
[/accordions]

Make the POST to FME Cloud!

So this is pretty cool and where we get into a bit of ajax and jQuery. The first line in the code below essentially says….when the submit button is clicked…do the following. First off we want our spinning ‘data download’ gif that we mentioned earlier to become visible. Next we are going to make our post. I’ve commented the code below so you can see what it’s doing but for the sake of clarity here’s a few observations.

First, the URL: component. Make sure you have opt_servicemode=sync&opt_responseformat=json set here. This tells FME to send its response back to the web browser in json format. Hopefully it’s going to send us a data download URL and we need to grab that and do something with it in our code.

Secondly note that there’s some ‘stuff’ appended onto the end of the URL with plus signs. This is using another cool bit of jQuery called a serialize command. This will take your form inputs (ie your published parameters) and tack them to the end of the URL. In this instance as I’m using check boxes in the form I’ll get something like ‘&commercial=on’ appended to my URL.

Thirdly we’re telling jQuery to create our data ‘payload’ as JSON and that the actual data itself will be a json representation of our polygon. This is where webhookinbox comes in useful. Before building your complicated URL directed to FME Cloud, go to webhookinbox and create yourself an inbox there. Copy the URL into your code and send your POSTs there to begin with. Go back and look at your inbox and you will be able to see the JSON that’s been sent. Make sure it looks like geoJSON (ie a bunch of coordinates!). I had a lot of goes at getting this right, mainly because I was fiddling around with other methods in my code to capture the polygon correctly but if you follow these instructions you should have no problem.

After the POST we are going to ‘capture’ what FME sends back to us. We create a new variable called mynewurl and we’re going to set it to the value that FME has sent back to the browser. This is your data download URL. We write it to the console again so we can look at what we have. Next we make that data download gif invisible again and finally we’re going to pop up a Bootstrap alert containing a data download button for the user which points at this data download URL.

[accordions title=”” disabled=”false” active=”” autoheight=”false” collapsible=”true”]
[accordion title=”Click to Expand Code”]

$("#submit2").click(function()
{
 // Pop up a spinning gif while we wait for FME to respond to our search request
 $('#loadingmessage').show();
 
 $.ajax(
 {
 url : "https://yourfmeserver.fmecloud.com/fmedatadownload/path_to_your_fme_workspace.fmw?opt_servicemode=sync&opt_responseformat=json" + "&" + $( '#userselection' ).serialize(),
 //ensure the form name in the serialize command above has a hash in front else it won't work!
 //also ensure your fields in the form contain a 'name' tag else again the serialise function won't work.
 processData : false,
 type: 'POST',
 dataType : "json",
 contentType: 'application/json',
 data: JSON.stringify(feature),
 success:function(data){
 //alert(data.serviceResponse.url);
 var mynewurl = data.serviceResponse.url;
 //Write the download URL that FME provides to the console so we can check it.
 console.log(mynewurl);
 //Hide the spinning download gif again.
 $('#loadingmessage').hide();
 //This creates a bootstrap alert after the submit2 button with the download URL that FME provided plus
 // a start a new search button which resets the page.
 
 $('#submit2').after('<div class="alert alert-success"><strong>Congratulations</strong> Your addresses can be downloaded <a href="'+ mynewurl + ' "><strong>HERE</strong></a><button type="button" class="btn btn-info btn-lg btn-block"><A HREF="javascript:history.go(0)">Start a new search</a></button></div>');
 
 console.log(data);},
 error: function(data) {
 $('#output').text(data.responseText);
 console.log(data.responseText); }
 }
 );
 }
 )
});
[/accordion]
[/accordions]

Configure FME Cloud

Not much to do here, just configure FME Cloud to deal with CORS correctly if it’s not set up already. Go to Manage > Administration > CORS and load a template and save. That’s pretty much it and now FME Cloud is CORS enabled.

FME cors dashboard

Build an FME Workspace to Handle the Data Download Request

There are only a few key aspects to this workspace. Most of it is self explanatory and can be deciphered from the image of the workspace which is annotated. The essentials are:

FME Workbench example

The SQL query I’m using looks like this:

SELECT * FROM your_database_table as ap
WHERE ST_Within(ap.geom,ST_GeomFromText('@Value(_geometry)',27700))=TRUE

You can see where I’ve used the FME placeholder @Value(_geometry) inside the SQL statement. Essentially this inserts the geometry coordinates we extracted using the GeometryExtractor transformer into the SQL query we’re running.

It’s a Wrap!

That’s just about it. Obviously you need to publish your workspace to FME Cloud and in the publish dialog make sure you set it to data download service. Ensure the base URL you are using in your webpage points to this download service name correctly and you should be up and running. Any (constructive) criticism on my methods here is welcome as are any ideas for expanding this. The finished webpage is shown below.

Address_downloader

Get Sample Code on GitHub

James Rutter runs the GIS and address management services at Surrey Heath Borough Council. He has an interest in combining the best of open source and proprietary technology systems and migrating the services they run to the cloud. Find him on Twitter @chobhamonian.

About FME Customers Data Download FME Cloud Leaflet

Tiana Warner

Tiana is a Senior Marketing Specialist at Safe Software. Her background in computer programming and creative hobbies led her to be one of the main producers of creative content for Safe Software. Tiana spends her free time writing fantasy novels, riding her horse, and exploring nature with her rescue pup, Joey.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts