Apologies if you subscribe to this blog and get this post twice – for some reason the images were missing the first time around…
This post is timed (sort of) to appear with an article in the recent FME newsletter. That article starts out: “Love it or hate it, crowdsourcing appears to be here to stay”, which made me chuckle since I know the writer’s opinion definitely doesn’t fall on the “love it” side.
But crowdsourcing does provoke those extreme polar views, so the fact we now have an OpenStreetMap Writer in FME is likely to either excite you greatly or make you shrug your shoulders and say “meh”.
What you meh-sayers need is a good use case; and here I hope to give you just that, with a few guidelines for best practice when uploading data to OpenStreetMap.org.
Simple Additions to OpenStreetMap
There’s the famous quote from Margaret Mead which says:
“Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has.”
OpenStreetMap is a little like that. A group of committed people around the globe, each making small updates and edits which culminate in something great. Maybe not changing the world, but certainly mapping it in a way that facilitates change.
Edits are generally made through a mapping interface, and small amounts of data can also be uploaded. The challenge there is for users to get their source data into the OSM format.
Since the upcoming FME 2011 already has the new OSM format writer in beta, any FME user can now convert data into OSM in preparation for upload.
Non FME-users, well firstly hang your head in shame at not making use of such a great product(!), and then get over to the EasyTranslator demo we have available via FME Server.
Since we created it – and since it became so popular after Don’s blog post – we’ve added support for OSM formats. That’s both Reading and Writing; so not only can you convert data from another format into OSM (from GPX for example), you can also convert any OSM dataset you’ve downloaded to another supported format, such as DXF, Shape, GML, KML, etc.
Bulk Additions to OpenStreetMap
Despite the crowdsource nature of OSM, there is also the possibility that a single organisation with a large, existing store of geodata, could donate or add it to the common stock; but that’s always been tricky because of the format barrier (converting data en-masse to OSM XML) and the need to process the data into a structure that will actually be useful in OpenStreetMap.
But with the new OSM format writer now organisations can make use of all the facilities of FME to read data from a spatial format or database – in batch if necessary – and to write it out to OSM XML ready for uploading.
Best Practice for OSM Imports
Now you know what’s possible, what’s stopping you?!
Well, it’s important to be aware that OSM needs more care than just “dumping” data into it. Problems could occur when you import data without prior care or thought, and that just reinforces the impression that crowdsourced data has quality issues.
But, of course, as an FME user you have access to a fine array of transformation and filtering tools to let you properly structure data before loading it.
You can use FME to:
- Filter out data which is not relevant, or which is confidential
- Combine or split layers and geometry to be more appropriate
- Do change detection to check if what you want to upload already exists
- Tile (or otherwise ‘chunk’) the data into more manageable pieces
And even when you’ve done this, the best course of action is probably to load the data into a tool like the JOSM (Java OpenStreetMap Editor) to be really sure what you’re about to upload really is new and useful. If in doubt, always check the OpenStreetMap Import Guidelines.
Here’s a small example workspace I just created. It shows how one might use change detection to tell if a set of data already exists or not.
The scenario is that I have a set of parks data for Austin, TX, and wish to upload it into OpenStreetMap.
The first thing to do is select an area of data from OSM, and download it for comparison:
Having downloaded the data I can add it into Workbench. I figure I only need to check against data tagged as ‘Leisure’
Change Detection is going to be an inexact science here, because even if my data matches an existing park, it’s unlikely to match exactly in name and coordinates. So what I’ll do is use a SpatialFilter transformer to check for overlaps, and additionally put a buffer round the incoming OSM data to be especially sure.
Here’s the workspace:
As you can see, the first section is about getting the data into the correct coordinate system, and putting a buffer round the OSM data. The Reprojector transformers bracketing the Bufferer put the data temporarily into a dynamic, equal-area coordinate system, so I can use feet or metres as a buffer measure.
Once prepped, the data goes into the SpatialFilter transformer. The OSM data is the ‘Base’, and the new parks data is the ‘Candidate’ data. I need to check the parameters carefully here.
The tests are OVERLAPS, CONTAINS and WITHIN, but the key parameters are Base Type (I’m testing against multiple OSM base features) and Pass Criteria. Pass Criteria is important. I only need one OSM feature to overlap to be an issue, whereas the default criteria is “Pass Against ALL Bases” so I had to change this.
And, of course, I want the “FAILED” features, since the PASSED features are those which overlap the OSM data. The results (in the FME Universal Viewer) look like this:
The original OSM is in green, and you can see the orange-coloured buffer around it. Blue features are those that failed the SpatialFilter; i.e. they do not overlap with the OSM data. They will get written to my OSM output and can safely be uploaded to OpenStreetMap.org
Incidentally, the purple-coloured constant attached to my OSM writer is a new feature for FME2011. It’s the ability to use a published parameter as a constant. So when I run the workspace (using Prompt+Run) I am prompted to enter a “created_by” value…
And this is then pumped directly into an attribute on the OSM writer.
If you are tempted to look in the city of Austin OSM will you see these additional parks? Not yet. I didn’t upload the output because the source of my input data is so old; so old in fact that it includes an airport that no longer exists. I think that’s why the OSM community is so sensitive about data imports. There’s no way they can be as up-to-date as a person on the ground with their GPS.
But that’s no reason why a geonerd shouldn’t be able to contribute from his or her desk, as there’s no way a person on the ground with a GPS can cover the same amounts of data you might be able to supply via a data upload.
So fire up your FME and join that group of thoughtful, committed citizens; just exercise caution as you do!
And don’t forget…
There are a series of upcoming FME User Meetings in October 2010. Don Murray (our President), Mark Stoakes (head of Professional Services) and Craig Vernon (Director of Sales) will be visiting Regina (SK), Edmonton (AB), Denver (CO), Seattle (WA) and Victoria (BC) for a series of presentations and all sort of other fun.
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)