Happy New Year FME’ers,
Besides a new calendar, January traditionally brings with it a new set of FME updates. As usual we’re putting together a full “What’s Great” list, the difference being that we’re going to post it to this blog, rather than as a page on FMEpedia, so keep an eye out for that.
Of course, a new year is also associated with New Year’s Resolutions – intentions on how you are going to transform your life for the better in the coming year. So in that vein, here’s my top ten list of transformers updates for FME 2014 that you should definitely resolve to make use of!
GeometryFilter
The GeometryFilter transformer filters (divides) data according to the type of geometry it consists of. For example, I might want to filter out point features from a stream of data before sending the remaining line features to an AreaBuilder. So far, so good.
The problem for this transformer – and the trigger for this update – is that the number of supported geometries in FME has increased hugely in recent releases. A better geometry model is great, but it did mean this transformer had so many output ports it could cover almost the entire Workbench canvas just by itself!
So, for FME 2014, we changed the transformer so that not all ports are exposed by default; you can now open the parameters dialog and specify which geometries you actually want to filter.
For example here (click to enlarge) the user is choosing to expose only the ports for filtering Points, Lines, and Null geometry:
Tip: Do you notice the “collection” port on here? I just found out that this is not the same as an aggregate. A collection is an aggregate, but one with different geometry types (non-homogeneous, e.g. lines and points). If you have an aggregate of one geometry type (homogeneous, e.g. all lines) then it will emerge from the Line output port. So now you know!
Databases
Databases are a great way to store data, but can certainly get a bit complex to write data to. For those occasions when you want to do very simple updates to a database table, FME 2014 introduces two new transformers; the DatabaseUpdater and DatabaseDeleter.
If it’s not obvious, the DatabaseUpdater updates a row with a particular value based on a match column or WHERE clause. The DatabaseDeleter deletes row using a similar search technique. The emphasis here is simple. Use the various Writers to handle the heavy stuff and use these for basic updates/deletes.
GeometryValidator
Apart from a brief note to myself that performance on 3D features is greatly improved in this release, the main update I noted here is a new option to repair donut holes that are touching the polygon boundary, like so:
If you use the GeometryValidator (Self Intersections in 2D test) on such data, the donut is now turned into a simple polygon feature whose outer shell skirts round the hole. Admittedly the data is still problematic – there is an obviously duplicate point in there – but that might still be preferable. Why? For the same reason most people use this transformer; spatial databases vary in what geometry is permitted and a duplicate point might be considered valid while a touching donut shell might not be.
Tip: For further reading, this great blog post talks about how a polygon like this is considered valid by one system but not another.
GeometryReplacer
This is only a minor update, but I’m sort of pleased to note that this one was my idea. The GeometryReplacer, you’ll remember, is a transformer that turns a text/attribute representation of geometry into a real feature. For example, if I read some data from a web site with the HTTPFetcher transformer, and it includes a field of geometry encoded in GeoJSON, I can use this transformer to extract that geometry.
As a more obscure example, some users will decant their geometry into FME binary using the GeometryExtractor, so that they have a backup copy they can restore later on using the GeometryReplacer.
Anyway, the update for FME 2014 is simply this: an option to remove the original attribute that contained the geometry:
Really the setting here is both a convenient way to remove this attribute (in the same transformer) but also a pointed reminder that you really should do so. Why? Well if you don’t remove this attribute when you are done with it, then FME is carrying the contents on through the entire workspace. This takes up processing time and can devour huge chunks of memory (attributes containing geometry are typically way, way larger than a standard attribute value).
In short, FME best practice suggests you don’t keep geometry in an attribute unless you really need it; this update both reminds and helps you to remove such data.
String Construction Updates
Since FME 2012 we’ve had what the French call “Paramétrage de Shelley” – the ability to construct parameter values from a mix of strings and calculations, named after the developer at Safe who first came up with the idea. Not all transformers got that behaviour – we figured they would not all need it – but we were obviously wrong as for FME 2014 we’ve added the functionality to a bunch of other transformers:
- StringSearcher
- Labeller
- LabelPointReplacer
- SectorGenerator
The StringSearcher lets you search in a constructed value, rather than just an attribute, like this user who is searching for the value UTM in a feature’s coordinate system (using a function in the integrated text editor). Of course you could also search in a set of concatenated attributes, or use a string function on an attribute, for example @UpperCase(@Value(AttributeToSearch))
And in the LabelPointReplacer you can now set the label and label height through a text/arithmetic editor:
If there are other transformers with parameters you think that would benefit from this type of functionality, then do let us know through the usual channels.
MapnikRasterizer
Now to the top 5 updates. Number five is the MapnikRasterizer. This is a new transformer and will be one of my favourites (I’m sure) once I get the time to play with it!
This transformer is essentially a tool for creating pretty maps from vector data. It’s a rasterizing transformer that includes a huge amount of parameters for managing the symbology of the output. I can create line styles, line patterns, point markers, polygon hatching, text/labels – and more. I can also select an image to use as a background to the output.
It’s a massive improvement over the Mapnik/Python hack that I previously created, because the Mapnik plugin is all built into FME and there’s no scripting or manual mapping to do. Anyway, it’s probably easier to show a couple of example outputs from my colleague Dmitri:
From what Dmitri tells me you can create output like this with far fewer transformers than before – in one example he now uses 15 instead of 48 – so this new transformer is well worth investigating.
Point Cloud
FME 2014 seems to be a bit of a renaissance for point cloud transformers; not that there was even really a dark age before. I count eight new point cloud transformers in this release.
The first two transformers – the PointCloudExtractor and PointCloudReplacer – serialize point cloud data in the same way as the GeometryExtractor/Replacer does for vector and the RasterExtractor/Replacer does for raster; so you can now write point cloud data into a blob attribute and read it back again and convert it to a geometry proper.
The other six new transformers are all to do with components in point cloud data. A component is a particular value for a point cloud point; good examples would be an X/Y/Z coordinate, return intensity, color, classification, and metadata (such as date/time). So now we have a whole bunch of transformers to handle these components; to add, remove, copy, keep, change data type, and rename – just as you would a set of attributes.
Here’s the PointCloudComponentRemover being used to remove some unused components:
From what I understand, a big change in FME 2014 is the ability to have your own custom components (instead of just a fixed, pre-defined set). So you can use these transformers (for example) to add custom components, or rename existing ones to new names. I’m not a big point cloud user but I can see this is a big deal.
The other point cloud transformer news is not about new transformers, but existing ones. In brief, the Scaler and Offsetter transformers have been updated to work on point cloud features; so now you can scale and offset the data just as you would do for vector and raster datasets.
Web Services
OK, well you and I both know that web services are the up-and-coming thing for nearly all things GIS, and that at Safe we have many web enthusiasts too; so it should be no surprise we have new web-related transformers in FME 2014.
The S3Uploader and S3Downloader are simple enough in that they upload to and download from an Amazon S3 (Simple Storage Service) ‘bucket’. In this way you can upload data that is to be shared with other users, or download data that is being shared with you. For more info see this page on FMEpedia. Of course, we have a lot of Amazon integration now, with the SNSSender and SQSSender too.
The other new web transformers for FME 2014 are the WebSocketSender and WebSocketReceiver. Web sockets are a little more advanced than a simple storage device. They are meant to be for two-way communication for server/client through a persistent connection; i.e. a connection is opened between client and server and a number of messages are sent back and forth.
You can look up WebSockets on Wikipedia, and probably find a number of general examples too. For spatial data, I’m thinking vehicle tracking would be an obvious use, as the data being passed back and forth is relatively light. A tracking device on a vehicle could open a connection to a websocket server and send location data through the connection, like this example (click to enlarge) from our FME Server Websockets tutorial.
Of course, throwing FME into the mix means your Websockets possibilities multiply, because FME can do any spatial filtering/processing you might want to do, and communicate with the Websockets server.
For example, FME (Desktop or Server) receives location from a vehicle (maybe an aircraft) and does a spatial overlay to look up where it is and whose responsibility it is (maybe an air traffic controller?) – then with the WebSocketSender pings a message to a particular server for the controller of that area. When the aircraft departs that area then we recognize that and the connection is closed. Something like that. Hopefully that makes sense – I’m just jotting down some random thoughts based on what I know of the technology. But that’s what I like about working at Safe – we try and stay ahead of the curve, so that when something like WebSockets becomes important to you, we already support it!
Raster Slope Calculations
When I studied GIS at college/university one of the examples I clearly recall was a land-use analysis. We assessed a region for different uses according to vegetation type, climate, and slope. Trying to replicate that exercise in FME was always difficult because it’s best done using a raster grid and FME didn’t have a way to calculate slope on cells in a raster DEM.
Well, no longer is that a problem. In FME 2014 we have new transformers for calculating the slope and the aspect (direction of slope) for individual raster cells. They are the RasterSlopeCalculator and RasterAspectCalculator.
There are a couple of different algorithms to choose from and you can choose whether to calculate slope in degrees or as a percentage:
I’ve got this high on my list because it adds some possibilities you might not at first think of. Besides my classic land use example, I’ve seen projects that calculate exposure to the sun (for assessing solar power usage) and I suspect you could do those in FME now. I also wonder if you could use this on data other than a DEM; for example you might use it on a raster dataset of rainfall values to see how the rainfall is distributed (where a steep ‘slope’ indicates rapidly changing rainfall). Something to consider.
The FeatureMerger
The FeatureMerger! One of our most popular transformers, and my article about it (and when to use it) was one of my most popular posts.
The FeatureMerger got a complete renovation in FME2013-SP2, so I suppose if you aren’t using a service pack then that’s one thing to look at. I wrote about it for a previous post when the interface got updated to handle multiple keys. But in FME 2014 there are a number of new functions too.
Suppliers First: This is a new option that works like the Clippers First parameter (point #4) in the Clipper transformer. If the features are in the right order then this can be used to turn the transformer from Group-Based into Feature-Based, with all the performance benefits such a move brings.
Comparison Mode: This new option allows you to decide how the requestor and supplier keys are matched. For example, in the old FeatureMerger, values of “10.0” and “10” would not be a match because the method used was string based. However, you now have the ability to compare attributes as numbers instead of strings, and so “10.0” could be a match for “10” if that’s what you want.
Supplier Prefix: When attributes are merged from supplier to requestor there is the question of what happens when there is a clash of names; i.e. both the supplier and requestor have an attribute with the same name but different values. This new parameter lets you add a prefix to all supplier attributes so you don’t have to worry about a clash, both attributes are kept but with different names.
The official launch for FME 2014 is going to be in just a couple of weeks I understand, so whatever version we have online right now is about as far from beta as you can get and can be safely used (officially it is our “release candidate”) to try out any of these transformer updates.
As with other launches we’ll be running an online contest and I’m thinking there will likely be some good prizes again – so keep an eye out for that and enjoy FME 2014.
Regards,
Mark
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
