Dear FME users,
If, 66 million years ago, you were a dinosaur going about your usual dinosaur business, you would have been more than a little startled when your day was interrupted by a giant flaming meteor landing nearby!
In the same way, in not many days time, the release of FME 2016 and the FeatureWriter transformer promises to shake most FME users to their very core!
So let’s check it out and see what it does!
A Brief History of FME
In 1993 (or should we say, approximately 12 million minutes ago) FME was created. It was a format translation tool with a small – but growing – number of Readers and Writers. The image to the right is the earliest known fossil evidence!
In time it evolved to include data transformations between a Reader and a Writer, and we coined the term “Spatial ETL” (Extract-Transform-Load) to describe it.
But, because Readers and Writers were separate objects to Transformers, the DNA of a translation was limited to this E-T-L structure. You couldn’t, for example, have T-E-T-L-T because transformers couldn’t come before Readers or after Writers.
So, FME2011 included a FeatureReader transformer, to allow reading of data to occur mid-translation. And now, FME2016 brings us the even more important FeatureWriter transformer, to allow writing of data to occur mid-translation.
What is the FeatureWriter?
As you can probably guess, the FeatureWriter is a transformer that has all the parameters of a Writer and writes data:
The big advantage here – like in the screenshot above – is that you can carry on with more transformers after writing data. Previously you could use a shutdown script, or chain several workspaces together, but this is a far superior solution.
Anyway, the FeatureWriter parameters dialog looks like this:
The top part of the dialog is obviously similar to the Add Writer dialog – it’s where you decide the format and output dataset, plus set any specific parameters. In the above example we’re writing Shapefiles to the folder C:FMEData2016Output (btw, did you notice we renamed the format from “Esri Shape” to “Esri Shapefile” to match what most folk in the GIS world apparently call this format!)
The middle part of the dialog is split into left and right parts. The left-hand side lists the feature types we have defined and lets us pick which on to deal with. Notice that the list is the same as the input ports we have on the transformer. In fact you can add new Feature Types by simply making new connections to the transformer:
The right-hand side is like the feature type parameters dialog. It has the same tabs (General, User Attributes, Format Attributes, and Format Parameters). These should be almost identical to the proper Writer feature type dialogs, meaning you can set dynamic parameters and feature type “fanouts”. Here, for example, is the user attributes tab:
Automatic attribute definition is particularly useful here, as it automatically updates to reflect what information is being connected to it.
The bottom part of the FeatureWriter dialog deals with the output from the transformer, in terms of information about the writing process:
The attributes defined here contain the name of the dataset written, the feature types written, and the number of features written. Since we can do further transformation after the data is written, it makes sense that we’ll want to know this sort of information. For example we could pass the list of feature types to another transformer to work with, here an FTPCaller:
Notice in the above how the list of feature types is exploded, and then used in the FTPCaller to define the files to be uploaded.
FeatureWriter and FME Server
FME Server is all about automation and so chaining workspaces together (or wrestling with Python) is a common way to automate a series of steps interrupted by writing data. For that reason the FeatureWriter is likely to be even more important than on a Desktop environment.
Another automation method is notifications, and you might also be able to do away with some of those. For example, when you publish a workspace to FME Server, you can choose to register it with the notification service and set a Writer to provide content to post to a topic:
That same dialog won’t let you pick data written with the FeatureWriter. Is that a problem? No! With the FeatureWriter you wouldn’t need to do that. You could just follow up the FeatureWriter with an FMEServerNotifier transformer:
This is a great example of how our workspace design will change as we start to use the FeatureWriter. Thanks to De Wet for pointing it out to me.
Additional Bits
There are a few other bits of functionality in this transformer that you might not have spotted. These are:
Open Folder
The dataset parameter has an option equivalent to “Open Containing Folder” on a Writer:
We may also add this to the context menu for the transformer itself, which would be useful.
Help
Pressing F1 on the transformer opens help for the transformer itself. But you can open format documentation by pressing the (i) button on the Writer selection field:
Adding Feature Types
To manually add a new feature type use the New option in the Feature Types window. It’s equivalent to Writers > Add Feature Type on the menubar. The Import option here is the equivalent to Import Feature Types. We don’t need a “copy from Reader” option, because you can just connect up a Reader feature type in the canvas to do that.
Dataset Fanout
A feature type fanout is the same mechanism as in 2015. The dataset fanout is now changed to match. i.e. if you want to fan out by dataset then you can either just set the dataset path to an attribute value, or open a text editor to create a dataset path through whatever text manipulation you need:
Incidentally, if you look at the dataset fanout for a normal Writer, you’ll see that the method is now similar to this too. You don’t specify a prefix or suffix or folder name – you just pick an attribute or concatenate something together using a text editor.
Limitations
This transformer cuts right to the very heart of how FME works. The changes are vast and I expect that you – the user – will try to use it in ways that we had never considered. What I’m trying to say is that this is just the first implementation, and there may be some limitations for scenarios that aren’t yet covered. Some of the limitations I know about are as follows:
No Attribute List on Canvas
If you look at the FeatureWriter there’s no way to expand an input port and show a list of attributes, and so you can’t do manual attribute mapping by drawing connections. The developers say they will look at this, but I don’t know. You should be using the new AttributeManager transformer to do this anyway. Incidentally, the AttributeManager will pick up Output Attribute names from the FeatureWriter (i.e. it handily knows what the output schema is just as it would with a real Writer)
Case Matching Functionality
This is probably one of the bigger issues. Some format Writers automatically convert attribute names to UPPER or lower case. The FeatureWriter does not support this. For example, if I write the attribute ParkName to Oracle, when it is expecting PARKNAME, the Writer will automatically fix this. The FeatureWriter will not. PARKNAME would end up with a null value.
You can use the BulkAttributeRenamer transformer to sort out your attribute names prior to the FeatureWriter, but my contact in the development team suggested that, for now, you avoid using such formats with the FeatureWriter. The key ones would be PostGIS, Oracle, or SDE30. If you do try these formats, check and double-check to make sure you are getting what you want.
Redirect to Data Inspector
Redirect to Data Inspector doesn’t have any effect on the FeatureWriter (or vice versa). Even though the log reports the data is redirected, the FeatureWriter will still write it!
Basically, be aware that this transformer might have a few shortcomings for now. But when I first used it, a lot of the problems I thought it had, were really shortcomings in my own thinking. It’s the type of change that is going to affect completely how we use FME. If I find any more limitations in the near future I’ll update this post with them.
But remember that you don’t have to use it! You can stick to using the normal Writer functionality if you are more comfortable with that.
Ultimately I think all users will embrace this transformer sooner or later because…
The Impact and The Future…
Like the meteor that wiped out the dinosaurs, this transformer will have a huge impact!
For the first time you can encapsulate an entire translation (Extract, Transform, and Load) all within transformers. In fact, in future FME versions, it’s not impossible that we will remove all Readers and Writers, and have FeatureReader and FeatureWriter transformers only!
Life should be way simpler. Look at the Writers menu on the menubar and imagine how easier it will be with these in a single FeatureWriter dialog.
Of course, the extinction of Readers and Writers might be a frightening thought. It is to me! But, the death of the dinosaurs – as Wikipedia puts it – “also provided evolutionary opportunities”, and the same will happen here. Your workspace practices will likely adapt to use the new functionality, and be able to thrive in ways you had never thought possible.
For example, what might you do with a FeatureWriter contained inside a custom transformer? We think it could form the basis for a “Custom Writer” – in a similar way to the (criminally underused) custom format (.fds files). It would be a big help in handling web formats, so look for us to create that capability in the near future.
Non-FME Notes
- I like to get my terminology correct, but astrophysics is just confusing. From what I can tell, a big rock in space is called an asteroid. A chunk that breaks off is called a meteoroid. When it enters the Earth’s atmosphere, it becomes a meteor. Whatever impacts the Earth’s surface is called a meteorite. Unless it is really bright – or you are a geologist – when it is a bolide. It’s not clear to me if an asteroid that doesn’t break up is still an asteroid when it impacts, so I went with meteor.
- Google the number of minutes in 22 years and you get lots of different results. I guess it depends on whether or not you take leap years into account. But it’s 12.5 million, or near enough.
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
