Not too long ago I was asked to write a post about Custom Transformers from a novice user’s point of view. Well, sometimes it takes a while for such seeds to grow, and this was no exception. It was only when I recently stopped to consider just how important custom transformers are to FME, that I remembered the request and thought I would finally do something about it.
So here’s that post. In fact it’s the first of two posts. This one will outline the basics of what custom transformers do, and how to create them. Part 2 will focus on advanced technicalities like sharing, exporting, and linking.
I hope both posts will be of use to new users, but that even experienced ones will learn something; for example part 1 is good material for FME trainers wanting to improve their messaging to students. And users wanting to be Certified FME Professionals: I would expect you to know all about custom transformers, and to demonstrate that in your example projects (just a little hint there!)
FME and Data Transformation
OK, so if you haven’t realized it yet, FME is not just about format translations; it’s about transformations. That’s why the most commonly created workspace is from Esri Shape format back to Esri Shape format: users are using FME to transform data even when they don’t need to translate its format.
And some of these transformations can get quite complex. It’s not the user’s fault: they simply find that FME is so good it can do almost anything, and so they cram in a multitude of transformers trying to do everything.
At this point a new user has often mastered the basics, and the most difficult aspect of FME shifts from actually defining the transformations to remembering which bits go where and why. It’s very easy to place transformers into an empty workspace, but much more of a challenge to plug them into an existing workspace with a spider web of connections.
This is where Custom Transformers can be so useful.
What is a Custom Transformer?
Most workspaces use a sequence of transformers to carry out a particular process; using transformers in series is what we call it in the training course. A custom transformer is simply a way to take that series and squash it down into a single object on the canvas.
Simply select a bunch of transformers like this, right-click, and choose Create Custom Transformer (click to enlarge):
Give that transformer a name, category, and description like so (click to enlarge):
And on the main canvas the previous sequence is now replaced with this (click to enlarge):
The definition of the custom transformer is now embedded in a separate canvas tab (click to enlarge):
Custom Transformer Benefits
How does that help? Well, there are multiple benefits:
- Tidier canvas
- Easier duplication
- Improved maintainability
- Sharable components
The tidier canvas part is obvious. By compressing a section of related transformers (more on “related” later) I’ve made the original canvas much cleaner. You can imagine that once your canvas is congested with hundreds of transformers, cleaning it like this is a huge benefit.
Easier duplication refers to the fact that we’ve now made a re-usable module, so I can duplicate that sequence of transformers in multiple places in the workspace, simply by placing a new instance of the custom transformer. We can do this because FME kindly includes it within the Transformer Gallery to be used just like any other transformer (click to enlarge):
Improved maintainability comes about because all of the instances of that transformer share the same definition. Without a custom transformer, if I had the same sequence of transformers scattered about a workspace and needed to make an edit, then I would need to search out each sequence and make the same edit multiple times. However, with a custom transformer I can simply edit the definition and the changes are propogated to all instances of that custom transformer. Neat, eh?
Having created a custom transformer, I can now share it with other users who might find that particular sequence useful; but we’ll cover that in the next post.
Selecting the Transformers to Include
Above I remarked that the transformers compressed into a custom transformer should be “related”. By that I mean they should form some logical unit or component. The reasons should be obvious. If I’m creating the custom transformer to make the canvas more readable, then it helps to be able to put a label (name) on the custom transformer – and I can hardly do that if all its components are entirely unrelated like this (click to enlarge):
Likewise, if I’m creating the custom transformer to be a reusable component, then it requires the selected transformers to be related.
In the example under “What is a Custom Transformer?” (above) you may have noticed I used a workspace from a previous post on creating QR codes. I wanted to create a custom transformer to handle the QR code generation, and be reusable with any data.
I was very careful to exclude the first and last transformers in the workspace because they were not related to creating QR codes. Those transformers handle tasks specific to the data being used, and wouldn’t necessarily be required for other data (and in fact could cause problems when used on a different dataset):
I guess the idea there is to generalize the custom transformer as much as possible, so that it will operate on any dataset. Then you will have improved its reusability.
OK, that pretty much covers the basic why’s and how’s of a Custom Transformer. Now let’s look at a few features that aren’t as obvious.
Attributes and Custom Transformers
Attributes are perhaps the most confusing part of a custom transformer. When you first create the transformer, the definition shows all the attributes in the data and all is well (click to enlarge):
You do a bit more work and then *BANG* – suddenly the attributes are gone (click to enlarge):
OK, the workspace doesn’t literally go bang – although it might feel that way if you don’t know what’s going on. So why does this happen?
The reason is simple: you’ve added a second instance of the custom transformer to the canvas. Once you do that there is no guarantee the same attributes will be available. This example illustrates the point (click to enlarge):
Here, the custom transformer was originally created connected to the parks dataset, which has name and name_alt as its attributes. Now there is a second instance connected to the emergency dataset, which has a different set of attributes. FME now has no way of telling which attributes may (or should) appear in the custom transformer and so hides them from the definition.
By hiding them from view there is no chance you will accidentally use an attribute in the definition (eg name), that won’t be provided in all instances.
Of course you, as the workspace author, may know for sure that each instance of the transformer WILL be provided with a certain attribute. In that case you can make the attribute re-appear in the custom transformer definition using an AttributeExposer transformer (click to enlarge):
So, the idea is that the author confirms an attribute exists (by using the AttributeExposer) before it can be used in the custom transformer.
In case you noticed – the second dataset does have the attribute but it is upper case (NAME, not name). FME is very case sensitive so, if you were banking on that attribute, an AttributeRenamer would need to be employed to fix the issue.
Luckily, the above is just an example. Since I don’t actually use the attribute name in the custom transformer, I’ve no need to be worried about its disappearance, or have to expose it.
Published Parameters and Custom Transformers
Published parameters can be a little confusing too. Normally, once a parameter is published the end-user is prompted for a value at run-time. In custom transformers, this is a little different.
When a parameter is published in a custom transformer, you can look at it as prompting the main workspace for a value; i.e. it appears on the parameters dialog for the custom transformer, but it doesn’t get used as a prompt at run-time.
To prompt for a custom transformer value at run-time, you first create a published parameter within the custom transformer, then – back in the main canvas – locate that parameter in the Navigator window and publish it again.
For example, inside my custom transformer is a RasterReplacer transformer. It creates the raster output. I want the user to be able to decide what format to write, so I publish that parameter (click to enlarge):
This makes the parameter available in the custom transformer parameters dialog:
Now I can actually prompt at run time by locating that parameter in the Navigator window, and publishing it:
At the moment my custom transformer definition has an ugly flaw: can you see what?
You probably spotted that there are a whole bunch of new attributes on the output. However, none of those attributes are part of the output feature; they are all solely used inside the custom transformer.
To conform to FME Best Practice, I should really clean them up and not output anything that isn’t required by the user. To do that I’ll use a little trick. Firstly I add a prefix to all the attributes I’ll use inside the transformer; for example _QRCoder_x instead of simply _x
Now I add an AttributeExpressionRemover transformer at the very end of the custom transformer definition, to remove all attributes that start with _QRCoder*
The final step is an AttributeRemover to get rid of the errant _http_status_code, and then file a PR with the development team to add the option to rename that (which is the reason why I’ve had to use a separate transformer – curses!)
Still, when I go back to the main canvas, I now no longer export excess attributes from my custom transformer:
Deleting a Custom Transformer
One final section: how do I delete a Custom Transformer?!
Well first of all I could simply erase all instances from the canvas, like this:
But that’s not the complete answer, because the tab for the custom transformer still exists, so I will right-click and choose ‘close’:
Surely that has removed it, right? Wrong! I’ve removed all instances, and closed the tab, but the definition for this transformer is still embedded in the workspace, and therefore still exists. The only sure way to really erase this transformer is to delete it from the transformer gallery like so:
Now I can be truly confident I have erased all trace of this transformer!
OK, I think that’s enough for now. In a follow-up post I’ll cover input and output ports, linked transformers, how sharing works, and – space permitting – how to create loops.
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)