As the person who assesses all certification projects, I think I see more than my fair share of customer workspaces. Most, if not all, of the best projects, will use Custom Transformers. They help tidy a workspace and let you structure a project nicely.
Basically, if you aren’t using Custom Transformers, then you should be. Especially now we’ve made a couple of awesome updates in FME 2014; not to mention a few smaller – but useful – updates too.
What is a Custom Transformer?
OK, for a moment let’s pretend you’ve never used a Custom Transformer and explain what they are. Basically a Custom Transformer is a sandwich! It’s a container for a set of regular FME transformers, all stacked together in hundreds of different combinations, and represented on the canvas as a single unit. Here’s what one looks like:
…and here’s the group of regular transformers (call them the sandwich fillings) that it represents:
It’s like having transformers for Bacon, Lettuce, and Tomato, and combining them together in a single transformer called the BLT!
In FME terms (sorry, no more sandwich analogies) I like to think of Custom Transformers having two uses. Firstly they save space on the main canvas (as you can see above). Secondly they let you re-use the same components without having to redefine them. For example, in the above workspace I could use the RandomizedSampler multiple times on the canvas, each instance of which would share the same definition.
A third use is that you can share Custom Transformers with colleagues and friends (they’re not so great Christmas gifts, though, so my wife tells me) and in fact you can find the RandomizedSampler transformer in – and download it from – the FME Store.
Custom Transformer Schema Handling in FME 2014
Let’s go straight to what is one of the most confusing aspects of Custom Transformers: handling the attribute schema.
Consider this scenario. I frequently create labels in my data from an attribute value; for example I label parks with their name:
This is a perfect example of where I can create a Custom Transformer. That way I don’t have to keep placing the same sequence of three transformers again and again. Here, for example, I can also label Buildings with their address:
The problem is, the label for the parks is stored in an attribute called ParkName; the label for buildings is stored in Address! So how do I define my Custom Transformer in such a way that I can select a different attribute for different datasets? If I’m stuck with ParkName then I’ll get a red (incomplete) transformer icon, like above, when I use it with Buildings.
The preferred method (i.e. what we suggest you do) is use something called Published Parameters. But we think many users will progress onto Custom Transformers before they work with Published Parameters. So we made some updates to FME 2014 to make everything more user friendly. When you create the Custom Transformer now there is a new parameter; this parameter is asking: “do you want us to take care of the attributes automatically?”:
By letting FME handle the attributes you don’t need to worry about these things called Published Parameters. It’s all handled automatically so that whenever you use the Custom Transformer you will be prompted for what attribute to use. Now it can be used on my Buildings data as well as the Parks:
So the initial Custom Transformer definition is all set up to be used anywhere.
But say I want to make an edit to my Custom Transformer that includes a new attribute; for example, maybe I want to concatenate an ID number so the label is ParkID-ParkName (or BuildingID-Address)? How can I have FME handle the ID attribute without going back to the parameter on the Create Transformer dialog? Simple! Look where this red arrow (below) is pointing inside the Custom Transformer definition:
Yes! We now have a parameters button on the Custom Transformer input ports, something that is most definitely new for FME 2014. Click on that, and we’re presented with a list of available attributes to expose:
This dialog lists all attributes that are connected to the transformer, wherever it might be, so that’s why I can see attributes from both the Parks dataset and the Buildings dataset. Basically it’s a way to manage attributes without having to place AttributeExposer transformers and manually type names. So now I could expose ParkId (or BuildingID – either would work) and use that in my labeling transformer too.
When I do, FME automatically handles this and exposes the attribute back on the transformer parameters dialog:
There’s a similar button on the output ports, so you can decide what attributes you wish to emerge from the Custom Transformer. That’s nice because it lets you remove any temporary attributes – for example those that are used within the Custom Transformer, but which don’t need to be sent back to the main workspace.
So, there you are: New for 2014, all parameter and attribute references handled automatically by FME.
Schema Handling Explained for Advanced Users
OK, if you’re an advanced user, who’s made use of Custom Transformers before, you might be wondering what’s going on with the above functionality. In particular, how does it equate to exposing attributes manually and creating published parameters? If you really want to know, here’s how…
When I chose that parameter, in the example above, FME automatically created a Published Parameter for me.
What you might look for next is that parameter being used in the LabelPointReplacer (I did) but, surprisingly, that’s not where it is used. You might think it would reference $(PARKNAME) but it still references the attribute itself:
Instead, when I hover over the source (Input port) schema, I notice the attribute itself is being referenced by the published parameter:
So the functionality is more than creating a simple Published Parameter, and more than just exposing an attribute. It’s really some behind-the-scenes handling of the attribute references in a more comprehensive way.
Think about it. If we’d used the Published Parameter in the transformer then the user couldn’t use that attribute anywhere else (unless they knew how it had been replaced by the published parameter and how to reference that parameter). But this way the user can just use that attribute anywhere, and not have to worry about parameters or attribute references or anything.
For example, here they could put a Sorter transformer in, and use that attribute to sort by.
It might look odd to an advanced user, but that attribute would get replaced with (say) Address, if I used it on the building dataset. If it helps, think of this behaviour as being like an invisible ParameterFetcher transformer. Give it a try and as soon as you hover over the input port and see the tooltip, I guarantee you’ll understand what’s happening.
So, in summary, if you are an advanced user there’s nothing to stop you using Published Parameter, AttributeExposers, or ParameterFetchers as you did before. But if you’re a more novice user, or you just want FME to handle attribute references for you, then this new functionality is the way to go. It really simplifies the handling of schema in Custom Transformers.
Custom Transformer Versioning
New for FME 2014 we also have this fantastic ability to handle Custom Transformer versions.
This is an important tool for when you are sharing your Custom Transformers with someone else, or using them on FME Server, and therefore only Linked Custom Transformers can be versioned (although they can be then embedded).
Consider what happens if I edit a Custom Transformer that someone else is also using. Firstly other users might have an older version of FME and therefore not be able to use the edits I made. Secondly, they might not want to use my edits. They might be happy using the older version. These are things we’ve had to consider carefully for Custom Transformers we’ve uploaded to the FME Store.
So let’s see the new functionality in FME 2014.
Take the previous labeling example. I’ve used File > Export as Custom Transformer to export it as a separate file that I can share. The Workbench title bar and the transformer properties tell me that this is Version:1
Now I will close and re-open the fmx file (yes, I have to do this), make a change (say rename the output port to ‘Output), and then press the save button. When I do so I am prompted as to what to do:
I can overwrite the existing version of the transformer, in which case existing users will be affected by the changes I have made, or I can save it as a new version. Saving it as a new version does not create a separate file, it merely creates a new version in the same file and increments the version number:
Note, if I ever want to rename the transformer, or start again from version 1, then I can simply use File > Export as Custom Transformer and create an entirely new branch. Also I can only edit a versioned Custom Transformer when I am using the same or higher FME as its latest version; for example if I was only using FME 2013 I wouldn’t be able to edit a Custom Transformer with 2013 and 2014 versions, not even the 2013 part of it.
OK. Back in my original workspace (where the Custom Transformer was used) I now see that the original transformers are Version 1, but if I refresh the transformer list and place a new instance, then it is now Version 2.
Note, if I was using a version of FME incompatible with version 2, then I would only ever get version 1. Also, if I want to upgrade from version 1 to version 2 I would do that by simply right-clicking on the Custom Transformer and choosing the menu option to ‘update to latest version’; I could also do the same by deleting the existing transformer and putting a new one in its place.
And if I’ve embedded the transformer, then I could use right-click > Reload latest version, like so:
Linked (exported) Custom Transformers have long had the ability to be password-protected. However, that limited the Custom Transformer to always being used in Linked Only mode; it could never be embedded.
Now, in 2014 or newer, you can embed a password-protected Custom Transformer. This makes it easier to use the transformer in a workspace without worrying about external file definitions.
If you ever try to edit the embedded version then, just as if it were linked, you’ll be asked to provide a password:
We feel this is really important for scenarios like FME Server, where you are more likely to use embedded Custom Transformers than linked and yet need some form of protection against unnecessary editing/copying.
Sometimes you have a number of Custom Transformer instances in a workspace and want to change the status of them all.
FME 2014 includes an update that prompts you to do so. When I try to change one from Linked to Embedded, I get asked whether I want to do the same for all instances of this transformer:
It’s a small fix, but likely a useful one; if you are going to embed a Custom Transformer, you’ll probably want to embed all of them and prior to 2014 you could only do that one instance at a time!
So, that’s what we’ve done to improve Custom Transformers in FME 2014. OK, enough analogies about sandwiches – I’m off to make one!
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)