Hi all,
Having just completed my tenth year working at Safe Software (where does the time – and the hair – go?) I thought I’d pass on some of this experience in the form of the top ten FME tips of all time!
These are all either things I wish I’d been told about earlier, or aspects of functionality that I think more users could benefit from.
Relative Dates
A relative date is all about the ability to process dates and times that are contingent upon the time a workspace is run.
For example, I want to use a workspace to find all the records in a dataset that were updated in the last week. How does the workspace calculate what the date was a week ago, and how can it tell whether a record is newer than that date?
In the below workspace screenshot, an attribute called RelativeDate is created to store the text value “1 week ago”, which FME recognizes and cleverly converts it into a proper date. You could also use values like “next Thursday” or “tomorrow”. Check the documentation here for more examples of relative dates.
The workspace looks like this:
You can see that it has correctly identified one of my original dates as being older than “1 week ago” and one of them as being newer. A great piece of functionality!
The First Writer
FME developers work hard to ensure the FME engine runs fast and efficiently. Sometimes thinking like FME developers helps to maximize your benefit from FME. One good example of this is Writer Order.
When a workspace has multiple Writers, the first Writer runs more efficiently. To take advantage of that, put the Writer that handles the most data first. The order of Writers can be configured in the Navigator pane, like in the below screenshot where I’m putting the second Shapefile Writer first because it processes more data. Check out this blog and article to learn more about this.
Memory usage can also be improved. In this workspace, this trick didn’t do much for speed (it was actually a fraction slower) but it did cut the memory usage in half! Also remember, it’s also the geometry of those features and the attributes. A complex geometry and lots of attributes can make for a larger set of data, even if there are fewer features.
Functions and in particular @Evaluate()
Anything that begins with an “@” character in FME is an FME Function. It’s a piece of functionality that goes to the core of FME and so underlies transformers (i.e. transformers use functions).
One such function is @Area(), which calculates the area of the current feature. Bonus tip for geeks: You can copy any transformer from the FME canvas and paste it into a text editor to see what functions are involved. For example, copying the AreaCalculator reveals the use of the @Area() function. Neat eh? This tip helps you understand the core of FME and how you can better use FME.
The @Evaluate function carries out arithmetical calculations and is a useful way of incorporating arithmetic calculations into a text editor.
For example, I’m reading a list of neighborhoods in the city of Vancouver, plus a list of public art, and I want to create a string that says:
"In <Neighborhood X> there are Y pieces of artwork per square m".
I will use the Text Editor, and incorporate any calculations using the @Evaluate() function, like so:
In @Value(NeighborhoodName) there are @Evaluate(@Value(NumberOfArtworks)/@Area()) pieces of artwork per square m
The @Evaluate() function is how I tell FME to carry out a calculation. Also notice I’m using our friend @Area() and also the fairly well-known @Value() (which returns an attribute value).
Don’t know which functions to use? That’s OK. Look at the left-hand menu in the Text Editor. There is a whole list of them available:
Notice there are also string and math functions (of which Round() is one). In case it’s of use, I’ve made the above workspace available as a template.
NB: That’s a tip in itself. I don’t think I’ve ever seen a user use an FME Template.They’re incredibly useful for transferring a workspace, data, and other resources, all in a single file. Think of it as a way of zipping up an entire translation. When we send examples to our development team, we nearly always use a template. It’s the “single file” benefit that we are using it for (like in my link above). Give templates a try in Workbench and think of a way in which it would benefit your work processes.
Number of Features “Written”
This is a debugging tip, which you can get more of in the Best Practices trail in the FME Academy.
The amount of data read is logged in the FME log window/file:
According to the transformer type – and what action they perform – they too can log features in the log window/file:
When the log window reports “Total Features Written” it actually means “Total Features Sent to the Writer”. In other words, the data is sent to the Writer and the number of features logged at that stage:
However… writing the actual data is still to occur, and there are many reasons why a feature might not actually get written.
So the tip here is to be careful of interpreting that number too literally.
Besides FME Writers rejecting features, there are also curiosities related to writing to various formats. For example, if FME writes a feature, but the database it is sent to rejects it, then that doesn’t count as a feature being written.
FME supports hundreds of formats and all of them handle data differently. It’s easy to make mistakes when translating data to other formats. Check the log file, the feature counts, and the output dataset to verify that your translation has done what you meant to do!
Flexible Change Detection
To use FME well, you need to be able to think flexibly. To illustrate, we’ll look at a few Change Detection tips.
Here are two parks datasets. I want to find out which parks appear in both datasets, based on a match on the ParkName attribute.
In this case, the Matcher transformer is the natural one to use. But I’m looking for features that are duplicates, so I should use the DuplicateFilter. Or maybe the ChangeDetector to find the differences? Or even – and this is where you need to be really flexible in your thinking – the FeatureMerger:
Here’s another example. Again I am testing two parks datasets, but here I’m testing their geometry for changes. Naturally I’m using the ChangeDetector:
The FeatureMerger won’t work because it doesn’t match on geometry. The SpatialFilter will but my tip is to use a transformer called the CRCCalculator.
The CRC Calculator calculates a unique value for a given combination of geometry and attributes and the FeatureMerger has the best performance of all transformers. It shows me which ParkName features match. It has a multitude of output ports that let me easily identify which features are which.
The typical use is to record a CRC number before and after a dataset is transferred. If the two CRC values differ then you know that the data has been corrupted during the transfer. However, that same ability makes it a fantastic means for change detection:
I calculate a CRC value for both old and new datasets then use FeatureMerger to compare those values because the comparison is only taking place on a single – and small – attribute. The performance should be quicker, more memory efficient, than actually comparing geometry.
As a bonus, I could save the CRC values for use next time so I don’t have to calculate values for existing data. You can also pick which CRC algorithm you want to use with FME:
Quick Add Shortcuts
You can add transformers to a workspace by clicking on the canvas and typing the name of the transformer you want with Quick Add. Here are two tips to help you use it more effectively. The first is from long-time FME user, Jeff Konnen (@jaykayone): search for transformers by their middle letters.
If you want an AttributeRemover transformer and start off by typing A-t-t-r then you’ll get a whole bunch of transformers listed, like so:
However! If you start typing u-t-e-r-e-m (the centre part of the transformer name) fewer transformers will match. You’ll get a much smaller list, like this:
In addition to Jeff’s idea, my suggestion is to use CamelCase as well. Notice that some transformers are made up of a single word (like the Clipper or Snapper). Whereas, others are made up of multiple words (like the NeighborFinder or the RasterSingularCellValueCalculator, an extreme case!).
You can search for such transformers using the first letter of each word, like this:
It is very useful with common transformers like the AttributeFileWriter (AFW) and AttributeRangeFilter (ARF).
Finally, pressing the tab key in the Quick Add switches from transformer name to transformer search. You don’t need to use the search window in the transformer gallery.
Open Containing Folder
Certified FME Professional Ben Moseley reminded me of the tool “Open Containing Folder”, found in the context (right-click) menu for all feature types on the canvas and on any parameters that reference a file name:
This function opens the folder in a file system dialog (e.g. Windows Explorer) so that you can locate the file(s). This is a very useful function for locating output files. When I write KML, particularly, I’ll do this so I can double-click the file and open it in Google Earth. It’s a good time saver. Thank Ben, for suggesting this (and the tool in the first place!)
Another option you’ll see on that above menu is Inspect. That’s another great time-saver. It was Safe employee Steve MacCabe who suggested that.
Finally, Matthew Brucker responded to me about the “FeatureReader’s ability to cache features”. He says it is helpful when building a workspace because you don’t have to re-read the data set every time.
I didn’t even know this existed! So my final tip here is to always read the What’s New file, attend the yearly What’s Great webinars, and keep up with product updates. You’ll often find a new feature – like this one Matthew mentions – that pays off your time handsomely!
Creators
FME user Jan Roggisch mentioned a piece of functionality that I absolutely couldn’t do without: Creator transformers.
Whenever I’m creating a workspace where the transformation part is going to be the most difficult, I use a Creator transformer (sometimes with the AttributeCreator). This generates fake data to test the workspace on. As Jan mentions, it’s great when “using transformers I’m not very familiar with, or where I know the input data model but haven’t got any actual data available when I start. I can then simply set up a few creators to help me get the transformation logic right and replace them with real data later“.
I also find it great to drop down a Creator to create a null feature to start up a workspace where there is no input!
You could also save a Creator as a custom transformer to have a small source dataset without the need for a Reader. For example, look at this Creator (click to enlarge):
I converted the outline of the city of Vancouver to XML (using the GeometryExtractor transformer) and pasted that XML into a Creator transformer (note the XML tab). Then, I turned that Creator into a Custom Transformer called VancouverBoundary. Instead of adding a Reader, I simply use Quick Add to place a VancouverBoundary transformer whenever I need the outline of the city of Vancouver:
That trick saves a lot of time. You can do something similar with the SQLCreator transformer. As Jan puts it: “being able to pull whatever I can express in SQL into FME as features is incredibly powerful, e.g. when working with a complex relational database model. This way I can quickly change my input data structure during testing/development without having to re-create datasets as files or tables, and wiring them in using proper readers.”
Now you know the power of “Creators”. Check out the VertexCreator too.This is a new combination of many other transformers such as the 2DPointReplacer and 2DPointAdder.
Organized Workspaces
If that tip was a little complex (everybody loves XML), here’s one from Lars de Vries (@larsdevries) that’s easier to understand. It’s about Best Practices and organizing your workspace efficiently.
Prefix your bookmark/inspector names with a number to control their order of appearance. For example, here I’ve added numbers to the bookmark names to get them nicely sorted in the Navigator:
The other thing I do is to give matching names to bookmarks that are nested inside each other, for example here (click to enlarge):
There I have a main bookmark (called Boundaries). This contains three other bookmarks (Boundaries:Neighborhoods, Boundaries:City Grid, Boundaries:Land Boundary). The bookmarks are properly ordered in the Navigator window and much easier to navigate to by using this naming convention.
Lars also mentioned naming your Inspector transformers this way helps to order the output in the Data Inspector (which is an excellent tip). It was also once suggested to me (by a student in a training class) that you could number transformers to make them more identifiable in the log, like so (Click to enlarge):
You won’t necessarily get a nice list in numerical order (see the above where transformer 4 comes first) but it does help clarify log files. This also helps identify which steps are related to which transformers.
Lastly, Matthew Longman’s tip was about the ability to change the order of Readers in a workspace, by dragging them in the Navigator like so:
What advantage does this bring? Well, several transformers – like the Clipper – have the capability of improved performance, if you can guarantee orders will reach them in the right order. This ordering tool is one way to achieve that.
Others
I did get a few other tips, so I’ll put them together into an “others” category.
Yves St-Julien suggested the triangle as a useful item to be aware of, and I agree. The triangle appears on the corner of an object (like a transformer) when you drag it in Workbench. When the triangle overlaps a connection or input/output port, that too is highlighted to let you know it’s possible to drop the transformer and have it automatically connected.
I’d also suggest you try experimenting with dropping these onto input/output ports. It can make multiple connections at once and save time.
Final Thought: It’s all about YOU!
If you’ve ever come across Safe Software FME community events, user conferences or webinars, you’ll know we’re the type of company that tries hard to help our users. If we can help you, and help you become the spatial superhero that you are, then it’s a win-win for us all.
So here is my final tip: make use of all the support opportunities we present to you. Here are a few that you may or may not know about.
- Free Training: In addition to all of the tutorials, articles, and Q&A in the FME Community, all of our training is free at FME Academy.
- Support: If you’re stuck and the FME Community doesn’t have the answers you need, ask us in live chat or make a support request. For bigger problems, we have an awesome network of FME Certified Professionals who can help you solve your problems.
Tell us your ideas in FME Community’s Ideas Forum: Did you know that FME will read Shapefile datasets zipped up into a single file with the extension .shz? It’s not a formal specification, but a user suggested it. We figured it would be a useful update and so it got implemented! Never be afraid to suggest something that would make your life easier!
Well, that’s my FME Tips article complete. It was called a “Top Ten” but in fact I managed to include thirty-two tips from either you or me. I hope you find at least one of them useful!
Regards
Mark
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
