Hi all,TopTenTips-TenYears

Having just completed my tenth year working at Safe Software (where does the time – and the hair – go?) I thought I’d pass on some of this experience in the form of the top ten FME tips of all time!

These are all either things I wish I’d been told about earlier, or aspects of functionality that I think more users could benefit from.

I’ve divided this article into two parts. This is part 1, which is my tips. Part 2 will highlight tips from YOU!.

EvangelistBanner7

EvangelistNumber10

Relative Dates

No, a relative date isn’t going to the movies with your cousin, it’s all about the ability to process dates and times that are contingent upon the time a workspace is run.

For example, I run a workspace and want to find all the records in a dataset that were updated in the last week. The question is, how does the workspace know what day it is, how can it calculate what the date was a week ago, and how can it tell whether a record is newer than that date?

Well let’s take two features. One is date-stamped as 20-Feb-2014, the other is 24-Apr-2014. I wrote this tip on 30th April, so the 24-Apr-2014 date is within the last seven days. First I’ll run them through a DateFormatter to turn them into an FME Date/Time (%Y%m%d). Then I create a new attribute (I’ve called it RelativeDate) and given it a value of “1 week ago”. I run that through a DateFormatter to turn it into an FME Date too.

Yes! That’s right, I can use values like “1 week ago”, “next Thursday”, or “tomorrow” and FME will recognize it and turn it into a proper date. I love that we can do that.

Then I can simply use a Tester to test whether OriginalDate > RelativeDate, which works because I’m using the FME Date format that can be treated as a number where the bigger the number, the newer the date (20140425 > 20140423, so it must be more recent).

The workspace looks like this:

TopTenTips-RelativeDates1

You can see from the processing numbers that it has correctly identified one of my original dates as being older than “1 week ago” and one of them as being newer.

So, relative dates. A great piece of functionality, in my opinion.

EvangelistBanner7

EvangelistNumber9

The First Writer

FME developers put a lot of thought into setting up the FME engine to run fast and efficiently, and sometimes knowing about these internal behaviours lets you figure out ways to maximize your benefit from them. One good example of this is Writer Order.

I wrote an entire blog article about this subject, and there’s also an article on FMEpedia, so I won’t go into it in full detail here. Basically our developers implemented a neat trick for workspaces with multiple Writers, by making the first Writer run more efficiently. You don’t need to do anything to get this to happen; it just happens anyway.

However, now you know that the first Writer has a better performance, what could you do to take the most advantage of that? That’s right! You would shuffle the order of your Writers so that the one handling the most data comes first.

How do you shuffle Writer order? In the Navigator window of Workbench. For example, here I move this second Shape Writer above the first, because I know it is going to be processing more data:

TopTenTips-FirstWriter1

It’s not just speed either, memory usage can also be improved. In this workspace this trick didn’t do much for speed (it was actually a fraction slower) but it did cut the memory usage in half! Also remember, it’s not just the number of features that makes for “the most data” – it’s also the geometry of those features and the attributes. A complex geometry and lots of attributes can make for a larger set of data, even if there are fewer features.

EvangelistBanner7

EvangelistNumber8

Functions and in particular @Evaluate()

I have a few tips here; about FME Functions in general and the @Evaluate() function in particular. But what is a function?

Tip 8a: Anything that begins with an “@” character in FME is usually what we call an FME Function. It’s a piece of functionality that goes to the core of FME and so underlies transformers (i.e. transformers use functions). There is an FME “Functions and Factories” document that covers all of the available functions.

One such function is @Area(), which calculates the area of the current feature. Undoubtedly the AreaCalculator transformer uses this function. In fact, I can prove it by (Tip 8b) copying an AreaCalculator from the canvas and pasting it into a text editor! Did you know you can do that? In this case I see some internal FME code that shows how the @Area function is used:

INCLUDE [                                                              
         if { ({Plane Area} == {Sloped Area}) } {                      
            puts {MACRO AreaCalculator_func @Area(SLOPED_AREAS,1)};    
         } else {                                                      
            puts {MACRO AreaCalculator_func @Area(1)};                 
         }                                                             
         ]

Neat, eh? Even if it doesn’t help you with a particular issue, I like this tip because it helps you to understand the underlying core of FME, and the more you understand the better you can be at using FME.

But the real reason for this talk of functions is (Tip 8c) to mention the @Evaluate function. This function carries out arithmetical calculations. It’s very useful because it provides a way of incorporating arithmetic calculations into a text editor.

For example, I’m reading a list of neighborhoods in the city of Vancouver, plus a list of public art, and I want to create a string that says:

"In <Neighborhood X> there are Y pieces of artwork per square m".

Can I do that in one transformer? Well, not quite. I need to reproject the data (it’s in Lat/Long to start with) and count the number of artworks per neighborhood (I’ve filed an enhancement request for the FeatureMerger to output the number of Suppliers as an attribute). But when it comes to constructing the string, I can use just the AttributeCreator. The question is, do I use the Text or Arithmetic editor?

TopTenTips-Functions2

Well, I will use the Text Editor, and incorporate any calculations using the @Evaluate() function, like so:

In @Value(NeighborhoodName) there are @Evaluate(@Value(NumberOfArtworks)/@Area()) pieces of artwork per square m

See the @Evaluate() function? That’s how I tell FME to carry out a calculation. If I left it out my output would literally include the string “@Value(NumberOfArtworks)/@Area()”. Also notice I’m using our friend @Area() and also the fairly well-known @Value() (which returns an attribute value).

Actually, I’m going to complicate this slightly by making the result per square kilometre (metres don’t really make sense) and rounding off the result to an integer:

In @Value(NeighborhoodName) there are @Evaluate(@round(@Value(NumberOfArtworks)/(@Area()/1000000.0))) pieces of artwork per square km

NB: That’s another tip (8d) there. If you want to get a floating point (i.e. non-integer) result, you can’t use integers in the calculation. That’s why I have the “.0” at the end of “1000000.0”. But! In FME2015 we’ve implemented an update so that you no longer have to do that. So in FME2015, 2/5 really will equal 0.4 (and not 0). I think that will make a lot of users very happy!

Anyway… Don’t know which functions to use? That’s OK. Look at the left-hand menu in the Text Editor. There is a whole list of them available:

TopTenTips-Functions3

Notice there are also string and math functions (of which Round() is one)

Hopefully this is a useful tip and introduces you gently to the world of FME functions. In case it’s of use, I’ve made the above workspace available as a template.

NB: That’s another tip (8e) in itself. I don’t think I’ve ever seen a user use an FME Template. We use them all the time at Safe; not necessarily as a “template” as such, but they’re incredibly useful for transferring a workspace, data, and other resources, all in a single file. Think of it as a way of zipping up an entire translation. When we send examples to our development team, we nearly always use a template. It’s the “single file” benefit that we are using it for (like in my link above). So give templates a try in Workbench and see if you can think of a way in which it would benefit your work processes.

EvangelistBanner7

EvangelistNumber7

Number of Features “Written”

There are various components in an FME translation, but usually it’s the trio of Readers, Transformers, and Writers.

Readers are components that read data. The amount of data read is logged in the FME log window/file:

TopTenTips-WriterFeatures1

Transformers are components that transform (restructure) data. According to the transformer type – and what action they perform – they too can log features in the log window/file, albeit in a way that can be quite cryptic:

TopTenTips-WriterFeatures2

Writers are components that write features to an output dataset. They too can log…. oh! Wait. This is different.

This is a bit of an oddity. When the log window reports “Total Features Written” it actually means “Total Features Sent to the Writer”. In other words, the data is sent to the Writer and the number of features logged at that stage:

TopTenTips-WriterFeatures3

However… writing the actual data is still to occur, and there are many reasons why a feature might not actually get written.

For example, in the above, the number of features in my output dataset is zero. That’s because I was trying to write polygon features to a Shape dataset that only permitted point features. The log does mention this:

TopTenTips-WriterFeatures4

…and it even provides a summary of the result:

TopTenTips-WriterFeatures5

But, still, that Features Written section of the log shows a count of 80 features.

So the tip here is to be careful of interpreting that number too literally.

Besides FME Writers rejecting features (where, yes, we could do a better job of logging) there are also curiosities related to writing to various formats. For example, if FME writes a feature, but the database it is sent to rejects it, then does that count as a feature being written or not?

You can also get more features in the output than is logged, not fewer. For example, if I write a line feature with 10,000 vertices to a format that only supports 2,000 vertices per line, what happens then? FME might actually divide the line into five separate features in order to write it, but then I have written 1 feature but have 5 in my output!

So, whenever you look at that number in the log window, have a little voice inside your head say:

MoreComplicated

…and don’t assume that what you read is exactly what you got!

EvangelistBanner7

EvangelistNumber6

Flexible Change Detection

When I say “Flexible Friend” you might – depending on who you are – think of this, or this, or even this! But I always think of FME as being – besides your best friend in the world of data – the ultimate in flexible.TopTenTips-Flexible1

So here are a couple of Change Detection tips which, besides being good in themselves, will hopefully help illustrate ways to make the most of FME by becoming more flexible in your thinking.

Here I have two parks datasets. I want to find out which parks appear in both datasets – based on a match on the ParkName attribute.

Thinking about it, I want to find a match, so the Matcher transformer is the one to use. But really, I’m looking for features that are duplicates, so I should use the DuplicateRemover. Or maybe the ChangeDetector to find the differences? Or even – and this is where you need to be really flexible in your thinking – the FeatureMerger:

TopTenTips-Flexible3

Tip 6a: In this case, the FeatureMerger is the first transformer I’d reach for. It’s not so much that I want to merge the data, than that the transformer nicely sorts out the merged from the unmerged data – thus showing me which ParkName features have a match. You’d need to test it to be sure, but I’d put money on the FeatureMerger having the best performance of all of these too. Plus it has a multitude of output ports that let me easily identify which features are which.

Here’s another example. Again I am testing two parks datasets, but here I’m testing their geometry for changes. Naturally I’m using the ChangeDetector:

TopTenTips-Flexible4

But is there a better method? Well, the FeatureMerger won’t work because it doesn’t do a match on geometry. The SpatialFilter will, and maybe that’s something to test, but here my tip is to use a transformer called the CRCCalculator.

Tip 6b: The CRCCalculator calculates a unique value for a given combination of geometry and attributes. The typical use – and I remember doing this in the days of magnetic tape drives – is to record a CRC number before and after a dataset is transferred. If the two CRC values differ then you know that the data has been corrupted during the transfer. However, that same ability makes it a fantastic means for change detection:

TopTenTips-Flexible5

I just calculate a CRC value for both old and new datasets then use my favourite FeatureMerger to compare those values (the help page for the CRCCalculator suggests the Matcher, but I’d stick to the FeatureMeger). Because the comparison is only taking place on a single – and quite small – attribute, the performance should be way quicker, and way more memory efficient, than actually comparing geometry.

As a bonus, I could save the CRC values for use next time. That way I don’t have to even calculate values for existing data.

Incidentally, if you’re wondering how CRC values are calculated I have to admit I don’t know! I’m sure Wikipedia does. But I do know something that Wikipedia doesn’t: there is an update coming in FME2015 for you to pick which CRC algorithm you want to use:

TopTenTips-Flexible6

Anyway, given the large number of users who do change detection, I hope these tips are of use.

EvangelistBanner7

It’s all about YOU!

So those were five (or more) tips I’ve learned during my time at Safe, and I pass them on for you to use.

The next five tips I think should come from users, rather than from me. So email me your favourite tips for FME – or tweet/comment below – and I’ll pick the five best ones for my next blog post. Remember to make them tips you think have benefited you most and that other users will be interested in.

I look forward to seeing what you come up with, and how closely it matches other tips I’m thinking of.

Regards,

NewBlogSignature

About FME AttributeCreator Change Detection ChangeDetector CRCCalculator DateFormatter Dates FeatureMerger FME Desktop FME Evangelist Functions Log File Performance

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)

Comments

2 Responses to “The Top Ten FME Tips of All Time! (Part 1)”

  1. Michael Habarta says:

    Hello
    What I feel is one of the biggest “cave at”s and has cost us a fairly reasonable amount of brain wrecking is related to the question: Whose attributes will survive a “FeatureMerger” ?

    It seems fairly obvious:
    If the Requestor carries an attribute it will “win” – so to say – against an attribute of the Supplier.

    If you want to be sure the Supplier wins, you have to rename or delete the attribute of the Requestor.

    If on the other hand you want to make sure the Requestors attributes are preserved, you keep them and everything is fine … it seems …

    Except for the following:
    You have multiple Suppliers, use a list and explode the list after the FeatureMerger …
    In this case the Supplier attributes win … and you wonder what happened !!!!

    Keep this in mind and save a lot of wasted debugging time …

    Michael

  2. Jan Roggisch says:

    Hi Mark,

    Thanks for this post, it taught me a few new things I didn’t know about FME. I especially enjoyed the bit about pasting transformers into a text editor!
    My tip would be one of my favourite classes of transformers in FME, “Creators”, which I find incredibly useful to improve productivity. A few examples:
    Creator: I quite often use this to quickly nut out a draft workspace that uses transformers I’m not very familiar with, or where I know the input data model but haven’t got any actual data available when I start. I can then simply set up a few of creators to help me get the transformation logic right and replace them with real data later.

    SQLCreator: being able to pull whatever I can express in SQL into FME as features is incredibly powerful, e.g. when working with a complex relational database model. This way I can quickly change my input data structure during testing/development without having to re-create datasets as files or tables, and wiring them in using proper readers.

    PythonCreator: this allows me to create features from pretty much anything I can work with in Python, e.g. I can inspect MXD files to find out which feature classes they reference and then identify all MXDs a particular feature class is used in.

    There are lots more examples of where creators are extremely useful for prototyping or speeding up workspace development.

    Cheers,
    Jan

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts