I discussed database updates recently, and mentioned that there are two general scenarios.

One way is when you receive a changelog – a list of updates to be made – and apply them to your control database. That’s a simple one, because you know in advance what has changed and therefore which features need updates.

The second way of updating is when you receive a whole new dataset, with no indication of what is new or changed. For that you’ll need to do change detection. Previously an FME Hub transformer by the name of UpdateDetector was commonly used. But in 2019 we have a great update to the ChangeDetector transformer that should make it your go-to transformer from now on…

Change Detection: What’s New?

In 2018 and earlier the ChangeDetector transformer compared an “original” dataset against a “revised” version and separated the features into Added, Deleted, and Unchanged.

However, what it did not do is identify records that had changed. If a record existed in both the original and revised, but one of its attributes was now a different value, then that counted as a new (Added) feature. This made it hard to carry out “upserts” (database updates where a record already exists) because it was harder to tell whether a feature was truly new or not.

In 2019, however, the transformer is capable of handling updated records. In fact the new transformer design shows this through an Updated output port:

So this is very important. It vastly widens the scope of change detection for this transformer.

However, there is the issue of matching original features to their revised counterpart. FME can’t decide a revised record has been updated, without an original record to compare against. This is done using a key attribute value, meaning that the 2019 ChangeDetector has new parameters to handle that:

Differences in the ChangeDetector transformer help with Change Detection in 2019

We’ll look at an example shortly. For now just notice that the parameters dialog has a parameter called Update Detection Key Attributes, through which to select an ID or key value.

Anyway, if that functionality sounds familiar, it might be because we already included it in an FME Hub transformer called the UpdateDetector…

Replacing the FME Hub UpdateDetector

The UpdateDetector was created to fill gaps in the ChangeDetector’s functionality. Judging by the number of downloads, it was very popular transformer. But now it is deprecated on the FME Hub:

The UpdateDetector transformer was used for Change Detection, but now the ChangeDetector is preferred.

The new ChangeDetector not only replaces this hub transformer, it exceeds it in both functionality and performance. The UpdateDetector will still function in an existing workspace, but we advise you to replace it with a new ChangeDetector.

Now let’s take a look at an example of how the new ChangeDetector works…

ChangeDetector Example

Let’s say I have an address database:

Besides that I have been given a new version of the data. I must now determine which address records have changed, so that I can push those changes to the database. I do that by simply adding a reader for each dataset, and passing it to a ChangeDetector:

Change Detection in FME 2019

From there I can see that 35 records changed between the original and revised dataset, with 2 new ones added. Also 13 records from the original data are absent from the revision. The majority of records are unchanged. Let’s look at the parameters I used:

  1. GlobalID is the update detection key. This is the attribute that tells FME how to match records in the Original data with records in the revised data.
  2. The selected attributes are the ones I am checking for changes. i.e. where two records have a matching GlobalID, check these attributes for differences.
  3. This flag tells the transformer to also check the geometry in a spatial dataset. Several advanced parameters control those exact checks (see below).
  4. This parameter defines a list in which to store the changes that occurred; for example which attribute values differed and how.

Just as the parameters are a little different, so can the output be…

New Change Detection Output

Because the ChangeDetector transformer now checks for matches – but not necessarily on every attribute – it’s possible that Original and Revised might count as a match, and yet not be totally identical. For example, attributes a, b, and c are a match, but d is different. The features are still a match though because you didn’t pick d in the Selected Attributes.

To handle that scenario a parameter allows you to output either – or both – of the matched features:

If you do output both features, then a Match ID attribute is added, so that you can identify which features counted as a match.

Additionally, by setting a List Name, the output features record the differences between Original and Revised. This record (for example) has two differences:

The list tells me that two attributes (OWNERNM1 and OWNERNM2) were modified with new values, whereas this list:

…tells me that the geometry of the feature was modified.

Also – just as the UpdateDetector did – this transformer sets the fme_db_operation attribute. Here is an example for a deleted record (was in the original, but not the revised):

This means that I can simply pass features to a database writer, specify the Feature Operation (fme_db_operation) and match column (here GlobalID again)…

…and my address database is automatically modified with updates, additions, and deletions as necessary.

New Tolerance Algorithm

You may have noticed one of the advanced geometry parameters called Vector Tolerance:

Despite the name, this is not the same as the tolerance setting I discussed previously for FME 2018. Why is this tolerance different? Because it’s not trying to find where two lines intersect and it’s not trying to adjust existing points. Instead it finds whether two features are within tolerance, using something called the Fréchet distance.

The Fréchet distance is a measure of the similarity of two spatial features (usually “curves”) derived using the distance between the two features.

The common – and simpler – explanation is that of walking a dog. Say you walk along a path with your dog on a lead (leash). You walk in a relatively straight line (the red line, A, below), but your dog moves from side to side, in order to sniff at trees (the blue line, B):

Geometric change detection with a Frechet distance

The question is, how long does the lead need to be for each of you to walk your respective path? In the above diagram the widest gap between dog and walker is marked F. If my dog lead is at least F length, then we can walk our respective path without pulling at each other.

This concept makes a great solution for change detection. In FME terms a feature is unchanged if the Fréchet distance between Original and Revised is less than the specified tolerance value.

We feel that this algorithm is an improvement over the past method. It works on more geometries and it also allows applying tolerance in lenient matching mode, which wasn’t possible before.

Fréchet? Which Fréchet?

Please feel free to skip past this part if you aren’t into computational geometry. However, for you connoisseurs of Fréchet distances, you’ll notice this is a True Fréchet, not the Discrete Fréchet (which only calculates distance between vertex points). There is also a Weak Fréchet and FME uses that when the Lenient Geometry Matching parameter is active:

A Weak Fréchet is when you say that the walker or the dog is allowed to backtrack their steps. In a True Fréchet each has to keep moving forward.

In this example the dog walker (A) keeps walking in a straight line. The dog walks straight at first (to b1) but then veers around and to the right (to b2). They are still moving forward along their path, but their path is in a different direction:

In a True Fréchet the walker cannot reverse their path to account for this deviation. The most they can do is stop at their current position. For example here the walker minimizes the lead length by stopping to wait at point a1, while the dog follows their meandering path:

That, perhaps, is where the analogy breaks down a bit. Frechet distances are calculated knowing the path in advance, whereas it’s impossible in real life to know what course a dog will take!

Anyway, in a Weak Fréchet, the walker is allowed to reverse their course. When the dog starts heading from b1 to b2, the walker can double-back to say a2, to make for a shorter Fréchet:

If you don’t understand, you really shouldn’t worry. Just look at the two paths and remember that the Lenient Matching option means two features are more likely to be classed as a match.

More Change Detection Information

The above example looked at the new change detection behaviour. However, in some cases you might want to carry out the same Added/Deleted/Unchanged process that the old ChangeDetector did. i.e. you don’t necessarily need to look for modified records.

If that’s the case, then simply leave the Update Detection Key parameter empty:

Then your features are assigned to Inserted or Deleted, according to whether they were Original or Revised input:

And – of course – if you don’t have 2019, then you can still download the UpdateDetector. The term “UpdateDetector” now only appears in Workbench as an alias for the ChangeDetector though; so access the transformer through the FME Hub, being sure to check the option for Show Deprecated in the interface.

Additionally, the Matcher transformer got a small makeover for 2019. Its parameters dialog was refreshed and gained the same tolerance parameter and algorithm as the ChangeDetector.


So that’s what’s coming up for Change Detection. Now you’ve read this article you won’t be surprised to see the entirely new ChangeDetector dialog when you upgrade to FME2019.

I believe this is a really useful update, and I look forward to using it. Oh, and in case you were struggling(!) here’s the answers to the canine spot-the-difference image:

Happy FMEing,


About FME Change Detection FME Evangelist Spatial Analysis

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)


10 Responses to “Upserts and Dog-Walking: What’s New with Change Detection in FME 2019”

  1. Bruce says:

    Great enhancement, and an easy sell!

  2. Yonas B. says:

    Thanks for the heads up. Refreshing and informative.

  3. Nic says:

    Great blog post Mark. Love the dog walking analogy!

  4. Brilliant improvement and blog post. Thanks Mark, you’re probably one of the few persons who make computational geometry understandable 🙂

  5. Anne says:

    Where can I find information on how to write the records that come out of the “updated” port? Only the fme_db_operation attributes appear to pass through. Do I have to do this manually with the AttributeManager? Thanks

    • Mark Ireland says:

      That’s interesting. I see that if my Original data has an extra attribute, then it doesn’t appear on the Updated port output. I suspect it’s because we assume that it is a deleted attribute (i.e. that is a change) even if it’s not selected. If you want to keep them, then perhaps a FeatureMerger will help to join that information together.

      I hope this helps. If not, perhaps you can post this question to our community forums at knowledge.safe.com, where we can have a better conversation about what might be happening.

  6. Hayden says:

    I think there are some important things left out here that I have had issues with using this transformer:

    Unless you select “Match selected attributes” as type, the detector even matches on non-exposed attributes. for example, if you are matching two excel files, it will match on the hidden format attribute of excel row number. These hidden attributes are not listed as selectable attributes for matching or excluding from a match. This caused me great confusion until I saw the change list was referencing hidden attributes.

    When you give a list name, it does create list attributes but keeps them hidden. So you have to take extra steps to reveal list changes after the change detector tranformer.

    • Mark Ireland says:

      Hmmm. I see the documentation says this:

      “Match All Attributes: matching is performed on all attributes, including unexposed format attributes.”

      I think that’s a bit extreme, but at least we know it’s working as designed. In fact, one of the developers remarked:

      “We often have cases where many attributes are not exposed, or not known until runtime, so there needs to be a mode where it matches unexposed attributes.”

      Again, I think format attributes is a bit extreme, but justifiable. You could always try a BulkAttributeRemover to get rid of all format attributes before carrying out the change detection. Or switch to selected attributes only, if possible.

      I tried the list option in 2020 and it does expose the lists (though not every element in the list – which it wouldn’t normally do anyway). Perhaps this is a new fix and you’re using an older FME? Alternatively, if you can’t see the list on the Updated features, it might be better to post this as a question to the FME Community (knowledge.safe.com) where others can help and we can have a proper conversation.

  7. David Mooney says:

    Hi Mark,

    Thanks for the information, I am wondering if you could shed some light on a problem I am having involving the Change Detector.

    What I am working on:
    1: I have Smallworld (SW) records in an alternative that have been changed in some way, Updated / Inserted / Deleted.

    2: I am currently use FME to transform the data from SW’s EO datamodel to SW’s DM datamodel and then wanting to write it to an Oracle DB, that matches the DM datamodel.

    3: I am using a Change Detector to compare the records in the alternative to the corresponding Oracle DB table

    4: I am than Inserting, Updating or Deleting that data in the corresponding Oracle DB table

    I have 20 SW records in the alternative that need to be transformed and then Inserted, Updated or Deleted from the Oracle DB.

    When I run the workspace the change detector reads in all of the records from the Oracle DB table ( lets say 10,000 records ) and compares them to the 20 SW records. The change detector than “moves” ~ 9,980 records to the Delete port and the remaining ~ 20 to the Insert, Update, Unchanged ports. It could be ligitimate than some of those 20 records should be deleted but I cannot tell with this output, as well its a bit slow when I really only wanna see what needs to happen to 20 records and not 10,000.

    – How can I use something like a changelog ( or other techniques ) to make this more efficient?
    – Could I query the Oracle DB before hand for records with matching ID’s and only compare those?

    • Nicole Lee says:

      Hi David! This is a great question for the FME Community. I think many community members have the same question. Do you mind posting it there too?

Leave a Reply

Your email address will not be published.

Related Posts