Hello FME’ers,
Hopefully, by now you’re aware that FME2016 Desktop has been released and is available on our website. If you didn’t notice, we went for a steampunk-type style (or cloudpunk as Don called it in our release webinar).
Either way, here are some useful links to explore before we get onto today’s topic:
- The FME 2016 Web Page: General information and a link to download 2016
- An FME 2016 Release Blog Post: Seven things to get excited about
- The FME 2016 What’s Great List: A more comprehensive list of what is new and what has changed for 2016
Having looked at those, or bookmarked them for later, let’s get onto the subject of this post: new test operators.
The Test Interface
The Tester and TestFilter are consistently at the top of the list of most-used transformers. If you want to know the top-25 most-used transformers… take a training course! It’s in there. I will say that the Tester is number 1 and TestFilter has risen up to number 11.
Additionally the test interface is used in other places in FME (for example the feature filter in the Data Inspector), so any time we add new functionality to the test interface, it benefits a lot of users. Although the updates for 2016 aren’t quite as mainstream, they will still be useful to many.
In short, the updates are new operators: Attribute Has a Value, Type Is, and Encodable In. Let’s look at them all.
Attribute Has a Value
The test interface already had tests for Null, Empty, and Missing. So you could check for individual types of nothing, but you couldn’t simply check if an attribute did have a value. You would have had to test “Attribute is Null AND Attribute is Empty AND Attribute is Missing” and use the negate option to invert all these:
In fact, that might even need to be an OR, rather than an AND. I’m not sure. However, with the new operator I don’t have to worry:
Basically, that one operator combines Not Null, Not Empty, and Not Missing into a single test. We’re not saying the value isn’t one of the other 700 ways of defining nothing (0, -999, etc), but it isn’t one of the three main types.
Type Is
The Type Is operator lets you select a data type and carry out a pass/fail test on that basis:
Above, for example, I’m checking to see if my ID field is set to numeric values. If not, then the test will fail.
You can see there are multiple data types to test against, including XML and JSON. I’m assuming those are going to be useful when you are handling the response from a web request and don’t know what format of data the server is going to return. You could test for XML and JSON and send each to a transformer appropriate for that type.
But mostly this will be useful for checking for bad features in your data. It (and the new AttributeValidator transformer) have allowed us to deprecate the AttributeClassifier transformer. I’ll be writing an article about the AttributeValidator and all its functionality shortly.
Encodable In
The Encodable In operator isn’t testing the type of encoding a string is being held in, it is testing to see whether the string could be converted to that encoding.
For example, here I test the names in the address database in our sample dataset to see if they could be encoded in UTF8:
The answer is yes (i.e. all features pass) because UTF8 is good for encoding special characters. So if I wrote this data to a dataset and set encoding to UTF8, the data would be correct.
However, here I’m testing the data against a more restrictive encoding, Windows 1252:
This time there are features that fail:
Looking in the Data Inspector I can see why:
…there are accents and Chinese characters in the failed features that the Windows 1252 encoding just won’t handle. Now I’ll know in advance that there is data that I would have problems writing in that encoding – and if I do try I can see that FME is right:
Basically, if you’re writing to a specific encoding, you can now use this option to ensure that all your attributes can successfully be converted to that encoding.
So, that was just a very quick outline of the new tests available in FME2016. Remember, that test interface is used in several places in FME, not just the Tester and TestFilter transformer, and you can use these operators anywhere they are available*
Hope you find these useful,
* I just found that the filter option in the Data Inspector isn’t working! Doh! It’s a high priority issue and it should be fixed shortly.
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
Leave a Reply