Hello FME’ers,NewTestOperatorsHeaderImage

Hopefully, by now you’re aware that FME2016 Desktop has been released and is available on our website. If you didn’t notice, we went for a steampunk-type style (or cloudpunk as Don called it in our release webinar).

Either way, here are some useful links to explore before we get onto today’s topic:

Having looked at those, or bookmarked them for later, let’s get onto the subject of this post: new test operators.

EvangelistBanner7

The Test Interface

The Tester and TestFilter are consistently at the top of the list of most-used transformers. If you want to know the top-25 most-used transformers… take a training course! It’s in there. I will say that the Tester is number 1 and TestFilter has risen up to number 11.

Additionally the test interface is used in other places in FME (for example the feature filter in the Data Inspector), so any time we add new functionality to the test interface, it benefits a lot of users. Although the updates for 2016 aren’t quite as mainstream, they will still be useful to many.

In short, the updates are new operators: Attribute Has a Value, Type Is, and Encodable In. Let’s look at them all.

EvangelistBanner7

Attribute Has a Value

The test interface already had tests for Null, Empty, and Missing. So you could check for individual types of nothing, but you couldn’t simply check if an attribute did have a value. You would have had to test “Attribute is Null AND Attribute is Empty AND Attribute is Missing”  and use the negate option to invert all these:

NewTestOperatorsAttributeHasValue1

In fact, that might even need to be an OR, rather than an AND. I’m not sure. However, with the new operator I don’t have to worry:

NewTestOperatorsAttributeHasValue2

Basically, that one operator combines Not Null, Not Empty, and Not Missing into a single test. We’re not saying the value isn’t one of the other 700 ways of defining nothing (0, -999, etc), but it isn’t one of the three main types.

EvangelistBanner7

Type Is

The Type Is operator lets you select a data type and carry out a pass/fail test on that basis:

NewTestOperatorsTypeIs1

Above, for example, I’m checking to see if my ID field is set to numeric values. If not, then the test will fail.

You can see there are multiple data types to test against, including XML and JSON. I’m assuming those are going to be useful when you are handling the response from a web request and don’t know what format of data the server is going to return. You could test for XML and JSON and send each to a transformer appropriate for that type.

But mostly this will be useful for checking for bad features in your data. It (and the new AttributeValidator transformer) have allowed us to deprecate the AttributeClassifier transformer. I’ll be writing an article about the AttributeValidator and all its functionality shortly.

EvangelistBanner7

Encodable In

The Encodable In operator isn’t testing the type of encoding a string is being held in, it is testing to see whether the string could be converted to that encoding.

For example, here I test the names in the address database in our sample dataset to see if they could be encoded in UTF8:

NewTestOperatorsEncodableIn1

The answer is yes (i.e. all features pass) because UTF8 is good for encoding special characters. So if I wrote this data to a dataset and set encoding to UTF8, the data would be correct.

However, here I’m testing the data against a more restrictive encoding, Windows 1252:

NewTestOperatorsEncodableIn2

This time there are features that fail:

NewTestOperatorsEncodableIn3

Looking in the Data Inspector I can see why:

NewTestOperatorsEncodableIn4

…there are accents and Chinese characters in the failed features that the Windows 1252 encoding just won’t handle. Now I’ll know in advance that there is data that I would have problems writing in that encoding – and if I do try I can see that FME is right:

NewTestOperatorsEncodableIn5

Basically, if you’re writing to a specific encoding, you can now use this option to ensure that all your attributes can successfully be converted to that encoding.

EvangelistBanner7

So, that was just a very quick outline of the new tests available in FME2016. Remember, that test interface is used in several places in FME, not just the Tester and TestFilter transformer, and you can use these operators anywhere they are available*

Hope you find these useful,

NewBlogSignature

* I just found that the filter option in the Data Inspector isn’t working! Doh! It’s a high priority issue and it should be fixed shortly.

About FME Data Validation FME Evangelist NULL Tester

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)

Comments

Leave a Reply

Your email address will not be published.

Related Posts