Here’s post number 5 on the SchemaMapper transformer. By now you should know all about this transformer, and be able to understand the concepts I’m talking about. If not, then please give yourself a refresher course starting at post #1.
I started writing this post to cover some aspects of the SchemaMapper we’d overlooked so far; and that’s what I have done. However, during the course of coming up with these examples, things got fairly complicated and all of the topics actually interacted with each other more than I expected.
So….. take that as fair warning that this post is longer than the rest – but all very useful if you’re intending to use the SchemaMapper transformer in a serious way.
For most of these topics I’m going to use the same example: translating a set of pipeline data.
I could be doing this for a number of reasons. Perhaps I am migrating all my data from one system to another, perhaps I am exporting data from my corporate GIS to supply as CAD drawings to some contractors, or perhaps I just wanted to merge a series of different source datasets. Either of these scenarios is realistic.
The point is that I wish to examine my source features and assign them different symbology, layers and attributes according to the status of the source dataset.
The disclaimer I’m adding here is that all my destination symbology is fake! Normally I would write to format attributes such as fme_color or autocad_color, or similar, and used values which were valid for those attributes.
However in the interests of making the examples easy to understand and not specific to any particular format, I’ve used made-up attributes output_color, output_style and output_width, and made up values such as Red, Blue, Green, etc. If I really wanted to then I could map these to the proper attributes at the end using a ValueMapper (but you don’t need me to show you that).
Remember that the Filter Field parameter sets up a sort of ‘where clause’ by which FME can match mappings in the lookup table to the features to which the mapping applies. So far we’ve only looked at a single clause, but multiple clauses are not much harder.
In fact an almost better reason to show this example is to demonstrate how the SchemaMapper can actually map aspects of the data other than schema.
So, the first part of my lookup table will deal with these field fields. I have six items defined in the lookup table:
- TypeAttr: The name of an attribute which defines the source layer
- TypeValue: Potential values of the source layer
- DiaAttr: The name of an attribute which defines the pipe diameter
- DiaValue: Potential values of pipe diameter
- StatusAttr: The name of an attribute which defines the pipe status
- StatusValue: Potential values of pipe status
When I select the lookup table as the SchemaMapper source the preview looks like this:
To make use of these I then set up multiple Filter Fields, like so:
So now I have a clause which says….
If DiaAttr = DiaValue AND StatusAttr = StatusValue AND TypeAttr = TypeValue
Or, to give one specific example…
If PipeDiameter = 6 AND PipeStatus = AsBuilt AND PipeType = Water
So, that’s how you define multiple Filter Field clauses.
Adding Attribute Mappings
In this scenario, one of the things I want to do is actually set values for the symbology of the features, rather than just their schema type and attributes:
- ColorAttr: The name of an attribute in which to define feature color
- Color: The value which to set the color attribute to
- LSAttr: The name of an attribute in which to define feature linestyle
- LineStyle: The value which to set the linestyle attribute to
- LWAttr: The name of an attribute in which to define feature linewidth
- LineWidth: The value which to set the linewidth attribute to
And this is a perfectly valid task for the SchemaMapper. But, although you might not realize it yet, the big question is, should I add these to my lookup table as COLUMNS or as ROWS?
Instinctively you (and I) would probably add them as columns, in which case the preview looks like this in the SchemaMapper wizard:
This looks OK; but then I’m only mapping three attributes. I could add more but as I do so my lookup table gets almost exponentially more complicated; and complicated isn’t good when part of the reason for using the SchemaMapper is so we can make changes simply and easily and outside of Workbench.
So, FME Best Practice says I should add each of my attributes as a separate row. This way I get many more rows in my lookup table, but it’s not as hard to decipher when I want to make edits. The SchemaMapper preview now looks like this:
Notice that color, width and style are now on separate rows rather than in separate columns on the same row. Also note that the only new columns are called SymbologyAttr and SymbologyValue, and that each combination of Type, Diameter and Status has three rows; one for each symbology attribute.
This works because FME doesn’t stop at the first match it finds, but carries on through and executes all mappings that it finds. And this technique makes the Mapping Fields dialog easier too, because even with multiple attributes to set I only need to define one mapping, like so:
When I run the workspace now I get (for example) this output:
The source feature here obviously represented a 10-inch Gas Pipeline, in its Design stage. Accordingly it has been assigned a symbology of a thin, red, dotted line.
Did you notice above how the Mapping Fields dialog did not have a value set for source attribute? This is different to the way I’ve demonstrated this before.
That’s because the task wasn’t about attribute renaming, but about setting values. So I defined the destination attribute (using ‘Destination Attribute’) and defined the value to set it to (using ‘Default Value’).
In fact, it’s possible to use virtually any combination of these three parameters, according to the situation you want to define.
- Source Attribute + Destination Attribute: Rename the source attribute to the destination (what we did in a previous post)
- Source Attribute + Destination Attribute + Default Value: Rename the source attribute to the destination and set it to this value
- Destination Attribute + Default Value: Create a new destination attribute with this value (what we did in the above example)
- Source Attribute + Default Value: Set the source attribute to this value, but don’t rename it
The thing here is that “Default Value” is a little misleading and you will probably understand it better if you just thought of it as “Value”.
Feature Type Mapping
OK. Now I want to include Feature Type mapping into the same Lookup table. But how should I do it? On the evidence of my previous examples, you might think that I would need to add two new columns, SourceType and DestinationType, and expose/create some new attribute with which to manage them through the Index Fields dialog.
That would be the proper way to do this.
However, because it’s merely a case of setting the format attribute fme_feature_type, the above example shows I can add these as new rows, just as I did for symbology, and not even touch the SchemaMapper setup. For example, I might add the row:
…which looks like this in the SchemaMapper preview:
See how I just have an extra row for Feature Type, and so my output Feature Types are all set, and can then be handled with a fanout or similar in the workspace.
Order of Mappings
However, all of the above becomes entangled with the order of lines in the lookup table. For example, given these two lines in my lookup table:
…what output_color will a matching output feature be given? Both lines have the same where clauses, so which one gets priority? Red or Green?
The answer is that the second line wins. That’s because – as shown above – FME will keep on applying mappings as it works its way from the top down of the table. So first the feature will be assigned the color red, and then this is overwritten by the color green.
You might want to be aware of this…
1) Just in case you make a mistake and duplicate a set of clauses by accident.
2) Because you can use it to set a “default value”. For example, what would happen if I put this line at the top of my lookup table:
Well, you could say this is a default or catch-all mapping. All features of type “Gas” will match this and be given the color green (regardless of size or status). But, further in the lookup table, features with a matching PipeSize and CurrentStat will get set to Red.
Therefore, in my output, all features which are coloured green are those which didn’t match a full filter and get turned to red. So green features will be an error which I need to fix – but at least I’ve uncovered the problem.
The key point is, though, that this will only work when the catch-all is ABOVE the other mapping. If it were below then it would reset anything which had already been mapped.
3) Because you can use it to do what I’m going to call “secondary mappings”. I only found out about this through experimenting with a lookup table like this:
OldType,Tree,NewType,Vegetation OldType,Hedge,NewType,Vegetation OldType,Grass,NewType,Vegetation NewType,Vegetation,NewColor,Green
Do you see what happens there? Certain features are mapped to type “Vegetation” and then type “Vegetation” is mapped to the colour green. I’m using the NEW type (rather than the old) and by doing so am saved from having to individually map Tree, Hedge and Grass to the new colour. I map them all to a new Feature Type and map that Feature Type to the appropriate color. But, of course, this only works provided the Feature Type mapping comes before the colour mapping.
As a consequence of this, you want to be really careful if you map an attribute back to itself – for example map from fme_feature_type to a new version of fme_feature_type. That’s because the change applies immediately and any following mappings need to adhere to the new value of the type.
And if you want to get really (really) complicated, then you would now have to consider whether the Index Fields dialog is applied before the Mapping Fields dialog, or if they are applied simultaneously a row at a time. But frankly, I don’t know and I didn’t dare find out. There are just some things which humankind was never meant to meddle with!
Oh oh! We’d better stop. The SchemaMapper Dragon is getting all dizzy….
Wait till next time Dragon, when we do it all dynamically!
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)