It’s been a while since I posted because of various deadlines for the upcoming User Conference. But when I can simply cut and paste someone else’s content (hey it’s not plagiarism if you credit them) then it is much easier!
So this quick posting is about new Dynamic functionality within FME2010. The very-nearly-complete functionality is now live in the latest FME2010 beta and it is very exciting to think of the possibilities.
In essence dynamic functionality is about reading ANY data and writing it out to ANY structure. This is a huge leap forwards since traditionally FME has always required a schema defined in advance.
The content of this post is mostly copied from a message posted to the FME User Group by Graeme Hiebert, a senior developer at Safe Software. I have only interspersed it with a few notes and screenshots where it helps to explain what’s happening. Click on the screenshots to get the full image.
The phrase “Dynamic Duo” in this post title doesn’t really mean anything, it just happens to have the word “dynamic” in there. I think it originally referred to Batman and Robin, but perhaps Safe’s Dynamic Duo are Graeme and Harbinder who are the developers responsible for this fine functionality.
Generating Dynamic Workspaces
In Graeme’s own words….
On Thursday evening [30th April 2009], build 6065 of the FME 2010 beta was uploaded.
This build contains what I would consider the first full implementation of support for dynamic schema. When a writer’s feature type is configured to use dynamic schema, it doesn’t have to define any attribute schema information: this is picked up from the reader at runtime.
For example, the following steps are all that are required to perform a simple SHAPE->SHAPE translation with reprojection:
1) Generate a workspace using the classic workspace dialog. Set it up for the source and destination formats as normal, but before hitting OK, check the new “Dynamic workflow” checkbox.
This will result in a workspace with a single reader feature type and a single writer feature type. The reader type is configured with “merge feature types” enabled, and the writer type is configured to determine its schema at runtime.
Executing this workspace will prompt for a value for the writer dataset, and then effectively copy the shapefiles from the source to destination.
[Mark’s Note: The benefit here is that you can now change the Source Dataset parameter to read ANY Shape file – and it will still read, transform, and write the data, with the proper attributes! One thing I did find was that I had to reset the Feature Types to Read parameter, which was still set to the original Feature Type]
2) To reproject [or otherwise transform] the features, you simply add a Reprojector to the single path from the reader type to the writer type.
3) In the words of Jeff Goldblum’s 1998 iMac ad, There is no step 3.
This isn’t much different from before, except that you now have only a single line to connect to.
Dynamic Destination Schemas
Something a little more interesting is going on in this workspace, however. The actual definitions for the schema on the writer feature types is being determined at runtime, based on the what the reader is providing. There is no actual destination schema information (attribute names and types) stored in the workspace, so changing the reader dataset will still allow the features to translate as expected.
This is similar to the dynamic mode of FME 2009’s GENERIC writer, with two important differences:
1) Dynamic schema definition is now available for virtually every writer. The only time to use the generic writer is when the format has to be selected at runtime. If the workspace will always be writing to SHAPE (as in this example), you can just enable the dynamic schema mode on the SHAPE writer’s feature type.
2) There is now a greater deal of control over how features’ types are named and how the source schema definitions are selected. [Since schema comes in threee parts – Feature Type, Attributes and Allowed Geometry – we now have THREE settings: one to define where the Feature Type definitions are to come from, another to define where the Attribute definitions are to come from, and a third to define where the Allowed Geometry setting should come from]
[Let’s pretend we want to rename the output Feature Types]. The above workspace performs the required reprojecting, but does not rename the shape files when writing out. The generic writer’s dynamic mode cannot be combined with a fanout as one might expect, because it doesn’t know how to find the source schema definitions once the feature type name has changed. [In fact, when you enable Dynamic Settings the Feature Type Fanout option is greyed out]
What we need to do is to tell the writer to take the feature type name from an attribute, but to use the original feature type’s schema information when defining the attributes of the shapefiles it generates. Normally it wants to select both of these automatically, from the original feature type of each feature being written.
When using dynamic schema, changing the feature type to come from an attribute name is as easy as opening the destination feature type properties dialog and pressing the “Customize…” button.
This brings up a modal window where you can define three aspects of the schema definition: where it gets the name of the feature type, how it determines which source schema type provides the definition for the feature, and whether the geometry type is taken from the source schema or forced to a particular value.
There are three ways to define the output feature type name:
automatic, fixed, or from an attribute. In this case, for example, we want the name to come from the attribute “_concatenated”, so we select the “From attribute” radio button and choose “_concatenated” from the popup list of attributes.
There is no need to change anything else in this dialog. The feature definition (attributes and geometry type) will still be chosen automatically based on the original feature type, so we just have to press the OK button to accept our changes.
[Mark’s Note: In effect this workspace is doing the dyanamic equivalent to a fanout. However, the same techniques allow you to set the output Attributes using an attribute, or to get them from a completely different reader called a Schema Reader… but more on that another time].
See for Yourself
If you want to see this technique in action then Graeme created a movie and uploaded it to the FME YouTube channel, FMEGuru. Here it is below. Very cool with rocking music. I may have to get him to make all our training videos!
This Edition of the FME Evangelist…
…was written to the tune of Bad Love by Eric Clapton. Man I wish I could play guitar even half as good as this.
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)