Hi FME’ers
When we introduced dynamic functionality (around the time of the last user conference) I received feedback that the term dynamic caused some confusion, particularly in comparison to the term generic. Now (in FME2010) that a dynamic option appears right on the Create Workspace dialog, it’s even more important that every FME user be clear about the different tools, and how – and when – to use them.

So I’m writing this post – a condensed version of what will appear in our training course – to give you a clear and consistent story on the subject.

And yes, once I’ve done this hopefully I’ll get around to that “dynamic workspace with SchemaMapper” that I’ve been promising for a while!

Dynamic vs Generic Functionality
Let’s start clarifying dynamic and generic by stating a common interest: both sets of functionality are designed to manage Data Schemas.

Traditionally FME workspaces have always required schema to be defined in full, and are tied to that schema. Generic and Dynamic functionality is designed to create workspaces which translate data regardless of data schema.

In brief, schema consists of Format, Feature Types and Attributes. Generic functionality takes care of the Format aspect of this. In other words they will read and write data of any format, without the workspace author defining that format in advance.

That leaves Feature Types and Attributes for Dynamic Functionality to take care of. In other words they will read and write data without the workspace author having to define the Feature Types or Attributes in advance.

Basic Dynamic Workspaces

Functionality Dynamic Workflow
Degree of Difficulty Easy
Purpose Content Transformation
Example Use-Case A workspace to read data with any schema (layers/attributes) and carry out a transformation such as coordinate system reprojection, before writing the same data as output.

Let’s start with a really simple case; the one where you tick the ‘dynamic schema’ option in the Create Workspace dialog.

As we say in our FME Desktop training course, you can transform either the structure or content of data. This example is purely content transformation.

Firstly let’s look at a traditional (static) workspace (below). The layout replicates the schema of the source data in a graphic visualisation.

Above: Feature Types in the source are shown separately and with attached attributes.

The equivalent dynamic workspace is this (below). Notice that the layout is very general, and does not attempt to depict the schema of the data. Notice also that all source attributes are merged onto one Feature Type, but that there are no destination attributes defined.

Above: The same workspace with the dynamic option checked

So, apart from a more compact workspace, what does the dynamic option buy me? Well, the problem with the static workspace is that I cannot read any datasets whose schema does not match that shown in the canvas. With the dynamic workspace I do not have that problem. You might say that the schema representation is universal, and so it will read (and write) any data.

Now (below) I can change the source (an SDF reader) to point to any dataset of that format, regardless of what layers (or tables) and attributes it contains. The one ‘gotcha’ is that I have to empty out the ‘Feature Types to Read’ parameter, which will otherwise restrict my reading to the three previously-defined types.

Finally, the obvious points are that a user can use this workspace, for any data, without having to manually make edits – which is great for them. Also, if I used the Generic Reader and/or Generic Writer, in a dynamic way, I could produce one single workspace capable of reading/writing any format of data, regardless of what schema it has.

Advanced Dynamic Workspaces
I claimed that the above example will read and write any data in a single workspace, and it will. However, the output is always a mirror image of the input, because the destination schema is copied dynamically from the source.

Some users will want to map the source data to a new schema, and write that dynamically. As we’d say in FME training, the “What you Want” side of the equation is not the same as “What you Have”.

As the above example stands, no changes you make will be applied because the output schema is still a copy of the input.

That’s because of this setting on the destination Feature Type (below). Notice how it defaults to selecting schema from the original source dataset.

So, if we don’t want to simply duplicate the input schema what are the options? Well, we can simply point it to another schema, and the two ways to do this are shown in the following examples.

Advanced Dynamic Example 1

Functionality Dynamic Schema Source
Degree of Difficulty Medium
Purpose Structural Transformation
Example Use-Case A workspace to read data with any schema (layers/attributes) and carry out a structural transformation in order to write the data with a different output schema.

Say the desired schema already exists in another form, a form which can be read by an FME reader. A classic example would be a database which is pre-defined and ready to receive data. You can add this schema to the workspace using a Workspace Resource.

Here my destination schema is already defined within a database, so I load it into the workspace as a Reader Resource using this function (below):

When added I get an entry like this in the Navigator window (below):

Now I point the destination Feature Type to use this schema, not the source (below):

In effect I’m saying: “Hey FME! The output of this workspace? I don’t want it to look like the input. I want it to look like this dataset”.

Now I just have to make sure the workspace contains the correct mappings to match the new destination.

This is a great fix for the scenario where users are uploading data through FME Server, and you want to force it into a consistent schema.

Of course, you could do it statically if you wanted, but by working dynamically you’re pulling in the new schema definition at run time, dynamically reading the source data and writing it out according to ad-hoc rules.

Advanced Dynamic Example 2

Functionality Dynamic Schema Source
Degree of Difficulty Difficult
Purpose Structural Transformation
Example Use-Case A workspace to read data with any schema (layers/attributes) and carry out a structural transformation using an external lookup table.

In this scenario the desired schema exists in another form, but isn’t one which can be readily interpreted by an FME reader. An example here would be a simple text or CSV file.

In future (FME2011?) we hope to have a better solution for this scenario, the main stepping stone being how should a schema be easily defined in such a basic format. As far as I know there isn’t a standard for this type of information.

So for now you handle this using a Custom Format as a reader resource. Aaron Koning calls these “Blueprint Readers” and here’s how they work.

Basically the trick is to return data from a Custom Format in the same structure as the new Schema Reader in FME. The Schema Reader returns:

fme_feature_type_name
attribute{}.name
attribute{}.fme_data_type
fme_geometry{}

So, if my CSV format schema definition looks like this:

Layer,AttributeName,AttributeType,GeometryType
Parks,ParkName,fme_varchar(20),fme_polygon
Parks,AlternateName,fme_varchar(20),fme_polygon
Parks,ParkArea,fme_decimal(8","0),fme_polygon
Parks,AverageParkArea,fme_decimal(8","0),fme_polygon

…then I create a custom format to read that CSV file, and to output a single feature with:

fme_feature_type_name = Parks
fme_geometry{0} = fme_polygon
attribute{0}.name = ParkName
attribute{0}.fme_data_type = fme_varchar(20)
attribute{1}.name = AlternateName
attribute{1}.fme_data_type = fme_varchar(20)
attribute{2}.name = ParkArea
attribute{2}.fme_data_type = fme_decimal(8,0)
attribute{3}.name = AverageParkArea
attribute{3}.fme_data_type = fme_decimal(8,0)

In this case my single Custom Format looks like this (below):

Then I just have to set the Custom Format parameter “Source Provides Schema Features” like so (below):

…and I can add it to my original workspace as a Reader Resource like this (below) and change the dynamic schema setting to point to the new resource (as shown previously).

When I run the workspace the log window warns me that:

'27' features with feature type 'Rivers' were ignored because no existing DEF line was found
'108' features with feature type 'BusLines' were ignored because no existing DEF line was found

…but that’s OK – I haven’t defined these in the lookup table yet, only parks. And when I read the output then I see this:

Hurrah! The right park features, with the right attributes, and the right Feature Type; and all through a dynamic workspace. Again, this is nothing you couldn’t do statically, but by using a dynamic writer you allow the destination schema to be automatically generated – and for changes in the output to be defined by edits to the lookup table (rather than edits to the workspace).

To conclude, these examples show that – whatever your level of FME use – you’ll find something useful in the dynamic functionality, and it’s well worth trying them out to see what you can make of them.

If you’d prefer a live demo and explanation of this functionality then you can tune in to an upcoming webinar we have scheduled on FME2010. See www.safe.com/FME2010Webinar for more details.

Another time I’ll go back to Generic workspaces, and highlight the new functionality for that in FME2010.

Regards,

About FME Custom Format Data Transformation Database Dynamic Schema Feature Types To Read FME Desktop Generic Reader Generic Writer Geoweb Safe Software Events Schema SchemaMapper Workspace Resources

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)

Comments

Comments are closed.

Related Posts