XQuery: Reading XML Contents
XQuery: Handling a Metadata Repository
Fun Stuff: FME2009 Wallpapers
Your Spy at Safe: New Tester Sneak Preview
FME Blog: New and Updated!
I don’t know about you but I get a nervous shiver any time I see terminology beginning with “X”:xml, xfmap, xslt, xsd, xpath, etc
However, I know that by avoiding such functionality I miss out on a great deal. It came to a head in last week’s FME developer demo when I was so blown away by one of the presentations that I just had to investigate some more.
So this issue is where I hope to take the letter X away from propellerheads, and put it in the hands of normal people like you and me. To do this we’ll use a new feature that admittedly has an X in it, but actually helps you handle XML data.
I’m talking about XQuery.
XQuery is a standard W3C language newly supported by FME2009. It is to XML what SQL is to relational databases. In other words it helps you use FME to pull information out of XML datasets, without having to use masses of StringSearcher transformers.
Want to learn more? So do I! In fact I’ll be learning how to use this stuff as I write this post, so please tolerate any errors I make (or skip to the insider info about the new Tester transformer!) and just read on…..
XQuery – Reading XML Contents
The problem with XML is that it is so flexible it needs a schema document (xsd file) with each xml file to explain the structure of the dataset. But given an XML dataset, even if the xsd file is included, reading it with FME requires you to manually turn the xsd into an xfmap to tell FME which source features are mapped to which FME features. Got a headache yet?
At this point most non-XML specialists will either give up or turn to using the Textfile reader and StringSearcher transformers to extract information from their XML file. But then your workspace needs to juggle with global variables and feature order to make sure you are attaching the right attributes to the right geometry.
This is why XQuery is so great. With very little work you can pull information out of an XML file and use it directly within a workspace. Of course, with a lot of work you can really be an XML superhero, but I’m going to keep this simple.
Safe’s Dmitri has some great examples on fmepedia but – since I feel guilty evangelizing technology I can’t even use – I’ll make up my own and try it.
Here’s what I have – a GML file (without schema) that represents traffic events in the City of Austin:
<fme:Entry gml:id="id799feed0-ab58-429e-8bca-0e7ec9d4aaed"> <fme:Title>Accident on E 45TH ST at CASWELL AVE</fme:Title> <fme:Content>CRASH</fme:Content> <gml:pointProperty> <gml:Point srsName="EPSG:4326" srsDimension="2"> <gml:pos>30.30712 -97.725121</gml:pos> </gml:Point> </gml:pointProperty> </fme:Entry>
Click the image below to see the full content:
The key information I want from this is the event title and the event location. So firstly I’ll deal with location by placing an XQueryExtractor transformer and pumping a null feature in to trigger it.
From the XML I see that I want the tag: <gml:pos> and that the “gml” namespace is defined by xmlns:gml=”http://www.opengis.net/gml” so I set up the XQuery transformer to read the XML file and retrieve that information using this simple script:
declare namespace gml="http://www.opengis.net/gml"; //gml:pos
It doesn’t get much easier than that. This returns me an attribute with a comma separated list of event coordinates.
I assume I could find the event title in the same script, but after some trial and error I go with the simple solution and just place another XQueryExtractor to find the tag <fme:Title> in the namespace xmlns:fme=”http://www.safe.com/gml/fme”
declare namespace fme="http://www.safe.com/gml/fme"; //fme:Title
Incidentally, by using two slashes before fme:title I am saying, I don’t care what level of XML hierarchy this is on, just find all fme:Titles and return them. A single slash is when you are specifying the exact level of hierarchy (/fme:Entry/fme:Title).
And that’s all I need to do to get the info into Workbench. Now I can start to process that data. Here’s the complete workspace I use (click for full image):
I don’t want to go too much into the FME side of things, because the setup there would depend on the exact nature of the data you are retrieving, but in my example:
StringReplacer: Strips out all XML tags from the returned information
AttributeSplitters: Split the two results up into an FME list attribute
ListExploder: Explode the first list to create one feature per record
ListIndexer: Retrieve the correct information from the second list for each feature
AttributeSplitter: Split the coordinate attribute into separate X/Y attributes
2DPointReplacer: Turn the X/Y attributes into a real spatial feature
AttributeKeeper: Clean up unwanted attributes before outputting to the Visualizer.
So there we are. It took me about 90 minutes to create this workspace and write this item alongside. The biggest difficulty I found was in understanding namespaces. In my idiot’s guide way of thinking, if XQuery=SQL then Namespace=Tablename; which seems to help me understand things.
In simple cases your XML states “xmlns=” and all is simple. Unfortunately some XML seems to have namespaces which are assumed (and not listed), and some have namespaces defined at lower levels in the content. Also you sometimes get xmlns:xxx=”…..” in which case your XQuery needs to be declare namespace xxx=”……” – ie using the xxx as the namespace name.
And I admit; this is hardly a complex example or one which uses anywhere near the full capacity of XQuery. However, it does match the most likely scenario for an FME user: you can get the information out of a random XML/GML file, with minimum fuss, and start processing the data using your existing FME skills; rather than spending a week cursing the inventor of XML (wikipedia tells me it’s one Charles Goldfarb if you really want someone to blame!)
This example can be found on the XQueryExtractor page on fmepedia.
XQuery – Writing to a Metadata Repository
Phew. Having done my part by tackling the easy stuff, here’s an example of what you can do if you really know your way around XQuery, like Tom Weir a developer at Safe Software.
The OpenGIS® Catalogue Services Interface Standard (CSW) is a specification by which to create an online repository of metadata about geospatial datasets. One product that does this – and which Tom uses in this example – is an open-source application called deegree.
So deegree is a metadata web service which you can post metadata to, or read metadata information from. Obviously posting metadata is something you are very likely to do when translating data with FME, and one of Tom’s examples shows how it might be acheived.
I won’t go into too much detail, but basically Tom has a metadata template stored as an XML file. On translating a dataset his workspace reads in the XML template and uses FME’s XQuery tools to update the metadata information within it. This metadata can then be published to the deegree service.
You can find full details about this example on this page on fmepedia.
XQuery – Reading from a Metadata Repository
But what really blew me away about Tom’s demo was the ability to read from the metadata service, and what this might mean for an FME Server application.
Tom’s example on fmepedia, is the first part in this grand design. His workspace allows a user to define the terms of a metadata search for spatial datasets. In other words you can specify the location (a BBOX filter) and type (a PropertyIsLike filter) of data you are looking for, and the workspace will submit a query.
But furthermore, the results of this query are processed – using XQuery of course – to extract all of the information about the datasets listed, and the extents and details of those datasets used to create a new dataset showing all that information.
There’s some pretty complex XQuery (and Python) in there though; wouldn’t it be good if, say, we wrapped all that into a CSW format reader to make it easier for you all? Then you could not just read CSW metadata with FME, but also with FME-extended applications such as Map 3D, ArcCatalog, GeoMedia, etc.
Now do you see the link up with FME Server? Using such a reader a simple CSW to KML workspace could be uploaded to an FME Server and defined as a network link service. You could open the network link in Google Earth, enter some metadata search terms, and see the results of your query spatially.
And to go one step further, if the workspace were made to write hyperlinks onto the KML output features, these could be links to further FME Server workspaces which would translate the data into a format of your choice and make it available for download.
In other words, you browse metadata – live – in Google Earth, click on a dataset outline and can download that data in the format, structure and coordinate system of your choice. All powered by FME and XQuery!
Fun Stuff: FME2009 Wallpapers
Want to give your computer a spiffy (and speedy) look?
Then check out the Downloadable Fun Stuff page on fmepedia for a whole bunch of FME2009 speedo-theme wallpaper.
Or if you are into the retro look then there is wallpaper from as far back as the days of exploding ice3‘s and Easter Island FME logos.
Your Spy at Safe: New Tester Sneak Preview
It’s more usually car manufacturers who have to keep an eye out for spies at their test tracks; but now you can take a look at some screenshots sneaked out from under the noses of our GUI development team.
The Tester is an important transformer to us. Feedback has shown that it is the #1, most-used transformer in the whole FME toolset! So on the one hand it’s a prime target for updates to improve user efficiency, but on the other we want to make sure new functionality doesn’t detract from its current capabilities.
Some work in progress for 2010 first of all moves the Tester into the same environment that most of our GUI was updated to for 2009. To see what the initial work looks like click the screenshot below:
But that’s just a preliminary to enable other changes that are going on. Notice that besides the usual AND/OR (All Tests/Some Tests) options, there is now a Custom option. And look what happens when you select it: each test clause is given an ID number and the full test spelled out in the text box below using these IDs:
Nice. But not really custom…. or is it?! Hurrah! The lower text box is editable, so you can change it about and get mixed AND/OR queries:
Actually, this isn’t really any functionality that wasn’t already available in the ExpressionEvaluator – but now it’s in a more obvious location and exposed in a way that most users will be able to understand and use.
These screenshots are a pre-beta version and very much open to change; for instance the up/down arrows may be removed because they don’t add anything but confusion. Also, these updates are only phase 1. In phase 2 we hope to add more operators, especially string operators like “begins with”.
So if you have any thoughts or suggestions on how to improve the Tester then now’s the time to let us know.
And that’s not to mention the new RangeMapper transformer…. but that’s another story!
FME Blog: New and Updated!
Apparently inspired by the upcoming Oscars, our marketing team tells me that “Now Playing” on our web site is a feature called “It’s All About Data”; the new name for our corporate blog.
As they tell me, “see for yourself the changes made to the look of this blog which now features a star-studded cast including internationally acclaimed writers Don Murray, Dale Lutz, Paul Nalos and Michael Weisman. Check it out today and see what the buzz is all about, visit http://blog.safe.com“
Show-times: Every Wednesday a new entry will be posted, so check back each week.
Reviews: “Two thumbs up!” – Don and Dale.
“…this is one blog you won’t want to miss!” – Lahkvir Brar (Marketing dept)
This Edition of the FME Evangelist…
…was written to the music of Mose Allison – “The William Faulkner of Jazz”! Saw him at the Pizza Express in London a few years ago. Fantastic. If you can see him on his current tour I highly recommend it. Just not on June 12th because obviously you’ll be in Whistler at the:
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)