I might not be a pope or a Roman emperor, but that doesn’t mean I can’t create my own calendar. The first month of my year I call Launchuary – the month of the new FME launch. Next up is Trainuary, when I complete all our training materials. We’re now entering Blogch, the month when I realize I haven’t written half the blog posts I planned to and scramble to catch up.
The problem is, FME 2016.1 is coming soon (probably when the FME World Tour starts on Tourpril 6th), and that will be a whole new bunch of articles. So… I’m going to quickly round off the 2016.0 launch with a summary of all the new functionality that I wanted to highlight, but haven’t had time to yet. In particular that is:
- Workspace Reader
- AttributeValidator Transformer
- Python Updates
- LogMessageStreamer Transformer
- New Formats
- Data Inspector: Mark Location
This Reader is a new format, but it’s one that simply reads other FME workspaces and provides information about them. For example, if I use this Reader to read all the fmw files in the FME training course, I see this:
I can see (from FileProperties) that there were 47 workspaces, and that table shows me their name, when they were created, and even when they were last run. I can see I have 100 readers in those workspaces, and can look to see what they are. I can even see how many bookmarks I have (not enough).
The nice part is that these tables all have keys to join them – for example workspace_id and reader_id – so I can find which feature types are used in which reader in which workspace. The only question is why would I want to do this?
Well, several users asked for this functionality because they have a large number of FME workspaces and wanted to keep metadata about them. One user said to me:
[we] create metadata documenting our translations and are hoping to expand the information that we extract from the workspace files. We’re currently extracting the input and output feature types and the date that the workspace was last run. We’re hoping in the future to document the bookmarks within the workspace.
This new reader gives such users the information they require to compile metadata about their workspaces; in particular I can see how users could check for old or out-of-date transformers, or could check usage of custom transformers. Yes, 2016 has new transformer upgrade capabilities, but with this reader you could check hundreds of workspaces all without actually having to open them in Workbench. The Statistics table has columns called num_custom_transformers and num_out_of_date_transformers, so that would be really easy to do!
The AttributeValidator is so explanatory – in both name and functionality – that it probably didn’t need a full blog article anyway. It’s just more important for you to be aware that this exists, so that you can try it yourself. For what it’s worth, the parameters dialog looks like this:
You can see that there are a number of tests that can be carried out on attribute data. It’s perhaps not anything you couldn’t already do with a variety of other transformers, but has the advantage of handling multiple attributes and multiple tests in a single, convenient interface. For example, in the above image both ParkName and NeighborhoodName are being validated against the Type rule.
So try it out and watch how this transformer makes your workspaces more compact than before. I’m guessing it will replace the Tester in certain scenarios, and be easier to use too. If you want to see the above example being demonstrated then check out the Deep Dive webinar that Dale and I presented earlier this year. There’s also a great YouTube video demo from Sterling Geo.
Up until now FME has supported Python versions 2.x. FME2016 introduced support for versions 3.4 or greater. In particular for 2016 the changes were:
- Added support for Python 3.4+
- Deprecated support for Python 2.6
- Added a compatibility parameter to Workbench
These Python updates affect you if you include Python code anywhere in a translation; either a startup/shutdown script, a PythonCaller transformer, a scripted parameter, or any other Python code you run using FME.
FME has shipped with v2.7 for a long time, so the 2.6 deprecation shouldn’t be a problem, and because we’re not dropping v2.x completely you don’t have to worry about suddenly upgrading all your existing scripts. Python version 3.4+ introduces new functionality (which is great) but in a way that is not compatible with v2.7. That’s where the FME compatibility parameter (located in the Navigator window) comes in:
This parameter isn’t so much picking the interpreter to use – there’s an FME option to do that – but more you telling FME which version of Python your scripts are compatible with. There are three options:
If you pick 2.7 then you are telling FME that scripts in your workspace are only compatible with v2.7. If you pick 3.4+ then you are telling FME that scripts in your workspace are only compatible with v3.4. If you pick “2.7 or 3.4+” then you are telling FME that your Python is compatible with either version. In all cases, of course, it’s up to you to ensure that your scripts really are compatible with the version specified.
FME then picks the interpreter automatically. It will first check the FME option (Tools > FME Options > Translation) to see if that interpreter version is compatible with your script. If not it will search your system for a suitable interpreter using the PATH environment variable. The full information is available under this help page.
The compatibility parameter is workspace-based; i.e. it applies to each workspace individually, so that one workspace might be compatible only with v2.7 while another might be compatible with 3.4. Of course this is most important when you intend to share the workspace, or run it on FME Server. In that case “2.7 or 3.4+” is the most flexible option because it will run on a platform regardless of what interpreter it has.
If you set it to 3.4 but publish it to a platform that only has a 2.7 interpreter, then the workspace will error out with the complaint that it can’t find an interpreter.
The FME Objects and FME Plugin APIs don’t yet support Python 3, but that is planned for FME2017.
Guy Kawasaki says a technical evangelist always tells the truth, so I’ll be honest and say that this transformer was a great idea… but it has a big limitation.
The great idea is that the transformer extracts messages from the current log file/window for use in the workspace itself. We noticed that users often scan the log after a workspace has run, in order to determine if it ran correctly or whether any problems occurred. With this transformer the same workspace can both carry out processing and scan the log!
When I tried this out I used a workspace with FeatureReader and FeatureWriter because I want to write my data before I examined the log:
But that’s where I hit this limitation: there is no control over when LogMessageStreamer transformer operates. In my workspace it started and finished before most of the transformers were executed. I got 19 messages from the log, which covered some of the reading, but none of the transformation or writing.
So, this transformer might be useful to you, or it might not. If you’re into parsing FME logs, try it and see if it can help you. Then give us some feedback so we can see if anyone is using it and whether it was useful.
For FME2016 my favourite new formats are Amazon Aurora (a MySQL-compatible online database), GTFS (General Transit Feed Specification) which is a format for public transportation data, IndoorGML (exactly what you’d expect given the name), and Adobe PRC. I’ve not used PRC yet, but it looks impressive. It’s a 3D format that embeds inside PDF files. It works on all platforms and in both 32-bit and 64-bit, so it’s very flexible.
I don’t know what the actual format count is in FME2016, but I’m sure it’s gone up. It always does. My one observation is that the new formats seem to me to be “niche formats” and not ones everyone is likely to encounter. That might mean our formats graph might be reaching a plateau; or it might mean it’s the start of an explosion of lots of specialist formats. There are a whole bunch of new formats for 2016.1, so I’m going to predict it’s the latter!
Data Inspector: Mark Location
The Data Inspector got a nice new update in 2016 with a Mark Location option. You can find it by right-clicking like so:
You might have noticed that already. But did you notice the Mark Location option on the toolbar? If you click that you get a bunch of options for that marker:
With that dialog you can place a marker at specific coordinates, plus also change the color, marker type, label, and symbol. The marker persists too, so open a new dataset in a new tab, and you’ll get the same location marked as in a previous tab. It’s a really very useful piece of functionality when inspecting data and – in particular – bookmarking a location to go back to later (click the Locate button and the display moves straight to it).
This is probably about the last bit of functionality that existed only in the old FME Universal Viewer application, and not the Data Inspector. In fact, you may not have noticed but the FME Universal Viewer no longer appears in the FME start menu. It’s like saying a sad goodbye to an old friend (albeit one I haven’t seen in five years).
There are a few other minor updates that I wanted to point out. Firstly the Tweeter transformer now lets you attach an image to a tweet. I know not everyone uses Twitter at all, let alone with FME, but for those who do I think this has great importance. Any alerts that you send (for example, “Alert: power outage”) could now include a raster map showing the affected area.
Secondly the new CoordinateReplacer transformer lets you search for specific X/Y/Z values and replace them with new ones. The VertexCreator got an update too that lets you search for a vertex at a specific index (i.e. vertex number rather than X/Y/Z) and replace that with a new one. Previously there wasn’t really a way to replace a single coordinate unless you wanted to start coding a solution in Python, and any time we can cut out the need for coding it’s a good update.
Finally the Sampler transformer includes a new randomized mode; for example instead of taking the first 100 features to sample, you could take 100 features at random. If you want to sample data, it’s more realistic and saves you having to create random numbers and/or sort the data randomly.
I mentioned FME2016.1 is coming soon, and it is going to have a lot of updates; almost enough to make it an entirely new release!
For example, you might have noticed that one thing I didn’t talk about for 2016.0 was the new “presentation mode” – where you can flip from bookmark to bookmark. That functionality never quite did it for me because you couldn’t control the order of presentation. In 2016.1 however, that capability has been added, and I’ll definitely be talking about it.
The Data Inspector is also getting some nice symbology-related updates, there are several new formats (including…. no, I don’t want to spoil it), some great HTML-related transformers, and a long-awaited piece of functionality that is related to the image on the right. I think you know what it is!
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)