This post is all about updates to FME Server in FME 2014.
Of course, all Desktop updates are really updates to Server, since they share the same core engine and since Desktop is the authoring environment for Server. But then there are a number of updates specific to the Server product itself that are very much worth mentioning.
Now I’ve started to update the training materials for Server 2014, I’m getting really enthused about the Resources update (#6 below) and the folder watch notification (#5). The Resources update is making it really easy to handle source data and use it for multiple workspaces, which is vital for FME Cloud. And the notification system in general – and folder watch in particular – is just really easy to use now.
So read on and I hope you’ll become similarly excited!
Let’s start out with the Notification Service, of which there have been a number of updates, and in particular updates relating to Amazon Web Services.
In short, Amazon SNS (Simple Notification Service) and SQS (Simple Queue Service) are both supported by FME Server 2014, on both the publishing and subscription side of the notification coin. This means that you can set up FME Server to react to – and run a workspace in response to – a message from either SNS or SQS; and you can also set up Server to send a message to SNS/SQS in response to a workspace event. All you need is the Amazon Resource Name (ARN) of the topic to be used:
Additionally there is a send (Subscription) option for Amazon S3 (Simple Storage Service) so you can use a workspace to trigger a dispatch of data to S3. For example, you could use FME Server to tile raster data and send the tiles to S3 to store them. Then Data Download requests for data could be supplied as a simple link to the S3 bucket.
For more information on all our integrations between FME Server and Amazon AWS, see this great page on FMEpedia.
Another notification update, this time the support for WebSockets as both a Publication and Subscription; i.e. FME Server can both receive messages from or send messages to a WebSockets server:
But the WebSocket support isn’t limited to one-off notifications like this. It can also be used to handle a continuous message stream. This is useful when the messages are coming in/going out at a faster rate than could be handled by starting up a workspace every time. In brief, a workspace is run on FME Server, it connects to a WebSocket server, and it keeps that connection open until the workspace is cancelled.
This – and much more – is also possible because FME Server itself is now capable of acting as a WebSocket server! So other applications can also communicate directly with FME Server through WebSockets for high-speed communications.
If you’re new to this then, rather than try to explain WebSockets and how they work, I’m going to point you to a great FME/WebSockets tutorial we have on FMEpedia.
This one is more of a fix than a new feature, but it’s important to mention because we had several users log this issue with support and many more who asked for improvements in this area – and we do like to keep our users happy 🙂
The FMEServerJobSubmitter is an FME transformer that acts as a way to chain several workspaces together. As explained by this FMEpedia article, you would create a workspace that does nothing except call a series of other workspaces (in this example, to load data, process data, then rasterize the data):
Then you run this workspace on Server and it causes all the others to run in turn. The problem is – or was – when you run this workspace on FME Server, it used an engine/license even though it’s not doing any processing itself – it’s just running other workspaces. That could cause an obvious problem where, for example, you only have one engine as the child processes could never get run.
You didn’t think that was right and neither did we, so in FME 2014 we updated the transformer so it no longer takes up a license or causes those sort of server deadlocks.
I’ll mention this briefly, because it’s not really an update to FME, and you may have seen it already from Don’s recent blog post or the recent (slightly off-centre) demo in the Deep Dive into FME Server webinar.
Zapier is an online tool for connecting different web services. For example, you can connect GotoWebinar with your MySQL database so that when a person signs up to your webinar a new record is added to the database. Or perhaps when a new case is created in your FogBugz bug tracker it triggers an email to be sent from your Gmail account.
The update here is that it now includes support for FME Server, at least in a beta form
The obvious use there is that you can use FME to incorporate location-based processing into your workflows. For example, in Don’s blog post he joins Salesforce to ArcGIS Online (and Google Maps Engine) via FME Server, so that each new FME customer added to Salesforce is automatically geo-coded and added to a map.
If you want to try out FME Server with Zapier you can click this invitation link to get access.
When you upgrade to FME 2014 and log in, the most obvious changes are new items on the main menu; one of these is titled Resources.
The Resources tools are a great way to manage data for use on FME Server, in a much improved way to the old capabilities. It used to be that you could upload data with a workspace, but that data was fixed in a set repository and tied to that specific workspace. However, the Resources functionality lets you upload any data and use it with any workspace. Here, for example, I’ve uploaded a file of parks data (Parks.zip) and a folder of engineering datasets:
If I want to use these in a workspace I just need to select the file in the configuration dialog, like so:
What’s really nice – and enables the #5 on my list – is that you can locate these directories with a file browser. That way I can manage my data directly, outside of the web interface (assuming I have access to the server file system):
The other way I can upload data is through the Server Publishing wizard in FME Workbench:
Here I’m uploading a text file to the old-style repository storage, but also uploading the Parks.zip file to the Temp resources folder.
Folder Watch Notifications
Folder Watch is a new type of Publication in the Notification Service. Basically it sets up FME to watch a directory and react (run a workspace usually) when a change is detected in that directory. That directory will be one that exists under the Server Resources structure.
So, for example, here I have Parks.zip in the Resources directory C:appsFMEServerTemp:
I create a workspace that translates that data and writes it to… wherever; a database, a file, a notification, etc. I want this workspace to run whenever I have new data, so I publish it to Server (putting the data into the Temp resource directory) and register it with a notification service:
NB: Notice that I’ve had to create a dummy text Reader in the workspace, to handle the incoming notification. That one always catches me!
Now I create a new publication that watches the Temp folder and triggers the workspace (via the ParksUpdater topic):
So now, whenever I replace Parks.zip with a new version in the Temp folder, the workspace will get triggered and the data translated.
Monitor View + View Job While Running
Another new menu item in 2014 is labelled Monitoring.
The Monitoring tool lets you keeps an eye on notifications as they are triggered, like this Directory Watch publisher:
As you might realize, this is a great tool for assuring yourself something is running correctly, but also for debugging. Previously it was difficult to test notifications, because I could never tell when they had been triggered. If something did go wrong I would have to check each piece of the system to find out where the issue lay. But now I can easily tell whether or not a notification has been fired, and this makes it far simpler to track down any problems in my setup.
For example, if a publisher notification has been triggered but I don’t see anything happening, then the issue must lie in the workspace that the notification is running.
A second – similar – update relates to running workspaces in general. In FME2014 I can now see a job is still running:
But I can still click on the job, choose the Log tab, and see the log file:
In other words I can look at the log of a workspace while it is still running! Again this is a great tool for assuring yourself that a job is progressing as expected; when you have a process that takes (for example) 12 hours, it’s useful to be able to peek now and then to confirm that all is well. And if all isn’t well, then you find out right away and can stop the workspace at once, so as not to waste time.
Scheduled tasks are an important part of FME Server, particularly for users running large-scale processing jobs. But – like with the logging scenarios above – you want to be notified about the status of the job, particularly if it’s failed for some reason. Up until now you could have used a script or transformer in the workspace, to tweet a message or email an administrator, but now in FME 2014….
Yes! We’ve added the ability to trigger a notification on completion of the workspace, so now you can send a message to signal a success/failure of the scheduled translation. I didn’t think this was a big deal but when I tweeted about it I got three re-tweets, which pretty much counts as going viral where I am concerned, so I’m thinking more users will make use of this than I originally thought.
The final new menu item in 2014 is labelled Migration. Previously we had (still do have) a Reader and Writer in FME Workbench that let you read the configuration from a server and then upload it to another and this was how you could migrate to a new machine or backup your Server system. But for FME 2014 we’ve wrapped that functionality up into a nice graphic interface:
So now you can easily backup and restore your FME Server setup, even on FME Cloud!
REST API Documentation
Click on this link and check out the new REST API documentation. Go on, I’ll wait…
It’s a thing of beauty. And as Keats said, “A thing of beauty is a joy forever: its loveliness increases”. In fact, if he were alive today I’d like to think he would write Ode to a REST API, it’s that good!
You might have noticed I scattered quite a few FMEpedia links throughout this article. That’s because we ran through the Server content on there and updated it with a bunch of new demos and tutorials, plus integrated it nicely with the existing documentation. I wasn’t involved in that work, but I’ll still admit it’s a great job 😉
And speaking of tutorials, don’t forget to try the new FME Server tutorial. It’s a great way to learn about FME Server and you get to sign in and try an FME Cloud machine that’s set up just for tutorial use. It includes a great email notification example, which is really easy to set up now. So please give it a go, or pass it on to your colleagues and customers to try.
I hope this article was interesting. This is a really exciting time for FME Server, as a number of long-term projects are just coming to fruition; yet even so, being privy to some of our future plans, I can only say you ain’t seen nothing yet!
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)