FME Golf Challenge Part 2: Your Solutions and a New Challenge

Greetings FME’ers,

Here’s the second part of my FME Golf follow up, where I showcase all of the entries from our users. I’ll get straight into it, so if you don’t know what this is about, I refer you to yesterday’s article.

Your Solutions

I saved all 14 entries to the root drive of my computer, to save you some characters, but I didn’t change the source paths if you hadn’t already. All I did was open each workspace and save it to 2016.1 to give everyone a level playing field.

Remember, the starting point of the challenge was a file 75,835 bytes in size. Here are the entries in the order I reviewed them.

Rajesh Dhull: Rajesh didn’t do the extreme renaming I did, but his workspace did include one other change. Somehow I hadn’t noticed that the CSV reader has a parameter for filtering; use that and you can get rid of the Tester. Clever!

Rajesh also replaced the Sorter transformers by using a sort parameter on the CSV reader. That’s not so clever, I’m afraid. My original workspace sorted by neighborhood name, which isn’t part of the CSV. So I guess that means disqualification. Sorry. Without that his workspace checked in at exactly 70,000 bytes.

Lars de Vries: Lars spotted the coordinate settings in the CSV reader, enabling him to remove the VertexCreator, and also cottoned-on to the trick of moving everything to the root drive. He also used the CSV filter parameter. But the big change was replacing the CSV reader and the PointOnAreaOverlayer transformer with a FeatureReader transformer with built-in spatial query. Man! Why didn’t I see that. It blew my workspace away with a 2016.1 result of 47,519 bytes.

Tom Farrington: Tom did a whole bunch of renaming like I did, to cut down on individual bytes, and like other users he found the CSV filter and X/Y parameters. There was obviously some clear thinking but he didn’t use the FeatureReader and his workspace checks in at 67,995 bytes.

Dennis Wilhelm: Dennis did the file renaming trick and the objects in his workspace were squished together in the middle (although not totally overlaid). The same oddity that I found. But the big thing he did that I hadn’t noticed before was replacing the AttributeRenaming with manual attribute mapping. How much did that save I wonder? In all his file was reduced to 67,420 bytes.

Kim van Winden: Kim is a developer, so I was expecting something impressive! He (she?) also used the FeatureReader transformer and renamed everything possible to reduce the bytes being stored. He removed user parameters, which had saved me a fair chunk of space, and “moved all transformers to the 0,0 location using the grid” so perhaps he deciphered how to arrange his transformers properly?

He also says he “chucked all comments out of the header” – I suspect he used the Edit Header tool, which is fine (why didn’t I think of it?!) and his file size: 45,974 bytes! A new leader.

Niek Goorman: Niek too used the FeatureReader (again, why didn’t I see that?!) and replaced the AttributeRenamer with manual mapping. A solid entry, but he didn’t go to the extremes of renaming that others did and he checks in with 53,214 bytes.

Nariman Emamian: Nariman – surrounded by FME gurus all shaving atom-sized parts off their workspaces – decided to go the other way and produce a workspace as large as possible! Nice try Nariman, but I think that puts you last in this contest, although top of the list for lateral thinking.

Roland Martin: Roland spotted some features that others hadn’t (for example the expanded attribute lists that I used) and used the CSV parameters and FeatureReader as the best workspaces have been doing. He also replaced the AttributeRenamer. His result? A very creditable 50,786 bytes.

Todd Davis: Todd filled my email inbox with three attempts, each improving upon the previous. His final workspace used a Generic Reader to read both source datasets. I hadn’t thought of that. It certainly saves a reader but at the cost of requiring extra transformers. He also replaced the AttributeRenamer and VertexCreator with an FMEFunctionCaller that called various functions such as @CopyAttributes. That’s fine. I did say functions are in the rules.

But what’s most remarkable is his file size: 35,229 bytes! That’s astounding. I’m going to try and replicate that because it is so much better than anyone else (so far…). His workspace is this:

Yves St Julien: Yves suggested the InlineQuerier would be a great solution if only it used Spatialite and not plain SQLite. We very well may implement that. Without that he still used that transformer and produced a reasonable result (it beat me) of 65,396 bytes.

Jonathan Moules: Jonathan got a file as small as 10kb, but only by taking rule-bending to a multitude of new dimensions. After I said you weren’t allowed to edit the fmw file by hand, he promptly created a second workspace to do that instead. Nice try, but I’m not going to allow that one – particularly since I can’t get the result to open in FME2016.1. Neither will I count compressing the workspace to a zip file. Or replacing the data with FFS datasets. Your first submission I will allow. It had the interesting idea of using the Textfile Reader instead of both the CSV and KML readers. I would never have thought of reading KML with a text reader. Unfortunately it means you need a whole slew of transformers to handle the data, and the workspace comes in at a hefty 124,525 bytes

Ulf Månnson: Ulf did the usual renaming of objects to A, B, C, etc and figured out the filtering and X/Y parameters on the CSV reader. He too used the Generic writer to write the data. But there was nothing else no-one else hadn’t tried and his workspace was a solid – but not solid enough – 57,867 bytes

Itay Bar-on: Itay is a certified professional whose work I am familiar with and I wondered if he would find something really unique. He did! He suggested using a custom format (fds) file. Roland Martin had said the same, but I dismissed it. Yes, the fds file is smaller, but then you really need a workspace in which to use it. Itay proved that by submitting an FDS with workspace which came to a combined size of 45450+25704=71,154 bytes. He did submit a more normal workspace solution that replaced the VertexCreator with an FMEFunctionCaller that called the @Coordinate function, and which used the FeatureReader and CSV filter like so many others. Its size was a pretty reasonable 52,244 bytes.

Bruce Harold: I left Bruce until last because I thought he would have some special tricks up his sleeve. He did suggest using an ArcGIS Online Service to do the work – which I assume you would at least call from within FME Workbench, Bruce – but said I would probably disqualify him for doing so. He was right. His workspace proper used a FeatureReader to read the KML, but ignored the spatial filtering parameters in favour of using a NeighborFinder. I hadn’t thought of that transformer and will maybe have to try it. He too used the CSV filter and X/Y parameters to bring his workspace down to a number (that still beat me) of 63,717 bytes.

So here’s the final results in table form.

Congratulations Todd. An excellent solution and one that I know is fair because I was able to replicate it. It blew me away just how much smaller your solution was than the second best. And it’s great to see both such a diversity of solutions at the same time as the same sneaky trick to save a few bytes being discovered by completely different people.

Merging the Best Solutions

So, what’s the best possible? I’m not 100% sure but I took all the options I hadn’t tried and applied them to my workspace, and got it down to 34,788kb. It still won’t fit on my Dragon 32 but it would go onto a Commodore 64. Wouldn’t be able to run it though, unless I could implement the FME engine using BASIC in 30kb or less!

Using Todd’s workspace as a base I used a single Generic reader, instead of two “normal” readers (reducing the workspace size by about 30%). I suppose it’s because there are no parameters involved. Todd’s workspace was a little hard to replicate because the best practice aspect was reduced to nearly nil. I added the Generic reader then imported the feature types separately, and had to fiddle with the attributes until I had the FunctionCaller figured out. I tried the FeatureReader instead but it didn’t help in this case.

Anyway, this is what the final workspace looks like:

It’s not pretty. As you can see, all objects are piled on top of each other. At first I wasn’t sure how to move them all to an origin of 0,0 – until I realized I just needed to turn on the grid and snap to that. In the workspace file that gives me:

POSITION="0 0"

…for every object, shaving a bit of storage space.

I was a bit cautious about editing the header – who knows if #! START_WB_HEADER is something that FME looks for in the file – so a bolder person might even be able to trim it a bit more. I shared a copy of the workspace for you to experiment with if you want. You may or may not get the exact same file size when you look at it on your system, but it should be pretty close.

A New Challenge

I hope you enjoyed this. Todd was kind enough to mention that:

That was a good challenge and created some interest among the main FME users here at SCIRT, and also meant a few people learnt about parameters and transformers that they didn’t know about (and we were unaware they didn’t know) such as geometry from csv/excel and inlinequerier.

So I thought I’d give you all another challenge to try. This one will be a bit more difficult, but I have confidence in the skills of my readers. I recently found a page on Wikipedia about something called Conway’s Game of Life. It’s a game involving what looks like a raster grid. Each cell in the grid either lives or dies according to a set of rules, producing a new grid from which to repeat the process again and again. The result is a living grid that could (depending on the starting positions) look like this:

So the challenge is: use FME to implement the rules of the Game of Life, and produce an output demonstrating it.

I tried and can confirm it is possible. How? Well, I’ll leave that up to you. Raster is the obvious solution, and would be simple if the RasterResampler had the right interpolation option (it doesn’t). You might still use raster, or create a numeric array (maybe a list), or use a grid of vector squares, or the Adjacent Feature Attributes, or… well there are lots of ways that might work. I’d prefer if the majority of the solution used FME, and not Python or any other scripting language.

So why not give it a try? I won’t set a deadline, but I promise it won’t be as long a wait as the golf. I’d be interested in seeing whatever you create, even if you don’t manage to fully complete a solution. Half of the fun, as Todd said, is to learn about parameters and transformers that you didn’t know about.

Community Rewards

I’m really happy to see the new Q+A forum is very busy on the revamped knowledge centre. The previous incarnation wasn’t as good, and I felt I was losing touch with our users. Not any more. Also, the scoring system means I can (and will) look up the account of everyone who entered the FME Golf tournament and give you all reputation points according to how good your result was, so your efforts won’t have gone empty-handed.

The other rewards on that site are badges. Most (like on GIS StackExchange) are to do with posting questions and answers, but very soon we should be able to start creating custom badges. I’m hoping to see some friendly rivalry spring up over badges. For example, Todd is going to be the only user with the FME Golf Champion badge! And one of you may soon be the lucky holder of a Game of Life Champion badge.

Additionally I’ve heard that there may be actual physical rewards for users with the most reputation/badges, which would be great. So please do make use of the knowledge centre and don’t forget to vote on questions and answers (and ideas and articles) to get everyone’s reputation moving upwards.

Regards

Mark

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)