Here’s the second post detailing the setup of the FME-based version of the TV game show Jeopardy! which took place at the recent FME user conference.
Last time we looked at the background to this game – its setup and general sequence of play – now we’ll look closer at one part of this sequence. The Answer Submission part of the game is typical of how most parts were setup and operated.
I hope this is of interest. There are certainly some points to note if you plan on implementing an FME Server project of this type, or if you are using SQLExecutor to call database operations from within a workspace.
The Question Answering Process
Before getting onto workspaces, let’s review this section of the process, but in a little more detail than last time.
Once a question has been selected from the FMEeopardy! board then that question is copied over from the QUESTIONS table into the ACTIVEQUESTION table (below – click to enlarge). To be honest I could have perhaps just set an ACTIVE field in the QUESTIONS table, but hey it seemed sensible at the time.
Notice that there is a TIMESTAMP field there, to show when the question was made active. Then we’ll be able to calculate who answered the question the fastest.
You’ll probably also have noticed a GEOM field. Out of habit I used the Spatial Object writer, which automatically creates a geometry field, even though I was writing non-geometry data.
Now the contestants click the link to open the dialog to answer the question (below – click to enlarge):
They select an answer, enter their name and password, and click submit. Then comes the interesting bit…
- Submitting the data to FME Server and running a workspace
- Processing the data in the workspace
- Returning a response to the web interface
Submitting the Data
The answer, name and password are all submitted via a HTML form. Here’s the HTML (for clarity I’ve removed any formatting tags):
<form method="post" action="/fmedatastreaming/fmeidol/SubmitIdolAnswer.fmw"> <input type="radio" name="ppQuestionAnswer" value="1">GML <input type="radio" name="ppQuestionAnswer" value="2">GPX <input type="radio" name="ppQuestionAnswer" value="3">VML <input type="radio" name="ppQuestionAnswer" value="4">KML Team Name: <input type="text" name="ppteamname" value=""> Password:<input type="password" name="ppTeamPassword" value=""> <input type="submit" value="Submit Answer"> </form>
- The first line defines the action – or FME workspace – to execute when the submit button is pressed.
- The next four lines define the possible answers and accept an input – the value will be 1, 2, 3 or 4.
- The next two lines define the username and password.
- The final lines define the submit button and close the form.
ppQuestionAnswer, ppteamname and ppTeamPassword are all defined in the SubmitIdolAnswer workspace as published parameters (below – click to enlarge). This is how we get the info into the workspace.
You might notice that all of the parameters DO NOT have a default value set. I’ll want to use the workspace to check if the user has left any fields empty. If I have default values in there then there’s no way to tell that (because empty fields will be replaced with the default values). You have to be careful if you test run a workspace in Workbench before uploading to server, because any answers you provide could get stored as new defaults without you noticing!
Processing the Data
So, here’s the workspace used to process the data (below):
At this scale all you need to notice is that it is well-annotated(!), that we don’t have a reader (just a writer), and that there are lots of branches depending on different actions, all of which end up at the same output.
Now (below) here is the first section of the workspace. The workspace triggers creation of a single feature with a Creator transformer.
The first thing we need to do is read details of the current question from the CURRENTQUESTION table. This is done with a SQLExecutor transformer. The SQL we use is simply:
select QUESTIONID, QUESTIONTEXT, QUESTIONCHOICE1, QUESTIONCHOICE2, QUESTIONCHOICE3, QUESTIONCHOICE4, QUESTIONACTIVE from CURRENTQUESTION
Having that information we can first flag the special case that is the Final Jeopardy question (this is special because it requires a wager amount to be submitted), and then check that the question is still active. If the question isn’t active then there is no point processing the data any further.
If the question is active then we can check the user’s credentials (below):
- Fetches the values of the published parameters (ie the user name, password and response) into attributes.
- Strips out bad characters (apostrophes and semi-colons) which could cause problems in my database insert statement.
- Checks for empty fields
- Fetches the team info from the PEOPLE table and checks the username/password
Assuming the answer has passed all these checks, then we do a few more (below):
- Check if this user has already submitted an answer (you don’t get to answer twice)
- Check if this is the final jeopardy question. If not, then continue, if so, then…
- Check the amount wagered is actually a number. The AttributeClassifier transformer will check if it is an integer (note that the NaN test is NOT the way to do this)
- Check if the amount is a positive number (to stop people wagering -$1,000,000 and purposely getting the question wrong!)
- Check you aren’t wagering more points than you actually have
The check for existing answers is done using a SQL count function:
select count(*) AS alreadyanswered from ANSWERS where TEAMID = @Value(TEAMID) and QUESTIONID = @Value(QUESTIONID)
The reason we put “AS alreadyanswered” into the SQL is that we want to specifically define what attribute this result should be returned in. Later in the SQLExecutor wizard we get to specify that “alreadyanswered” should be exposed in the workspace, so that we can use it in the Tester (alreadyanswered > 0).
Now we’re finally done checking you out (it’s almost worse than crossing a border) we can accept your answer (below):
The timestamp creation employs a neat trick. An attribute called TIMESTAMP is created with its value set to “NOW”! The DateFormatter can interpret relative dates such as TODAY, NEXT THURSDAY and NOW, and so happily converts my NOW into the chosen format of %s (seconds since the epoch).
Then I write the whole lot into the ANSWERS table using the SQL command:
insert into ANSWERS (TEAMID, QUESTIONID, ANSWERNUM, TIMESTAMP, FJWAGER) values (@Value(TEAMID),@Value(QUESTIONID),@Value(QuestionAnswer),@Value(TIMESTAMP),@Value(WagerAmount))
We record which team is answering, which question they are answering, what their answer was, and when they answered it. We also record a wager amount – though this will be zero most of the time.
Note how we inject attribute values into the SQL using the @Value function. If there were going to be spaces in the values (which there aren’t here) then we’d need to put quote marks around them, for example:
insert into people (TEAMID, TEAMLEADER, TEAMNAME, TEAMPASSWORD, ENTERIDOL) values (fmeteamid.NextVal,’@Value(TeamLeader)’,’@Value(TeamName)’,’@Value(TeamPassword)’,’@Value(EnterIdol)’)
This SQL (by the way) is used in another workspace for registering an FMEeopardy! team. The other interesting thing about this – apart from the quote marks – is the use of the Sequence called fmeteamid. fmeteamid.NextVal gives me the next value of this sequence number in order to provide a unique (and consecutive) value for TEAMID.
Returning a Response
OK. You may have noticed that all of the above screenshots end in a dangling connection. That’s because I edited out the part that returns the response to the user. Now let’s check them out (below):
See how they all lead to Concatenator transformers (some are off the left-hand side of this screenshot) which create HTML. This is the HTML we want the users to see in their web browser.
So each Concatenator is used to give a different output response. They do this by simply pasting together hard-coded HTML strings with attributes and other responses, to create a valid HTML document that will open in a web browser.
Here’s the Concatenator which creates HTML to tell you that you have missed out one or more required fields in the form (below):
Note how the above again includes the code for the HTML form. That’s because if your first attempt to submit an answer failed we want to automatically give you the option to try again. A successful attempt does not need to include this HTML.
Note also how we write everything to the attribute text_line_data. That’s because this is the format attribute which the TextFile writer expects data to be stored in. So, by putting HTML code into that attribute and writing it to a TextFile writer we automatically get a valid HTML document out.
But how do we get that HTML to display automatically? That’s simple. Remember we are calling these workspaces as a streaming service:
So the data is automatically streamed back to the browser. The final question is, how does the browser know what to do with the data? Another simple one. When you add a TextFile writer to the workspace, one of the settings is MIME type (below). By setting MIME type to text/html FME Server is able to tell the web browser to treat the streamed data as a HTML document.
Problems, Limitations and Best Practice
Obviously you can’t go through a project like this without experiencing some problems. Here are the ones I faced, and in particular here are some best practices I ignored to my cost:
One Writer Limitation
If you want to stream data from an FME Server workspace then you can (currently) only have one writer in the workspace. For example here I used the TextFile writer. I couldn’t do this AND write data to another format (streamed or not). I believe this limitation is likely to be resolved fairly soon (maybe 2010).
NB: I’ve just been told you can have two or more writers, but you can’t use both at once, and if you want to stream either one they both have to have the same MIME type.
Look at how many Concatenators I used to create HTML. Look at how they all used the same header:
<html> <body style="color: black; font-family:arial; width: 1024px; margin-left: auto; margin-right: auto; text-align: center; background-color: #FFFFFF; background-image:url(/img/FMEIdol.gif); background-repeat:no-repeat; background-position:center; width:1024px;height:768px;">
Now imagine I want to change the background colour! Just think how many Concatenators (in how many workspaces) I would (and did) need to edit! It would have been much, much better to keep this standard content in a template file, read it in with the AttributeFileReader transformer, and use that in the Concatenator.
You wouldn’t think I’d need to be told this, but the data source for my database should have been a published parameter, and this parameter shared by all the SQLExecutors. I didn’t do this, with the result that when we transferred the project onto a machine with a different Oracle service name, I had to go through every SQLExecutor and every Oracle Reader/Writer, in every workspace, and manually change the service name.
If I had used published parameters then the worst would have been one change per workspace, and if I’d put it in the HTML template (above) then I would only have had to make one single change.
I didn’t do this, and so if you want to implement this project you too will have to edit every instance of this setting. Sorry!
If you are migrating from one FME Server to another (say from a test installation to a final installation as we did with this project) then there is a special reader and writer you should use to migrate the workspaces from one repository to another (below):
You shouldn’t do what I did, which was to copy C:Program FilesFMEServerServerrepositoryfmeidol from one machine to another. Admittedly you get the fmw files, but the new server doesn’t recognize them because everything in the repository is sync’ed with an internal database. You can’t just copy workspace files. RTFM Mark! It’s in the FME Server Administrator’s Guide.
The other migration issue is not to put hard-coded server names into your workspace links. For example, this is BAD
<form method=”post” action=”http://firefly/FMEServer/fmedatastreaming/fmeidol/SubmitIdolAnswer.fmw”>
…because when I migrate from test machine firefly to machine fmeidol, I need to go through every URL and change the server name (and remember I have these embedded in a thousand Concatenators). It’s much better to have a relative URL such as:
<form method=”post” action=”/fmedatastreaming/fmeidol/SubmitIdolAnswer.fmw”>
What is really nice about FME2010 is that we’ve re-added the option to have a longer recent workspace list in Workbench. When you have 10 to 15 workspaces then it’s great to have a direct link to each of them.
However, one issue I faced was knowing whether the workspace I had on my computer was the most recent from the repository. There were only two of us editing these workspaces, but it still made me concerned that we would publish over the top of each other’s edits.
For best practice I’d suggest that you ALWAYS download the workspace from the server repository if you want to make edits. Don’t rely on a local copy when you haven’t used it for a while. And don’t make multiple copies of the same workspace – eg download it to different locations – because you are sure to become confused if you do.
So, my suggestion to the Server development team is this. Can we get a better method of workspace control? For example, one of the following:
- A lock to denote that a workspace has been downloaded for edits
- A way to compare a local workspace to a repository version
- A means to edit a workspace on the server, without having to download it first
So doing this as a game is all very well, but what practical application does any of this have, especially since it’s all non-spatial?
Well the html streaming is a great tool for responding to data translation processes. For example, Aaron Koning’s Data Validation Example uses this technique to tell a user whether or not their data passes a quality assurance check. We don’t always need to be sending spatial data.
And on the subject of Data Validation, when you have a group of testers determined to do their best to cheat their way to victory, you quickly begin to realize the need to rigorously test and clean user input from published parameters, and this workspace in particular has a whole battery of tests you might find useful.
Ah, published parameters. I hope I’ve demonstrated not just how to get that information into FME Server, but why it’s so important to keep common settings (like database connection parameters) outside of the workspace. This is why we’re doing so much work on generic readers and writers – so workspaces need the minimum of edits for any given change.
Also, the whole project shows how easy it is to run workspaces from a custom interface. In this case it was a (rather unsophisticated) web interface, but if you attended the Maps for the Masses workshop then you’ll have seen just how easy it is to run a workspace from a URL in Adobe PDF files or from data visualized within Google Earth. The key point is that you are making processes and data available to users who have no knowledge of FME; in fact in FMEeopardy you saw nothing of the standard GUI, to the extent you’ve only got my word for it that it was actually running on FME Server!
So, that’s how the answer submission process worked in FMEeopardy. All of the other sections to the contest worked in very much the same way – accepting input from the user, calling a workspace, processing the data, and returning a stream of HTML as confirmation.
Whew! That was a long post. I don’t think I’ll need a third one though, since all the info is in here. What I will do is start to upload the workspaces and related scripts to fmepedia and put an installation guide in there in case anyone wants to try it out. It would be very simple to modify the source data to put your own questions in there.
The page will be: http://www.fmepedia.com/index.php/Category:FMEeopardy
This Edition of the FME Evangelist…
…was written to the music of Me First and the Gimme Gimmes. They do excellent punk covers of Don’t Cry for me Argentina, The Phantom of the Opera, and O Sole Mio, but today you get Leaving on a Jet Plane.
Mark IrelandMark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)