As you might know, I’m British by birth, so queues and queueing come naturally to me.

The BBC says, the queue draws on “notions of decency, fair play and democracy.” It suggests that the queue for the Wimbledon tennis tournament is the ultimate, and having experienced it I have to agree. “The Queue” (as it’s known) even has its own official 30 page guide!

But although waiting in queues is equally fair, the actions at the end of the queue might not be. That’s why, for example, supermarkets have a special checkout for “ten items or fewer,” because one person with a monster-sized shopping list could block the queues for shoppers with just one or two items.

And when you think about it, the same issue applies to the queue in FME Server…

FME Server Queues and Engines

Up until recently FME had one queue and a number of engines. It generally worked because one particularly large job couldn’t block the queue; at most it would tie up just one engine. A user could route jobs to a particular engine, but only using config files, and could set priority only for each individual job.

However, a large organization might have multiple users submitting multiple large jobs. In that scenario these jobs could hog all the engines and cause smaller (maybe more important) jobs, to get stuck waiting.

In short, it didn’t work where an organization had a number of departments with unequal requirements, where some users needed to share just a small part of a server for a small amount of time.

So in 2018 we added the ability to create multiple queues, and map them to multiple engines. This technique Don calls Server Capacity Management.

Adding Queues and Assigning Resources

Here’s my FME Server installation as shown by the web interface. I have eight engines available:

Engines in FME Server Web Interface

Underneath that I have a new table displaying a single default queue, but a button with which to create others:

Queues in FME Server Web Interface

So I can create a queue for the knowledge team (my team at Safe) to submit jobs:

Creating Queues in FME Server Web Interface

There is a Priority parameter because in 2018 you set job priority at the queue level. Job level priority is deprecated (although for now it still exists as an advanced parameter). A smaller number indicates a higher priority.

Notice that you assign the engine to the queue, not the queue to the engine. Here I’ve assigned Engine 7 to the queue, with a priority of 3. Engine 7 is also assigned to the default queue, but there its priority is 5. So if there are jobs in both queues waiting for an engine, and Engine 7 becomes free, the knowledge team jobs will get priority.

I could remove Engine 7 from the default queue completely, saving that engine only for jobs submitted to the knowledge team’s queue. Doing that makes the Priority setting redundant. Priority only has an effect where assigning the same engine to multiple queues.

Repositories

You’ll also have spotted that each repository can be assigned to a queue. Like engines, I assign the repository to the queue, not the queue to the repository. Unlike engines, each repository is assigned to a maximum of one queue. If I add a repository to a second queue, it must be removed from the first:

By default, FME assigns all jobs to the queue of their respective workspace repository.

So I could set up a Knowledge Team repository and assign that repository to a Knowledge Team queue. Or I might assign a Knowledge Team repository to a “Safe Staff” queue along with a Support Team repository, a Sales Team repository, etc.

Or I might not assign any repositories, in which case the user running the job specifies the queue to submit it to.

Speaking of which…

Specifying a Queue

The above section shows how system administrators set up a queue. However, there are also mechanisms for users to submit jobs to a specific queue.

First of all, if a repository is assigned to a queue, and the user is happy using that queue, then nothing need be specified. They just submit the job for processing.

If a user wants to pick a queue at runtime, then they can do that in the web user interface:

Select Queue at Runtime

Queues are also assigned (and managed) through the REST API. The command /transformations/jobroutes gives a way to create a tag (queue) and to assign an engine to that tag. Then a workspace can run using /transformations/submit with a tag set using TMDirectives in the transformation request.

What Ifs

Example Setup

Let’s say I have eight engines and am setting up queues for the support, knowledge, and QA teams at Safe. I might set it up like this:

Queue Repositories Daytime Nighttime
Engines Priority Engines Priority
World Tour Demos World Tour Demos 1,2,3,4 1 1,2,3,4 1
Server Playground Demos 2,3,4 2 2,3,4 2
Support Team Short Term Support 3,4 3 3,4 3
Support Team Long Term Support 7,8 3 7,8 5
Knowledge Team Knowledge 3,4 3 3,4 3
Short Term Tests QA Short Term Tests 5,6 4 5,6 3
Long Term Tests QA Long Term Tests 7,8 4 3,4,5,6,7,8 4

So…

It’s important to remember that if the QA team starts a job on engine 3 at night, then it blocks other jobs on that engine. A job from the knowledge team that arrives later will not be able to interrupt a running job. It will, however, get priority once the QA team job is complete and the engine becomes free.

Also remember that queues and repositories are not fixed. A member of the knowledge team could still submit a job to the support queue instead of their own.

On Demand Changes

This new queuing system is very flexible because it is quickly and easily managed through the web interface.

But what’s better is that you can also manage queues using API calls (click to enlarge):

That means you could set up a scheduled process to remap queues at various times of day. For example, at 5pm you open more engines up to long-term jobs, and at 8am the next day, revert to more engines for short-term job queues.

Conclusion

So what does this all mean? Well, it means that you can fine tune FME Server to provide the best service at the best time to your entire workplace. Multiple queues allow greater flexibility and the option to separate large jobs from small.

It’s not as well documented as the Wimbledon queue system, but there’s way less etiquette to worry about, and it certainly provides better options when you’re managing server capacity.

And for more information about other new FME Server functionality, check out the Deep Dive webinar, or attend an upcoming FME World Tour event.

PS: In fact, Wimbledon has multiple queues too! One for the current day’s play, and one for the following day! Yes, people really do start queuing that far in advance when there are some interesting games on. And what do you get after queuing? Rained on…

Cloud Architecture FME Cloud FME Evangelist FME Server Interface Performance Queue Repositories Server Engine

Mark Ireland

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)

Comments

2 Responses to “Queues and Data Processing Jobs: FME Server and the Art of Queuing”

  1. Ekki says:

    Hi Mark, thanks for your article. I’d like to find out how I can force a job to a particular queue without setting the repository queue. I might publish a particular repository to a client in which most jobs may run on any queue. One of them however must always run on a particular queue (e.g. on a particular engine) to avoid clashes between parallel jobs. In there past there used to be the tm_tag parameter which could be pre-set in order to route the job to a particular engine. This method still works when running the job directly through the request URI. But it doesn’t seem to work when running the job through the UI. Is there a workaround other than having to remind the user to always select the particular queue under advanced?
    Thanks, Ekki

  2. Hi Ekki,
    Using the advanced option when manually running jobs is currently the only way to override the queue set by the repository. What might be more practical in this situation is moving that particular workspace to a different repository, and setting that up with it’s own queue with the desired engines assigned to it. The rest api and the tm_tag parameter are named the old way, but honour queues now instead, which is what you’ve found out.
    You could also add this request to our ideas exchange, as future enhancement for job queues: safe.com/ideas
    Thanks,
    Jen

Leave a Reply to Jen Luther Thomas Cancel reply

Your email address will not be published. Required fields are marked *