Understanding Tomcat Executor thread pooling
October 30, 2010 1 Comment
In the default configuration, Tomcat will always create a bounded worker-thread pool for each Connector (with max-size 200). Mostly, this is not something that you’ll need to change (other than maybe increasing the max threads to accomodate for higher load). However, like I discussed in my previous post, Tomcat has a propensity for caching a lot of scaffolding objects (like PageContext and tag buffers) in thread-local context in each worker thread. Because of this, there are instances where you might want Tomcat to be able to close threads down to clean out some memory. Also, having each connector maintaining it’s own pool makes it harder to set a firm top-limit on what load your server will accept. The answer to this is to use a shared Executor.
By having all connectors share the same executor, you can configure with more predictability how many simultaneous requests that is allowed to run across your entire application. The Executor also brings the ability of having a thread-pool that can shrink as well as grow to accomodate load. At least in theory…
The standard, built-in executor that Tomcat uses by default is the StandardThreadExecutor. The configuration is documented here: http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html
The configuration options include the somewhat misnamed paremter “maxIdleTime” and here is what you need to be aware of regarding the standard executor and closing idle threads.
The standard executor internally uses a java.util.concurrent.ThreadPoolExecutor. This works (somewhat simplified) by having a variable size pool of worker-threads that, once they have completed a task, will wait on a blocking queue until a new task is entered. Or until it has waited for a set amount of time, in which case it will have “timed out” and the thread will be closed. The crux of this is that since the first thread to complete a task will be first in line to get a new task, the pool will behave in a First-In-First-Out (FIFO) way. This is important to keep in mind when we examine how this will affect the Tomcat executor.
maxIdleTime is really minIdleTime
Because of the FIFO behaviour of the java ThreadPoolExecutor, each thread will at minimum wait for a new task for “maxIdleTime” before being eligable for closure. Moreover, again because of the FIFO behaviour of the thread pool, for a thread to be closed it’s required that a period of time at least equal to maxIdleTime passes without ANY request coming in, since the thread that has been idle the longest will be first in line for a new task. The effect is that the executor will not really be able to size the pool to fit the average load (concurrent requests), it will rather be sized according to the rate at which requests come in. This may sound like a distinction without a difference but in terms of a web-server, it’s quite significant. For example, 40 requests come in at the same time. The thread-pool will be expanded to 40 to accomodate the load. After that, you have a period where only one request comes in at a time. Say each request takes 500 ms to complete, that means it would take 20 seconds to cycle through the entire thread-pool (remember, FIFO). Unless you have your maxIdleTime set to less than 20 seconds, the pool will continue to hold 40 threads indefinitly, even though the concurrent load is never more than 1. And you don’t want to set your maxIdleTime too low either – that will risk flapping behaviour where threads are killed too soon.
To get a more predictable thread-pooling behaviour that attempts to size to average load rather than to rate of requests coming in, it would be preferable to have an executor that worked on a “Last-In-First-Out” (LIFO) basis. If the pool would always assign the thread that had been idle to SHORTEST period of time to incoming tasks, the server would be better equipped to close down threads during periods of lower load (and in a more predictable manner). In the very simplistic example above, the initial load of 40 followed by a period with a load of 1, a LIFO pool would correctly size down to 1 after the maxIdleTime period. Of course, it may not always be required (or desired) to have such an aggressive puring strategy but if your goal is to minimize the amount of resources reserved by Tomcat, the standard executor might unfortunately not be able do what you expect it to do for you.