[Geoserver-devel] Avoiding un-needed memory and CPU consumption in abandoned WMS requests

GeoServer GeoServer Developer
1

Hi all,
recently I was investigating an OOM reported by a user
that was basically just using OpenLayers with tiles
and meta-tiling on a single machine (so one user connected
to GeoServer).

The result of the investigation is not completely new, but
it's worrisome anyways.
Basically the user was moving around a lot using OL,
panning and zooming, and the VM was configured as default,
which on the platform of this example meant only
having 64M or memory.
Each request resulted in the building of a 3x3 meta tile,
thought of course not all requests triggered that as the
code prevents the same meta tile to be computed in parallel
by more than one thread.

I've added some machinery to get a count of the concurrent
request working in parallel and usually the count was 6
(which is the default Firefox max connections) but if
someone starts zooming around while OL is still asking
for the tiles of the current level, boom, one can
easily get up to 30-40 concurrent requests and the OOM
is pretty much guaranteed.

The thing is, Firefox gives up on the older requests,
but GeoServer does not know that until it actually tries
to write anything to the response, which happens only
after the rendering is fully done.
Given that each meta tile uses 2+MB of memory,
it does not take much to fill up a 64MB heap
(especially since good part of it
is already filled with the HSQL EPSG database cache,
around 19MB, hopefully switching to H2 will give
us some breathing room in the future).

We really need to find a way to make GeoServer stop working
on requests that the client has dropped.

I've looked a bit around, here is what I've found.
Apache in CGI mode kills the cgi process as soon
as the connection is dropped.
In Java we cannot, because we're using threads, and the
threads share resources, one cannot kill one without
bad consequences.

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.

Alternatively, or in parallel to this, we could make sure no more
than X threads are rendering.
This could be done by using a concurrent queue limited in
size, each rendering action trying to push a token into it and
end up waiting if full.
This would solve the OOM, but would
make all the new request wait for the older ones to be
dropped, basically making GS WMS unusuable for a while.
Failing everything else, this may not be a such a bad idea.
With a little generalization we could apply this at the
dispatcher level and allow the administrator to set limits
to the number of requests GS is serving for each service
(typically you can serve much more WFS requests in parallel
than WMS ones).

Another option that comes to mind is to get our hands
dirty and write plugins that leverage container specific
api to check if the connection is still alive.
Downside, it would work only for specific versions of
specific containers, and I haven't checked if such an API
exists at all.

Well, do anybody have experiences on this? Suggestions?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2

Andrea Aime ha scritto:
...

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.

Hmmm... just looked into the source code of Jetty and Tomcat,
no go:
- Jetty ServletOutputStream subclass flush() won't do anything unless
   there is anything written in the output, and if there is, the
   stuff is sent do the client
- Tomcat will send the headers to the client as soon as you call it.

It seems mostly a no go. There may still be an option, but not
a very good one: if the client is asking to get the exception text
inside an image we could set the headers before starting to render,
flush them, and then periodically flush hoping that will result in
actually hitting the real output stream (the servlet containers
do wrap it).
However this will break the current dispatcher architecture...

All of a sudden using a strategy per container does not look so
bad anymore...

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

3

I don't really have anything useful to contribute except for some thoughts about the solutions presented.

First I don't view this as too much of a GeoServer flaw, but more just a limitation of the platform it runs on. I don't think its unreasonable to expect people running a server that is going to be supporting access from a tiled client to run with the appropriate amount of memory. I recently saw these same issues and just upped the memory to 256m (did not try with 128m) and saw no more issues running the styler for a few hours.

That said if it is agreed that this should be addressed I guess my preferable solution would be the rendering queue idea as I see it fitting in nicely with the longer term goal of having GeoServer support the idea of jobs and a job queue that can be tweaked by the server admin, as well as provide real time job information.

However I see container specific plugins as an interesting path as well. And it has the nice effect of not having to touch the core (besides having to perhaps add an extension point here or there).

I see either solution as viable, but i also see just documenting the limitation and stressing appropriate configuration as equally viable.

2c.

-Justin

Andrea Aime wrote:

Hi all,
recently I was investigating an OOM reported by a user
that was basically just using OpenLayers with tiles
and meta-tiling on a single machine (so one user connected
to GeoServer).

The result of the investigation is not completely new, but
it's worrisome anyways.
Basically the user was moving around a lot using OL,
panning and zooming, and the VM was configured as default,
which on the platform of this example meant only
having 64M or memory.
Each request resulted in the building of a 3x3 meta tile,
thought of course not all requests triggered that as the
code prevents the same meta tile to be computed in parallel
by more than one thread.

I've added some machinery to get a count of the concurrent
request working in parallel and usually the count was 6
(which is the default Firefox max connections) but if
someone starts zooming around while OL is still asking
for the tiles of the current level, boom, one can
easily get up to 30-40 concurrent requests and the OOM
is pretty much guaranteed.

The thing is, Firefox gives up on the older requests,
but GeoServer does not know that until it actually tries
to write anything to the response, which happens only
after the rendering is fully done.
Given that each meta tile uses 2+MB of memory,
it does not take much to fill up a 64MB heap
(especially since good part of it
is already filled with the HSQL EPSG database cache,
around 19MB, hopefully switching to H2 will give
us some breathing room in the future).

We really need to find a way to make GeoServer stop working
on requests that the client has dropped.

I've looked a bit around, here is what I've found.
Apache in CGI mode kills the cgi process as soon
as the connection is dropped.
In Java we cannot, because we're using threads, and the
threads share resources, one cannot kill one without
bad consequences.

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.

Alternatively, or in parallel to this, we could make sure no more
than X threads are rendering.
This could be done by using a concurrent queue limited in
size, each rendering action trying to push a token into it and
end up waiting if full.
This would solve the OOM, but would
make all the new request wait for the older ones to be
dropped, basically making GS WMS unusuable for a while.
Failing everything else, this may not be a such a bad idea.
With a little generalization we could apply this at the
dispatcher level and allow the administrator to set limits
to the number of requests GS is serving for each service
(typically you can serve much more WFS requests in parallel
than WMS ones).

Another option that comes to mind is to get our hands
dirty and write plugins that leverage container specific
api to check if the connection is still alive.
Downside, it would work only for specific versions of
specific containers, and I haven't checked if such an API
exists at all.

Well, do anybody have experiences on this? Suggestions?

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

4

Andrea Aime ha scritto:

Andrea Aime ha scritto:
...

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.

Hmmm... just looked into the source code of Jetty and Tomcat,
no go:
- Jetty ServletOutputStream subclass flush() won't do anything unless
   there is anything written in the output, and if there is, the
   stuff is sent do the client
- Tomcat will send the headers to the client as soon as you call it.

Maybe the problem should be attacked the other way, on the input size.
When calling read methods on the input stream they should return -1
if the stream is ended. But what if the connection has been dropped?
_maybe_ they will throw an exception instead.
Worth a try I guess

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

5

Interesting topic...

To be honest I don't really think "ability to attend 40 concurrent requests with just 64M heap" should be a requirement for GeoServer, yet, the problem scales up and upping the heap a bit just mitigates it.
The actual problem, from a robustness/reliability pov, of giving up on no longer needed requests is an amazing challenge though.
I agree with Justin this fits nicely for the longer term plans, and by thinking of it, I beleave there _might_ be a way to do so in a more standard/supported way Servlet API wise.
Let me rant about it:
If we already had some sort of concurrent job queue, the jobs would be running on a separate thread than the calling one. Jobs would obviously have some kind of identifier, and the calling thread would be waiting for the worker thread to be ready to deliver.
So, say the worker thread is taking "too long" to respond. Actually, let's say we want to periodically check on the incoming connection status (every X secs?) and decide whether to ask the Job to abort.
Since the calling thread is detached from the working thread, when we check for opening connection status we could ask the client to HTTP redirect to a different URL. This URL would contain the identifier of the running Job. So if the redirect call succeeds the client will issue another connection to the new URL and be attached to wait for the Job to finish. But if the redirect call fails (at the server) the HttpServletResponse.sendRedirect(String url) call will throw an IOException and we can safely ask the working Job to abort.
Of course we'd need to first check the client would accept redirects, but since most of them will be web browsers they should.

What do you think? too crazy?

I know this idea would not be easily implementable in the _short_ term, but might be worth having it in mind for the Job design.

my 2c/

Gabriel

Andrea Aime wrote:

Andrea Aime ha scritto:

Andrea Aime ha scritto:
...

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.

Hmmm... just looked into the source code of Jetty and Tomcat,
no go:
- Jetty ServletOutputStream subclass flush() won't do anything unless
   there is anything written in the output, and if there is, the
   stuff is sent do the client
- Tomcat will send the headers to the client as soon as you call it.

Maybe the problem should be attacked the other way, on the input size.
When calling read methods on the input stream they should return -1
if the stream is ended. But what if the connection has been dropped?
_maybe_ they will throw an exception instead.
Worth a try I guess

Cheers
Andrea

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

6

Gabriel Roldan ha scritto:

Interesting topic...

To be honest I don't really think "ability to attend 40 concurrent requests with just 64M heap" should be a requirement for GeoServer, yet, the problem scales up and upping the heap a bit just mitigates it.

Neither I do, the issue is that the clients drops the connection but the server does not notice. There are 6 real connections
there and 34 ones that we're working on for nothing.
The issue here is that a single client can push GeoServer to use
a ton of memory when zooming/scrolling a lot.

The actual problem, from a robustness/reliability pov, of giving up on no longer needed requests is an amazing challenge though.

Indeed.

I agree with Justin this fits nicely for the longer term plans, and by thinking of it, I beleave there _might_ be a way to do so in a more standard/supported way Servlet API wise.

I'm all ears (eyes, whatever)

Let me rant about it:
If we already had some sort of concurrent job queue, the jobs would be running on a separate thread than the calling one. Jobs would obviously have some kind of identifier, and the calling thread would be waiting for the worker thread to be ready to deliver.
So, say the worker thread is taking "too long" to respond. Actually, let's say we want to periodically check on the incoming connection status (every X secs?) and decide whether to ask the Job to abort.

Right. The main issue is, how do we check on the incoming connection?
Because if we can, we can just tap that into a renderer listener
and stop the rendering without the need to create our own job queue.
It would be similar to the timeout listener we have now, it creates
a timer that triggers the end of rendering, it's self contained and
quite simple, at least to me:
https://svn.codehaus.org/geoserver/trunk/src/wms/src/main/java/org/vfny/geoserver/wms/responses/RenderingTimeoutEnforcer.java

Since the calling thread is detached from the working thread, when we check for opening connection status we could ask the client to HTTP redirect to a different URL. This URL would contain the identifier of the running Job. So if the redirect call succeeds the client will issue another connection to the new URL and be attached to wait for the Job to finish. But if the redirect call fails (at the server) the HttpServletResponse.sendRedirect(String url) call will throw an IOException and we can safely ask the working Job to abort.
Of course we'd need to first check the client would accept redirects, but since most of them will be web browsers they should.

What do you think? too crazy?

I know this idea would not be easily implementable in the _short_ term, but might be worth having it in mind for the Job design.

I see... not sure if WMS clients handle redirects transparently, it something we'd have to check. Also, if the client is behind a high latency connection,
it might take some time before the redirected request jumps on the
new URL, finding the right timing for such control could be
tricky. A solution that allows us to check the connection status every
100ms or less would be better,
it took me a split second to bring GS to use 40 connection by
rolling my mouse scroller and thus zooming in rapidly.
We'd also have to redo part of the
the dispatcher, at the moment it's linear and in control of how
response is setup.

But yes, if we can't find a simpler solution and we get resourcing
to host such a change, sure, why not

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

7

I agree it's an interesting topic, not sure what scenarios we're focusing on though. Guessing:

I think Andrea is talking about the case where someone zooms in using an OpenLayers client (tiled or not), racking up a request queue of maybe 5-6 maps before getting to the one of interest.

I like the dynamic aspect of Gabriels proposal, which I think is for responses that take more than a second to render. So I guess it would not help much with the former. Not sure how many client apps would play along with the redirect either ?

My understanding is that the header-write trick Andrea mentioned during our retreat should in theory work, it's just that none of the containers let us do it? How about we propose patches for Jetty and Tomcat that allow headers to be flushed ?

I skimmed through the final draft of the 3.0 servlet spec, looks like they still intend to gather all the headers before sending anything to the client, so there is nobody is going to do this for us anytime soon. 256Mb RAM and documentation is all good, but this problem presumably increases with the number of users. The perceived performance and reliability boost could be substantial too.

-Arne

Gabriel Roldan wrote:

Interesting topic...

To be honest I don't really think "ability to attend 40 concurrent requests with just 64M heap" should be a requirement for GeoServer, yet, the problem scales up and upping the heap a bit just mitigates it.
The actual problem, from a robustness/reliability pov, of giving up on no longer needed requests is an amazing challenge though.
I agree with Justin this fits nicely for the longer term plans, and by thinking of it, I beleave there _might_ be a way to do so in a more standard/supported way Servlet API wise.
Let me rant about it:
If we already had some sort of concurrent job queue, the jobs would be running on a separate thread than the calling one. Jobs would obviously have some kind of identifier, and the calling thread would be waiting for the worker thread to be ready to deliver.
So, say the worker thread is taking "too long" to respond. Actually, let's say we want to periodically check on the incoming connection status (every X secs?) and decide whether to ask the Job to abort.
Since the calling thread is detached from the working thread, when we check for opening connection status we could ask the client to HTTP redirect to a different URL. This URL would contain the identifier of the running Job. So if the redirect call succeeds the client will issue another connection to the new URL and be attached to wait for the Job to finish. But if the redirect call fails (at the server) the HttpServletResponse.sendRedirect(String url) call will throw an IOException and we can safely ask the working Job to abort.
Of course we'd need to first check the client would accept redirects, but since most of them will be web browsers they should.

What do you think? too crazy?

I know this idea would not be easily implementable in the _short_ term, but might be worth having it in mind for the Job design.

my 2c/

Gabriel

Andrea Aime wrote:
  

Andrea Aime ha scritto:
    

Andrea Aime ha scritto:
...

I looked into the servlet API but could find no "supported" way to
actually guess if the client connection is still alive
or not, it seems one has actually to try and write something
on the output.
I asked on the Sun J2EE servlet forum and got a couple of answers:
http://forums.sun.com/thread.jspa?threadID=5408542

The idea of trying to flush() periodically seems to be a good
one, I've read in other places that flushing the output
stream should not turn the response into committed status:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4362880

The reason it's important that flush() does not commit
the response is that by the time one commits the response
the headers have to be set and cannot be modified,
and our dispatch system sets them only after the response
object has been created (the fully rendered image in our case).
Since we want to try periodic flush() call during the rendering
we would be in troubles, as the headers are set only after that.
        

Hmmm... just looked into the source code of Jetty and Tomcat,
no go:
- Jetty ServletOutputStream subclass flush() won't do anything unless
   there is anything written in the output, and if there is, the
   stuff is sent do the client
- Tomcat will send the headers to the client as soon as you call it.
      

Maybe the problem should be attacked the other way, on the input size.
When calling read methods on the input stream they should return -1
if the stream is ended. But what if the connection has been dropped?
_maybe_ they will throw an exception instead.
Worth a try I guess

Cheers
Andrea

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers