[Geoserver-devel] memory problems

Hi all,

I'm increasingly experimenting with serving rasters through geoserver,
and getting some severe memory problems recently. Part of an example
stack trace follows below.

Problem is that after some time of serving, geoserver gets out of heap
space, and just serves error in the tomcat log instead of images.
Tomcat is started with -Xmx2000m, so I'd say heap space should be
enough. JAI and JAI-imageio are installed, cvs versions of not so long
ago. Jdk is sun-jdk-1.5.0_09. Geoserver is 1.5.x of yesterday, geotools
is 2.3.x of yesterday.
CoverageStores usually are tiled geotiffs (rgba) with internal
overviews.
wms usually requests png rgba images.

I'd like to know if there are ways to (memory) optimize this setup, and
besides that if there is some way to monitor and/or debug memory usage
by geoserver.

Things I thought I could try (please comment):
- update jdk to 1.5.0_10
- maybe update jdk to 1.6? Any issues to be expected?
- use geotools' mosaic and pyramid builders instead of geotiffs?
- ???

Below is some example stack trace part, though I doubt it is of much
use...

Cheers,
Vincent.

SEVERE: Service exception occurred
org.vfny.geoserver.wms.WmsException: Java heap space
        at
org.vfny.geoserver.wms.WmsExceptionHandler.newServiceException(WmsExceptionHandler.java:65)
at
org.vfny.geoserver.servlets.AbstractService.sendError(AbstractService.java:888)
at
org.vfny.geoserver.servlets.AbstractService.doService(AbstractService.java:550)
at
org.vfny.geoserver.servlets.AbstractService.doGet(AbstractService.java:334)
at org.geoserver.request.Dispatcher.dispatch(Dispatcher.java:196) at
org.geoserver.request.Dispatcher.handleRequestInternal(Dispatcher.java:56)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)
at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:347)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767) at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595) Caused by:
java.lang.OutOfMemoryError: Java heap space Feb 14, 2007 1:30:27 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:25 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:23 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:20 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:18 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:13 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:11 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:10 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:30:00 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint
WARNING: Java heap space java.lang.OutOfMemoryError: Java heap space
Feb 14, 2007 1:29:56 PM
org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer paint

Vincent Schut ha scritto:

Hi all,

I'm increasingly experimenting with serving rasters through geoserver,
and getting some severe memory problems recently. Part of an example
stack trace follows below.

Problem is that after some time of serving, geoserver gets out of heap
space, and just serves error in the tomcat log instead of images.
Tomcat is started with -Xmx2000m, so I'd say heap space should be
enough. JAI and JAI-imageio are installed, cvs versions of not so long
ago. Jdk is sun-jdk-1.5.0_09. Geoserver is 1.5.x of yesterday, geotools
is 2.3.x of yesterday.

Damn... no, there should be some persistent data structure that's accumulating references, the java version of a memory leak.
When the free memory decreses with requests, there's a memory leak
around that should be found and killed without mercy.

I'll have to setup a stress test for WMS with rasters and find
what's going on.

Hmmm... we should setup some kind of stress testing that can
be run automatically, usually when there's a memory leak it's enough
to start Geoserver with a small heap (32/64MB heap) and leave
it running for some minutes under stress, if there is a leak,
it just OOM after a few while.

Thank you for reporting this Vincent, very much appreciated :slight_smile:
Cheers
Andrea

Vincent Schut ha scritto:

Hi all,

I'm increasingly experimenting with serving rasters through geoserver,
and getting some severe memory problems recently. Part of an example
stack trace follows below.

Quick note to let you know Alessio has some ideas on why this is
happening and he's working on it. Keep fingers crossed.

Cheers
Andrea

On Wed, 14 Feb 2007 17:15:42 +0100
Andrea Aime <aaime@anonymised.com> wrote:

Vincent Schut ha scritto:
> Hi all,
>
> I'm increasingly experimenting with serving rasters through
> geoserver, and getting some severe memory problems recently. Part
> of an example stack trace follows below.

Quick note to let you know Alessio has some ideas on why this is
happening and he's working on it. Keep fingers crossed.

Cheers
Andrea

OK, thanks for the quick action.
BTW, I'm sorry to tell you but from today I won't be in the office
until 26 Februari, and I won't be able to do much if any testing. I'll
pick thins up again after I returned. If feels rather inconvenient to
find a bug like this just before you go away for more than a week...

Hope your ideas are correct, Alessio!

Cheers
Vincent.