[Geoserver-users] GeoServer crash by tilecache seeding

2StepForward · September 8, 2008, 11:40am

Hi List,

I have a problem with the seeding process of some image data.
I run a Linux OS and GeoServer as server support. It takes 256 MB space for
image opertation and I get this error from tilecache:

Traceback (most recent call last):
  File "./tilecache_seed.py", line 9, in ?
    TileCache.Client.main()
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Client.py", line 108, in
main
    seed(svc, base, layer, map(int, sys.argv[3:5]), map(float,
sys.argv[5].split(",")))
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Client.py", line 92, in seed
    svc.renderTile(tile)
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Service.py", line 138, in
renderTile
    data = layer.render(tile)
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Layer.py", line 411, in
render
    return self.renderTile(tile)
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Layers/WMS.py", line 29, in
renderTile
    tile.data, response = wms.fetch()
  File "/srv/www/cgi-bin/tilecache_sh/TileCache/Client.py", line 64, in
fetch
    raise Exception("Did not get image data back. \nURL: %s\nContent-Type
Header: %s\nResponse: \n%s" % (self.url(), ctype, data))
Exception: Did not get image data back.
URL:
http://localhost:8081/geoserver/wms?layers=CGISTMS%3ASH25T&styles=&service=WMS&width=256&format=image%2Fpng&request=GetMap&height=256&srs=EPSG%3A25832&version=1.1.1&bbox=32600360.0%2C6060000.0%2C32601000.0%2C6060640.0
Content-Type Header: application/vnd.ogc.se_xml
Response:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE
ServiceExceptionReport SYSTEM
"http://schemas.opengis.net/wms/1.1.1/WMS_exception_1_1_1.dtd">
<ServiceExceptionReport version="1.1.1" > <ServiceException>
      java.lang.OutOfMemoryError: Java heap space
Java heap space
Details:
org.geoserver.platform.ServiceException: java.lang.OutOfMemoryError: Java
heap space
        at org.geoserver.ows.Dispatcher.exception(Dispatcher.java:1177)
        at
org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:198)
        at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)
        at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
        at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)
        at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)
        at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)
        at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:347)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:459)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1054)
        at
org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
        at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1045)
        at
org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:69)
"tilecache_seed.py.out" 109L, 8909C

I don't know why, because I have give a amount of 600 MB memory to the
GeoServer and same things will happend.

Maybe not the tilecache will force this problem, could it be. Further the
geoserver crashes by production of some fetched image data?

Edit: I have additionally posted this massage in the tilecache forum, so i
don't know how of both, GeoServer or tilecache force this problem

Has anyone a idea what the problem could be, or a tip where I could search
for this crashes

Thanks a lot,
Klaus

--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19370310.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · September 8, 2008, 1:16pm

2StepForward ha scritto:

Hi List,

I have a problem with the seeding process of some image data.
I run a Linux OS and GeoServer as server support. It takes 256 MB space for
image opertation and I get this error from tilecache:

...

Java heap space
Details:
org.geoserver.platform.ServiceException: java.lang.OutOfMemoryError: Java

...

Edit: I have additionally posted this massage in the tilecache forum, so i
don't know how of both, GeoServer or tilecache force this problem

Has anyone a idea what the problem could be, or a tip where I could search
for this crashes

The amount of memory GeoServer needs depends on various factors.
Usually in tile caching there are three major factors:
- the metatiling factor, which decides how big the images requested to
geoserver really are
- the image type
- the amount of threads used during the seeding (not sure if TileCache
is able to do parallel seeding).

Say you want back a jpeg or png image, it has to be generated full color, so it would be 32bits per pixel. A tile is 256x256, let's assume
you are using a metatiling factor of 4, then you're really requesting
a 1024x1024 image each time, meaning 4MB per image.
If you're seeding with 2 parallel threads, it would mean a total of 8MB,
which is not much.

You said "It takes 256 MB space for image opertation"... hum... what's
the metatingling factor you're using?

Also, beware of the jai cache, it's configure by default to use 50%
of the available memory, reducing your 600MB to 300MB available for
image operations. You can tune this in the config/server panel.

Hope this helps
Cheers
Andrea

2StepForward · September 8, 2008, 2:22pm

Andrea Aime-4 wrote:

The amount of memory GeoServer needs depends on various factors.
Usually in tile caching there are three major factors:
- the metatiling factor, which decides how big the images requested to
geoserver really are
- the image type
- the amount of threads used during the seeding (not sure if TileCache
is able to do parallel seeding).

Say you want back a jpeg or png image, it has to be generated full
color, so it would be 32bits per pixel. A tile is 256x256, let's assume
you are using a metatiling factor of 4, then you're really requesting
a 1024x1024 image each time, meaning 4MB per image.
If you're seeding with 2 parallel threads, it would mean a total of 8MB,
which is not much.

I get back a full color 32bits png format. The metatileing has an factor of
31....
The original file has a dimension of 8000x8000 pixel and will separeted in
256x256.
If i know the seed process will use only one thread to produce each tile, so
we get an amount of 31 MB per
fatch.

Additionaly I have to say that, this map has 60teen images of 8000x8000
pixels, I know this is a lot.
And all images where handled by ImageMosaic-Plugin.
A long time the produce process runs normaly, but in other case it crashes
like the above error.

Best Regards,
Klaus
--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19373104.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · September 9, 2008, 6:20am

2StepForward ha scritto:

Andrea Aime-4 wrote:

The amount of memory GeoServer needs depends on various factors.
Usually in tile caching there are three major factors:
- the metatiling factor, which decides how big the images requested to
geoserver really are
- the image type
- the amount of threads used during the seeding (not sure if TileCache
is able to do parallel seeding).

Say you want back a jpeg or png image, it has to be generated full color, so it would be 32bits per pixel. A tile is 256x256, let's assume
you are using a metatiling factor of 4, then you're really requesting
a 1024x1024 image each time, meaning 4MB per image.
If you're seeding with 2 parallel threads, it would mean a total of 8MB,
which is not much.

I get back a full color 32bits png format. The metatileing has an factor of
31....
The original file has a dimension of 8000x8000 pixel and will separeted in
256x256.
If i know the seed process will use only one thread to produce each tile, so
we get an amount of 31 MB per
fatch.

Hem, not exactly? Metatiling factor of 31 means an image of 7936x7936 pixels, per 4 bytes per pixel, means 246MB per fetch. Given that the
JAI tile cache is eating half of your &00MB at least (double check in
the config/server section) I'm not surprised you're occasionally
running out of memory. If you really need to deal with such big
metatiles, you'll have to increase the memory you give GeoServer
or reduce the JAI cache size to, say, 20%, if you're tile caching
you don't need it to be very big anyways.

Hope this helps
Cheers
Andrea

2StepForward · September 9, 2008, 9:13am

Andrea Aime-4 wrote:

Hem, not exactly? Metatiling factor of 31 means an image of 7936x7936
pixels, per 4 bytes per pixel, means 246MB per fetch. Given that the
JAI tile cache is eating half of your &00MB at least (double check in
the config/server section) I'm not surprised you're occasionally
running out of memory. If you really need to deal with such big
metatiles, you'll have to increase the memory you give GeoServer
or reduce the JAI cache size to, say, 20%, if you're tile caching
you don't need it to be very big anyways.

Hi Andrea,

yes, you are rigth. The factor is 31,25 and it will take 250MB per fetch.
If the specifiation in the startup.sh is -Xms3m -Xmx600m than it will get
300MB by a JAI cache size of 50%.

But I don't know why it isn't enouth for JAI image operation. It will be
needed only 250MB per fetch, isn't it?

Thank you for your help and on this place I want to say what the GeoServer
team has done is a great job and a really good solution in case of
GIS-Server support.

Best Regards,
Klaus
--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19388663.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · September 9, 2008, 12:56pm

2StepForward ha scritto:

yes, you are rigth. The factor is 31,25 and it will take 250MB per fetch.
If the specifiation in the startup.sh is -Xms3m -Xmx600m than it will get
300MB by a JAI cache size of 50%.

But I don't know why it isn't enouth for JAI image operation. It will be
needed only 250MB per fetch, isn't it?

Well, during its operation GeoServer will cache other stuff. The EPSG
database is partially cached in memory, the XML schemas used for WFS operation are too, and then all the bytecode for the classes used
by GeoServer is kept in the heap as well. That means you need at least
(rough guess) 64 or 128 more to make it working for good.
A simple way to do so is to reduce the JAI tile cache to 30%.

Cheers
Andrea

2StepForward · September 10, 2008, 10:42am

Andrea Aime-4 wrote:

Well, during its operation GeoServer will cache other stuff. The EPSG
database is partially cached in memory, the XML schemas used for WFS
operation are too, and then all the bytecode for the classes used
by GeoServer is kept in the heap as well. That means you need at least
(rough guess) 64 or 128 more to make it working for good.
A simple way to do so is to reduce the JAI tile cache to 30%.

Hello Andrea,

sadly I've got two crashes today, followed in a short time of half an houre.
Yesterday and today I've try it out to reduce the JAI cache size to 30% by
an amount of 600MB max space.
It seems to me, that this could not be the problem for the permanently
crashes.
Also I have try it out with a space of 1,5 GB and this should be enougth for
all.

Here is backtrace code:

WARN [geoserver.ows] -
java.lang.OutOfMemoryError: Java heap space
  at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
  at org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)
  at org.geoserver.ows.Dispatcher.response(Dispatcher.java:631)
  at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:192)
  at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)
  at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
  at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)
  at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)
  at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)
  at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:347)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:459)
  at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1054)
  at
org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
  at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1045)
  at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:69)
  at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1045)
  at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:47)
  at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1045)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
  at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
  at
org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
  at
org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
  at
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
  at
org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:178)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
  at
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)
  at
org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)

For me it look like a problem with the file flush, but I can't find any
correlation to the setup or the server file system.
Maybe you or anyone knows where it is a case of search to solve this
problem.

Thanks,
Klaus
--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19410287.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · September 10, 2008, 12:28pm

2StepForward ha scritto:

Andrea Aime-4 wrote:

Well, during its operation GeoServer will cache other stuff. The EPSG
database is partially cached in memory, the XML schemas used for WFS operation are too, and then all the bytecode for the classes used
by GeoServer is kept in the heap as well. That means you need at least
(rough guess) 64 or 128 more to make it working for good.
A simple way to do so is to reduce the JAI tile cache to 30%.

Hello Andrea,

sadly I've got two crashes today, followed in a short time of half an houre.
Yesterday and today I've try it out to reduce the JAI cache size to 30% by
an amount of 600MB max space.
It seems to me, that this could not be the problem for the permanently
crashes.
Also I have try it out with a space of 1,5 GB and this should be enougth for
all.

Here is backtrace code:

WARN [geoserver.ows] - java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
at org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)

...

Yikes, file strategy? Did you change it by hand? Go into the geoserver
web.xml and set it to SPEED if you care only about WMS, or to PARTIAL-BUFFER2 if you need WFS as well. File strategy will dump all
the data into a temporary file before sending to the client, it's
highly inefficient... did you change it by hand? Which version of
GeoServer are you using?

For me it look like a problem with the file flush, but I can't find any
correlation to the setup or the server file system.

No, OOM errors don't work like that. You have something misbehaving that
is filling up the memory, then comes anything that needs to allocate
a bit more and you see the stack trace of that one. Basically, you
see the last drop that makes the glass pour, but not what really filled it. The only way to spot it is to run GeoServer in a memory profiler
(such as Yourkit) and the skill to interpret the result of the profiling
session.
I can do that, but I need a totally reproducible case, your data, sld,
and tilecache configuration.

Cheers
Andrea

2StepForward · September 10, 2008, 1:45pm

Andrea Aime-4 wrote:

Yikes, file strategy? Did you change it by hand? Go into the geoserver
web.xml and set it to SPEED if you care only about WMS, or to
PARTIAL-BUFFER2 if you need WFS as well. File strategy will dump all
the data into a temporary file before sending to the client, it's
highly inefficient... did you change it by hand? Which version of
GeoServer are you using?

Yes, I have changed serviceStrategy by hand to SPEED. This one could not the
reason.
It runs a long time with the SPEED configuration.
I use the GeoServer version 1.6.3.

Best Regards,
Klaus
--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19413963.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · September 10, 2008, 1:52pm

2StepForward ha scritto:

Andrea Aime-4 wrote:

Yikes, file strategy? Did you change it by hand? Go into the geoserver
web.xml and set it to SPEED if you care only about WMS, or to PARTIAL-BUFFER2 if you need WFS as well. File strategy will dump all
the data into a temporary file before sending to the client, it's
highly inefficient... did you change it by hand? Which version of
GeoServer are you using?

Yes, I have changed serviceStrategy by hand to SPEED. This one could not the
reason.
It runs a long time with the SPEED configuration.
I use the GeoServer version 1.6.3.

The stack trace clearly shows the file strategy is being used instead:

java.lang.OutOfMemoryError: Java heap space
  at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
  at org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)
  at org.geoserver.ows.Dispatcher.response(Dispatcher.java:631)
  at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:192)

See this:
org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)

can you double check?

Also, I suggest you try out 1.7.0-RC2, since the 1.6.x series is dead,
1.6.5 is already in the making, and no developer is working
on that branch anymore.

Cheers
Andrea

2StepForward · September 10, 2008, 2:36pm

Andrea Aime-4 wrote:

The stack trace clearly shows the file strategy is being used instead:

java.lang.OutOfMemoryError: Java heap space
  at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
  at org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)
  at org.geoserver.ows.Dispatcher.response(Dispatcher.java:631)
  at
org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:192)

See this:
org.vfny.geoserver.servlets.FileStrategy.flush(FileStrategy.java:115)

can you double check?

Hello,

yes I'm sure that's my web.xml serviceStrategy is configured on SPEED.
I don't know why it beliefs that it is configured on BUFFER, or
PARTIAL-BUFFER2, but our right it shows a BufferedOutputStream at least.

Or I miss understand this and it beliefs on configuration FILE ->
FileStrategy.flush(FileStrategy.java:115)?

Here is a part of my web.xml:

<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application
2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
<display-name>GeoServer</display-name>
<description>Web Mapping, Feature and Coverage Services</description>

<context-param>
    <param-name>serviceStrategy</param-name>
    <!-- Meaning of the different values :

         PARTIAL-BUFFER2
         - Partially buffers the first xKb to disk. Once that has buffered,
the the
           result is streamed to the user. This will allow for most errors
to be caught
           early.

         BUFFER
         - stores the entire response in memory first, before sending it off
to
           the user (may run out of memory)

         SPEED
         - outputs directly to the response (and cannot recover in the case
of an
           error)

         FILE
         - outputs to the local filesystem first, before sending it off to
the user
      -->
    <param-value>SPEED</param-value>
  </context-param>

<context-param>
    <param-name>enableVersioning</param-name>
    <param-value>true</param-value>

This should be right, isn't it?

Cheers,
Klaus
--
View this message in context: http://www.nabble.com/GeoServer-crash-by-tilecache-seeding-tp19370310p19414598.html
Sent from the GeoServer - User mailing list archive at Nabble.com.