[Geoserver-users] Geoserver slowdowns

Blaz · March 12, 2009, 8:52am

Hello all. I have a very annoying problem. We had our web gis application
running on GS 1.6.3 and all was working ok. Then we upgraded 1.6.3 to 1.7.2
and then 1.7.3 and we started having problems.

Geoserver (1.7.3, same goes for 1.7.2) works fast as hell for first 20
minutes or so. Then it slows down drastically (generation of 1 tile (jpeg or
gif or png) which prior to slowdown took 100ms now takes 1.7s). But this
only happens for 1 coverage: mosaic with 80.000 tiles, 1MB per tile,
~~1000x1000 pixels per tile (only shown on lowest level so only 2-4 tiles
get read from the disk). All other coverages and datastores work fast as
they did.

So why would this slowdown happen on 1.7.3 if it was working ok on 1.6.3?

I've also checked the log and so something weird?

After the line: asking delegate for write to
org.geoserver.ows.DispatcherOutputStream 1000s of chars appear for no
apparent reason? And this happens many times (according to log).

http://www.shrani.si/f/2N/7t/4wvCVasa/log.jpg </img>
--
View this message in context: http://www.nabble.com/Geoserver-slowdowns-tp22471444p22471444.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · March 12, 2009, 9:10am

GeoUser ha scritto:

Hello all. I have a very annoying problem. We had our web gis application
running on GS 1.6.3 and all was working ok. Then we upgraded 1.6.3 to 1.7.2
and then 1.7.3 and we started having problems.

Geoserver (1.7.3, same goes for 1.7.2) works fast as hell for first 20
minutes or so. Then it slows down drastically (generation of 1 tile (jpeg or
gif or png) which prior to slowdown took 100ms now takes 1.7s). But this
only happens for 1 coverage: mosaic with 80.000 tiles, 1MB per tile,
~~1000x1000 pixels per tile (only shown on lowest level so only 2-4 tiles
get read from the disk). All other coverages and datastores work fast as
they did.

So why would this slowdown happen on 1.7.3 if it was working ok on 1.6.3?

It's called a regression. Stuff that used to work and does not anymore
because other unrelated improvement/fixes impacted that code path
and nobody noticed.

Simone, did anything change in the way the readers are cached in a
mosaic? Or else, in the way mosaics are built?

I've also checked the log and so something weird?

After the line: asking delegate for write to
org.geoserver.ows.DispatcherOutputStream 1000s of chars appear for no
apparent reason? And this happens many times (according to log).

http://www.shrani.si/f/2N/7t/4wvCVasa/log.jpg </img>

Ugh, that is debug level output, but anyways, we should never dump
to the log a raw image (assuming the long sequence of funny chars
is actually that, a dump of an image binary into text).
However, I've tried setting the logging level to "VERBOSE" and
I cannot get that output on my machine. What kind of requests
are you making to get that?

Also, can you look in the logs and see if you find any
OutOfMemory error?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Blaz · March 12, 2009, 10:52am

Andrea Aime-4 wrote:

Ugh, that is debug level output, but anyways, we should never dump
to the log a raw image (assuming the long sequence of funny chars
is actually that, a dump of an image binary into text).
However, I've tried setting the logging level to "VERBOSE" and
I cannot get that output on my machine. What kind of requests
are you making to get that?

Also, can you look in the logs and see if you find any
OutOfMemory error?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

No memory leaks, requests are just regular wms get map requests...

--
View this message in context: http://www.nabble.com/Geoserver-slowdowns-tp22471444p22473015.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Ugo_Taddei · March 12, 2009, 12:08pm

Hello,

we have recently come across a problem with open files in gs (version 1.6.x) probably caused by the mosaic plugin. (Mosaic layers were the only ones we had.)

We worked around by calling http://localhost:8080/geoserver/admin/freeMemory.do

which garbage collected and closed the files. (Not sure it's a geoserver problem.) OS was Linux.

Cheers,

Ugo

GeoUser wrote:

Hello all. I have a very annoying problem. We had our web gis application
running on GS 1.6.3 and all was working ok. Then we upgraded 1.6.3 to 1.7.2
and then 1.7.3 and we started having problems.

Geoserver (1.7.3, same goes for 1.7.2) works fast as hell for first 20
minutes or so. Then it slows down drastically (generation of 1 tile (jpeg or
gif or png) which prior to slowdown took 100ms now takes 1.7s). But this
only happens for 1 coverage: mosaic with 80.000 tiles, 1MB per tile,
~~1000x1000 pixels per tile (only shown on lowest level so only 2-4 tiles
get read from the disk). All other coverages and datastores work fast as
they did.

So why would this slowdown happen on 1.7.3 if it was working ok on 1.6.3?

I've also checked the log and so something weird?

After the line: asking delegate for write to
org.geoserver.ows.DispatcherOutputStream 1000s of chars appear for no
apparent reason? And this happens many times (according to log).

http://www.shrani.si/f/2N/7t/4wvCVasa/log.jpg </img>

--
Ugo Taddei

Fraunhofer Institut Intelligente Analyse- und Informationssysteme (FhG IAIS)
http://www.iais.fraunhofer.de
Department Knowledge Discovery - IAIS.KD -
Working Group Spatial Decision Support
http://www.iais.fraunhofer.de/kd.html
phone (+49)2241-14-2184 fax (+49)2241-14-2072
Schloss Birlinghoven, D-53754 Sankt Augustin, Germany

Andrea_Aime3 · March 12, 2009, 12:25pm

Ugo Taddei ha scritto:

Hello,

we have recently come across a problem with open files in gs (version 1.6.x) probably caused by the mosaic plugin. (Mosaic layers were the only ones we had.)

We worked around by calling http://localhost:8080/geoserver/admin/freeMemory.do

which garbage collected and closed the files. (Not sure it's a geoserver problem.) OS was Linux.

The mosaic plugin keeps a cache of open files to get a very
noticeable perf. improvement as opposed of having to open the
files over and over.
That can hit with the max limit of open files that the OS
allows for, but it can be worked around by using, for example,
ulimit under linux.
Freeing the memory also has the side effect of forcing the
cache to be wiped out, and that closes the file descriptors
as well.

Thought, usually that issue results in hard exceptions about
not being able to open more files, not slowdowns?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Ugo_Taddei · March 12, 2009, 1:55pm

Andrea Aime wrote:

Ugo Taddei ha scritto:

Hello,

we have recently come across a problem with open files in gs (version 1.6.x) probably caused by the mosaic plugin. (Mosaic layers were the only ones we had.)

We worked around by calling http://localhost:8080/geoserver/admin/freeMemory.do

which garbage collected and closed the files. (Not sure it's a geoserver problem.) OS was Linux.

The mosaic plugin keeps a cache of open files to get a very
noticeable perf. improvement as opposed of having to open the
files over and over.
That can hit with the max limit of open files that the OS
allows for, but it can be worked around by using, for example,
ulimit under linux.
Freeing the memory also has the side effect of forcing the
cache to be wiped out, and that closes the file descriptors
as well.

Thought, usually that issue results in hard exceptions about
not being able to open more files, not slowdowns?

Exactly that was the problem in our case (socket exception, I think), not a slowdown. However, a slowdown can be caused by the garbage collector being called.

With a mosaic with 80k tiles, you're bound to hit ulimit pretty soon, when people use the wms a lot. Not?

Cheers
Andrea

Cheers,

Ugo

--
Ugo Taddei

Fraunhofer Institut Intelligente Analyse- und Informationssysteme (FhG IAIS)
http://www.iais.fraunhofer.de
Department Knowledge Discovery - IAIS.KD -
Working Group Spatial Decision Support
http://www.iais.fraunhofer.de/kd.html
phone (+49)2241-14-2184 fax (+49)2241-14-2072
Schloss Birlinghoven, D-53754 Sankt Augustin, Germany

Blaz · March 13, 2009, 10:28am

Yes, it's weird, we get no exceptions. All other layers work normally. Even
other mosaics (which are not as big as the problematic one).

Everything goes back to normal when i restart GS.

Ugo Taddei-2 wrote:

Andrea Aime wrote:

Ugo Taddei ha scritto:

Hello,

we have recently come across a problem with open files in gs (version
1.6.x) probably caused by the mosaic plugin. (Mosaic layers were the
only ones we had.)

We worked around by calling
http://localhost:8080/geoserver/admin/freeMemory.do

which garbage collected and closed the files. (Not sure it's a
geoserver problem.) OS was Linux.

The mosaic plugin keeps a cache of open files to get a very
noticeable perf. improvement as opposed of having to open the
files over and over.
That can hit with the max limit of open files that the OS
allows for, but it can be worked around by using, for example,
ulimit under linux.
Freeing the memory also has the side effect of forcing the
cache to be wiped out, and that closes the file descriptors
as well.

Thought, usually that issue results in hard exceptions about
not being able to open more files, not slowdowns?

Exactly that was the problem in our case (socket exception, I think),
not a slowdown. However, a slowdown can be caused by the garbage
collector being called.

With a mosaic with 80k tiles, you're bound to hit ulimit pretty soon,
when people use the wms a lot. Not?

Cheers
Andrea

Cheers,

Ugo

--
Ugo Taddei

Fraunhofer Institut Intelligente Analyse- und Informationssysteme (FhG
IAIS)
http://www.iais.fraunhofer.de
Department Knowledge Discovery - IAIS.KD -
Working Group Spatial Decision Support
http://www.iais.fraunhofer.de/kd.html
phone (+49)2241-14-2184 fax (+49)2241-14-2072
Schloss Birlinghoven, D-53754 Sankt Augustin, Germany

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://www.nabble.com/Geoserver-slowdowns-tp22471444p22493760.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Simone_Giannecchini3 · April 8, 2009, 11:12am

Hi guys,
resurrecting an old thread.
These slowdowns have two causes which can kick in at the same time to
make things very bad:

1> the current version of most plugins uses deferred loading via JAI
ImageRead, which at the time seemed a very good option to me. However,
this means opening up a file and leaving it open for a while, closing
it eventually when garbage collecting the stream references. With a
mosaic of thousand of small image this can clobber the GC and make it
freeze your machine (well, depending on wich type of GC in use )
2> due to the same cause as outlined in point 1, we open up many files
at the same time and this can slowdonw machines or make them run out
of file descriptors

I am currently doing some work on the mosaic plugin to outcome these
problems. The first part of this work will end up on gt 2.5.x / gs
1.7.x the remaining will be performed on gt-trunk.

Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Fri, Mar 13, 2009 at 12:28 PM, GeoUser <blazrepnik@anonymised.com> wrote:

Yes, it's weird, we get no exceptions. All other layers work normally. Even
other mosaics (which are not as big as the problematic one).

Everything goes back to normal when i restart GS.

Ugo Taddei-2 wrote:

Andrea Aime wrote:

Ugo Taddei ha scritto:

Hello,

we have recently come across a problem with open files in gs (version
1.6.x) probably caused by the mosaic plugin. (Mosaic layers were the
only ones we had.)

We worked around by calling
http://localhost:8080/geoserver/admin/freeMemory.do

which garbage collected and closed the files. (Not sure it's a
geoserver problem.) OS was Linux.

The mosaic plugin keeps a cache of open files to get a very
noticeable perf. improvement as opposed of having to open the
files over and over.
That can hit with the max limit of open files that the OS
allows for, but it can be worked around by using, for example,
ulimit under linux.
Freeing the memory also has the side effect of forcing the
cache to be wiped out, and that closes the file descriptors
as well.

Thought, usually that issue results in hard exceptions about
not being able to open more files, not slowdowns?

Exactly that was the problem in our case (socket exception, I think),
not a slowdown. However, a slowdown can be caused by the garbage
collector being called.

With a mosaic with 80k tiles, you're bound to hit ulimit pretty soon,
when people use the wms a lot. Not?

Cheers
Andrea

Cheers,

Ugo

--
Ugo Taddei

Fraunhofer Institut Intelligente Analyse- und Informationssysteme (FhG
IAIS)
http://www.iais.fraunhofer.de
Department Knowledge Discovery - IAIS.KD -
Working Group Spatial Decision Support
http://www.iais.fraunhofer.de/kd.html
phone (+49)2241-14-2184 fax (+49)2241-14-2072
Schloss Birlinghoven, D-53754 Sankt Augustin, Germany

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://www.nabble.com/Geoserver-slowdowns-tp22471444p22493760.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users