[Geoserver-users] Out of Memory errors

GeoServer GeoServer User
1

Windows Server 2008

Geoserver 2.3.5

Java 1.7.13 64bit

Hi all,

Every so often Geoserver stops responding for about 20 minutes at a time. This generally happens every couple of days depending on usage. The errors reported in the log are as follows:

2013-12-20 12:05:07,990 WARN [geotools.jdbc] - There is code leaving feature readers/iterators open, this is leaking statements and connections!
2013-12-20 12:17:52,501 ERROR [geotools.map] - Call MapContent dispose() to prevent memory leaks
2013-12-20 12:17:54,357 ERROR [geoserver.ows] -
java.lang.OutOfMemoryError: Java heap space

Geoserver eventually restarts itself and starts responding again.

I am going to assign some more memory but it already uses 1.5GB and I suspect that may simply allow it to last longer before crashing? My question is, is there anything else I can do? Is this fixed in later versions? Does the data source type matter? We’re using Oracle which I know can be a memory hog for client connections.

Tom

2

On Fri, Dec 20, 2013 at 2:21 PM, Tom (JDi Solutions) <
tom.dean@anonymised.com> wrote:

Windows Server 2008
Geoserver 2.3.5
Java 1.7.13 64bit

Hi all,

Every so often Geoserver stops responding for about 20 minutes at a time.
This generally happens every couple of days depending on usage. The errors
reported in the log are as follows:

2013-12-20 12:05:07,990 WARN [geotools.jdbc] - There is code leaving
feature readers/iterators open, this is leaking statements and connections!
2013-12-20 12:17:52,501 ERROR [geotools.map] - Call MapContent dispose()
to prevent memory leaks
2013-12-20 12:17:54,357 ERROR [geoserver.ows] -
java.lang.OutOfMemoryError: Java heap space

Geoserver eventually restarts itself and starts responding again.

I am going to assign some more memory but it already uses 1.5GB and I
suspect that may simply allow it to last longer before crashing? My
question is, is there anything else I can do? Is this fixed in later
versions? Does the data source type matter? We're using Oracle which I
know can be a memory hog for client connections.

Hmm... I can't think of any known memory leak in recent versions of
GeoServer.
The best thing to do is to make it go OOM, or very close to it, and then
use jmap -histo and report
back what is filling the memory.
A full memory dump, which can also be obtained with jmap, would also be
useful.

Both options are described here (not the first two usages, the 3rd and the
4th):
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html#jmap

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

3

Hi all,

Thanks all for your responses. I will get jmap running and see what that reports. We restart Apache nightly because it uses huge amounts of memory because of Oracle client connections and I wonder if this is similar. Either way Apache doesn’t actually crash (it just keeps growing its memory footprint) whereas Geoserver does and it sometimes does it more than once per day which suggests that restarting it nightly isn’t going to resolve the problem.

The Java startup params we’re using are:

Initial Java Heap Size (in MB)

wrapper.java.initmemory=512

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=4096

which I only recently upped. They were set to half that before. That to me seems like a lot of RAM to grant something that serves a very small number of requests (probably no more than a few thousand tiles per day)

Tom

···

On 20 December 2013 13:21, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Windows Server 2008

Geoserver 2.3.5

Java 1.7.13 64bit

Hi all,

Every so often Geoserver stops responding for about 20 minutes at a time. This generally happens every couple of days depending on usage. The errors reported in the log are as follows:

2013-12-20 12:05:07,990 WARN [geotools.jdbc] - There is code leaving feature readers/iterators open, this is leaking statements and connections!
2013-12-20 12:17:52,501 ERROR [geotools.map] - Call MapContent dispose() to prevent memory leaks
2013-12-20 12:17:54,357 ERROR [geoserver.ows] -
java.lang.OutOfMemoryError: Java heap space

Geoserver eventually restarts itself and starts responding again.

I am going to assign some more memory but it already uses 1.5GB and I suspect that may simply allow it to last longer before crashing? My question is, is there anything else I can do? Is this fixed in later versions? Does the data source type matter? We’re using Oracle which I know can be a memory hog for client connections.

Tom

Rapidly troubleshoot problems before they affect your business. Most IT
organizations don’t have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

Hi Tom,
For reference, my start up is:
-Xmx5G -Xms2G -XX:MaxPermSize=256m

(so 2Gb start, 5Gb max).

Despite that, across 9 instances, I’ve never seen any of the tomcat instances use above 1.5Gb; it’s typically 1Gb.
In a typical day an instance handles about 6500 requests, 5000 of which are TMS (only about 500 are new tiles). Plus about 1100 WMS/WFS requests, which almost all go to Oracle.

So you’ve definitely got anomalous going on compared to our systems anyway.

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 23 December 2013 13:55, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Hi all,

Thanks all for your responses. I will get jmap running and see what that reports. We restart Apache nightly because it uses huge amounts of memory because of Oracle client connections and I wonder if this is similar. Either way Apache doesn’t actually crash (it just keeps growing its memory footprint) whereas Geoserver does and it sometimes does it more than once per day which suggests that restarting it nightly isn’t going to resolve the problem.

The Java startup params we’re using are:

Initial Java Heap Size (in MB)

wrapper.java.initmemory=512

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=4096

which I only recently upped. They were set to half that before. That to me seems like a lot of RAM to grant something that serves a very small number of requests (probably no more than a few thousand tiles per day)

Tom

---------- Forwarded message ----------
From: Jonathan Moules <jonathanmoules@anonymised.com…>
Date: Fri, Dec 20, 2013 at 2:07 PM
Subject: Re: [Geoserver-users] Out of Memory errors
To: “Tom (JDi Solutions)” <tom.dean@anonymised.com>

Hi Tom,
We have a fairly similar setup (Windows server pointing at Oracle) but I’ve never seen that, though we do restart GeoServer nightly; might you want to try that?

Also, what are the JAVA parameters you’re using at startup (the ones that set the memory).

Oh, and if you increase the logging level to Geotools-Developers, what do the logs say then when this is evidenced?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Rapidly troubleshoot problems before they affect your business. Most IT
organizations don’t have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

On 20 December 2013 13:21, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Windows Server 2008

Geoserver 2.3.5

Java 1.7.13 64bit

Hi all,

Every so often Geoserver stops responding for about 20 minutes at a time. This generally happens every couple of days depending on usage. The errors reported in the log are as follows:

2013-12-20 12:05:07,990 WARN [geotools.jdbc] - There is code leaving feature readers/iterators open, this is leaking statements and connections!
2013-12-20 12:17:52,501 ERROR [geotools.map] - Call MapContent dispose() to prevent memory leaks
2013-12-20 12:17:54,357 ERROR [geoserver.ows] -
java.lang.OutOfMemoryError: Java heap space

Geoserver eventually restarts itself and starts responding again.

I am going to assign some more memory but it already uses 1.5GB and I suspect that may simply allow it to last longer before crashing? My question is, is there anything else I can do? Is this fixed in later versions? Does the data source type matter? We’re using Oracle which I know can be a memory hog for client connections.

Tom

Rapidly troubleshoot problems before they affect your business. Most IT
organizations don’t have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

5

On Mon, Dec 23, 2013 at 2:55 PM, Tom (JDi Solutions) <
tom.dean@anonymised.com> wrote:

Hi all,

Thanks all for your responses. I will get jmap running and see what that
reports. We restart Apache nightly because it uses huge amounts of memory
because of Oracle client connections and I wonder if this is similar.
Either way Apache doesn't actually crash (it just keeps growing its memory
footprint) whereas Geoserver does and it sometimes does it more than once
per day which suggests that restarting it nightly isn't going to resolve
the problem.

The Java startup params we're using are:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=512

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=4096

which I only recently upped. They were set to half that before. That to
me seems like a lot of RAM to grant something that serves a very small
number of requests (probably no more than a few thousand tiles per day)

And it is... normally we suggest not to go beyond 2GB of heap unless you
have the CPUs and bandwidth to serve
many concurrent requests, or have to serve very large output maps (for
printing purposes).

Cheers
Andrea

--
*== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
06/01/2014 ==*

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

6

Just looking to jmap this now and Geoserver is using over 4GB or RAM but jmap reports the following: “Unable to attach to 32-bit process running under WOW64” which suggests it’s a 64bit jmap looking at a 32bit java instance but Geoserver is running out of the same install as far as I can tell allbeit jmap is in the JDK and Geoserver appears to be running from the JVM. The only way I can think of getting round this is to restart Geoserver and force it to run in the JDK version but that would lose the 4GB memory footprint I was hoping to analyse. Anything I can do without having to restart?

Jonathan, one question: do all your Oracle connections use the same login? My understanding of this is limited and there are other possible explanations but a theory I have is that Oracle will not share a client connection between two different logins and it allocates all its RAM via the client program. If that’s correct and you use a single login that would explain why your memory usage is manageable and ours isn’t. We use a separate login for each of our clients to help ensure data separation and we’ve got 14 configured in there. I also suspect that Oracle memory configuration has more to do with how much memory the client application uses but I’ve never found anything that explains how this actually works particularly well!

···

On Mon, Dec 23, 2013 at 2:32 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 23, 2013 at 2:55 PM, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Hi all,

Thanks all for your responses. I will get jmap running and see what that reports. We restart Apache nightly because it uses huge amounts of memory because of Oracle client connections and I wonder if this is similar. Either way Apache doesn’t actually crash (it just keeps growing its memory footprint) whereas Geoserver does and it sometimes does it more than once per day which suggests that restarting it nightly isn’t going to resolve the problem.

The Java startup params we’re using are:

Initial Java Heap Size (in MB)

wrapper.java.initmemory=512

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=4096

which I only recently upped. They were set to half that before. That to me seems like a lot of RAM to grant something that serves a very small number of requests (probably no more than a few thousand tiles per day)

And it is… normally we suggest not to go beyond 2GB of heap unless you have the CPUs and bandwidth to serve
many concurrent requests, or have to serve very large output maps (for printing purposes).

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

7

Hi Tom,
Yes, we’re using one single username/password for it as we don’t currently require any data segregation. There are actually two Oracle stores used though - using different schemas (but again, same username/password).
I leave it to wiser minds than mine to figure out how anything interacts with Oracle. :slight_smile:

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 23 December 2013 15:13, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Just looking to jmap this now and Geoserver is using over 4GB or RAM but jmap reports the following: “Unable to attach to 32-bit process running under WOW64” which suggests it’s a 64bit jmap looking at a 32bit java instance but Geoserver is running out of the same install as far as I can tell allbeit jmap is in the JDK and Geoserver appears to be running from the JVM. The only way I can think of getting round this is to restart Geoserver and force it to run in the JDK version but that would lose the 4GB memory footprint I was hoping to analyse. Anything I can do without having to restart?

Jonathan, one question: do all your Oracle connections use the same login? My understanding of this is limited and there are other possible explanations but a theory I have is that Oracle will not share a client connection between two different logins and it allocates all its RAM via the client program. If that’s correct and you use a single login that would explain why your memory usage is manageable and ours isn’t. We use a separate login for each of our clients to help ensure data separation and we’ve got 14 configured in there. I also suspect that Oracle memory configuration has more to do with how much memory the client application uses but I’ve never found anything that explains how this actually works particularly well!

Rapidly troubleshoot problems before they affect your business. Most IT
organizations don’t have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

On Mon, Dec 23, 2013 at 2:32 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 23, 2013 at 2:55 PM, Tom (JDi Solutions) <tom.dean@anonymised.com> wrote:

Hi all,

Thanks all for your responses. I will get jmap running and see what that reports. We restart Apache nightly because it uses huge amounts of memory because of Oracle client connections and I wonder if this is similar. Either way Apache doesn’t actually crash (it just keeps growing its memory footprint) whereas Geoserver does and it sometimes does it more than once per day which suggests that restarting it nightly isn’t going to resolve the problem.

The Java startup params we’re using are:

Initial Java Heap Size (in MB)

wrapper.java.initmemory=512

Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=4096

which I only recently upped. They were set to half that before. That to me seems like a lot of RAM to grant something that serves a very small number of requests (probably no more than a few thousand tiles per day)

And it is… normally we suggest not to go beyond 2GB of heap unless you have the CPUs and bandwidth to serve
many concurrent requests, or have to serve very large output maps (for printing purposes).

Cheers

Andrea

–

== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to 06/01/2014 ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it