[Geoserver-users] Out Of Memory Error / Memory Leak

All,

Wondering if anyone can point us in the right direction to solve this java/memory issue. Yesterday afternoon our Tomcat went down, wasn’t picked up until this morning. Stop/started Tomcat and appeared to work again. Looking at the logs appeared to be a “java.lang.OutOfMemoryError: PermGen space” problem. We have spare memory on the server so we upped the JVM container settings (-Xmx & -XX:MaxPermSize), we then restarted Tomcat again.

Looking at the logs in more detail we found a few things that seemed worrying such as:

· IOException while loading persisted sessions: java.io.EOFException

· The web application [/geoserver] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.

· CSSEngine: exception property.syntax.error:org.w3c.css.sac.CSSParseException: Unexpected token: 0 (see LexicalUnits)

System details

· GeoServer: 2.3.3

· Apache Tomcat: 7.0.41

· JVM Version: 1.6.0_45-b06

· Windows Server 2008 R2; 64-bit; 4GB RAM; virtual

I’ve attached some logs (I’ve edited some for brevity); the 2014-05-15 logs relate to the initial OutOfMemoryError and the 2014-05-16 one includes info on other errors listed above on restarts. Brief timeline to help navigate the logs:

· Initial problems: 2014-05-15 15:45

· Stopped Tomcat: 2014-05-16 07:44

· Started Tomcat: 2014-05-16 07:50

· Stopped Tomcat: 2014-05-16 08:07

· Increase JVM memory options

· Started Tomcat: 2014-05-16 08:08

System has been working great for some time, recently has started to be accessed more, but no major changes that we’re aware of. Any pointers greatly appreciated.

Many thanks,

Rob



<br>DISCLAIMER: The information contained in this communication/message from Robert.Langford@anonymised.com..5490... sent on Fri May 16 09:19:08 2014 is confidential. It is intended solely for the addressee(s) geoserver-users@lists.sourceforge.net<br><br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act.<br>Please immediately contact the sender, Robert.Langford@anonymised.com if you have received this message in error.<br><br>For the full disclaimer please access http://www.salford.gov.uk/e-mail. Thank you.<br>

geoserver.log (16 KB)

tomcat7-stderr.2014-05-15.log (41.9 KB)

tomcat7-stdout.2014-05-15.log (1.74 KB)

hs_err_pid3628.log (16.5 KB)

tomcat7-stderr.2014-05-16.log (78.2 KB)

Hi Rob,
How much memory are you giving to your instance and what was it before?

My own 2.5-snapshot and 2.4.3 instances both show this line in a few places (and it’s in my 2.3.2 logs too):

2014-05-12 14:26:05,020 ERROR [geotools.map] - Call MapContent dispose() to prevent memory leaks

Any indications as to anything else with the word “memory” in the logs anywhere? (just guessing as to it being possibly culpable; I’m not a dev so don’t know).

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 16 May 2014 09:19, Langford, Robert <Robert.Langford@anonymised.com> wrote:

All,

Wondering if anyone can point us in the right direction to solve this java/memory issue. Yesterday afternoon our Tomcat went down, wasn’t picked up until this morning. Stop/started Tomcat and appeared to work again. Looking at the logs appeared to be a “java.lang.OutOfMemoryError: PermGen space” problem. We have spare memory on the server so we upped the JVM container settings (-Xmx & -XX:MaxPermSize), we then restarted Tomcat again.

Looking at the logs in more detail we found a few things that seemed worrying such as:

· IOException while loading persisted sessions: java.io.EOFException

· The web application [/geoserver] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.

· CSSEngine: exception property.syntax.error:org.w3c.css.sac.CSSParseException: Unexpected token: 0 (see LexicalUnits)

System details

· GeoServer: 2.3.3

· Apache Tomcat: 7.0.41

· JVM Version: 1.6.0_45-b06

· Windows Server 2008 R2; 64-bit; 4GB RAM; virtual

I’ve attached some logs (I’ve edited some for brevity); the 2014-05-15 logs relate to the initial OutOfMemoryError and the 2014-05-16 one includes info on other errors listed above on restarts. Brief timeline to help navigate the logs:

· Initial problems: 2014-05-15 15:45

· Stopped Tomcat: 2014-05-16 07:44

· Started Tomcat: 2014-05-16 07:50

· Stopped Tomcat: 2014-05-16 08:07

· Increase JVM memory options

· Started Tomcat: 2014-05-16 08:08

System has been working great for some time, recently has started to be accessed more, but no major changes that we’re aware of. Any pointers greatly appreciated.

Many thanks,

Rob



<br>DISCLAIMER: The information contained in this communication/message from [Robert.Langford@anonymised.com..5490...](mailto:Robert.Langford@anonymised.com) sent on Fri May 16 09:19:08 2014 is confidential. It is intended solely for the addressee(s) [geoserver-users@lists.sourceforge.net](mailto:geoserver-users@anonymised.comrceforge.net)<br><br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act. <br>Please immediately contact the sender, [Robert.Langford@anonymised.com](mailto:Robert.Langford@anonymised.com.5490...) if you have received this message in error. <br><br>For the full disclaimer please access [http://www.salford.gov.uk/e-mail](http://www.salford.gov.uk/e-mail). Thank you.<br>


“Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.”
http://p.sf.net/sfu/SauceLabs


Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Sorry should have included this before:

JVM original options:

· -Xmx1024M

· -Xms48

· -XX:SoftRefLRUPolicyMSPerMB=36000

· -XX:MaxPermSize=128m

· -XX:+UseParallelGC

JVM new options:

· -Xmx2048M

· -Xms48

· -XX:SoftRefLRUPolicyMSPerMB=36000

· -XX:MaxPermSize=256m

· -XX:+UseParallelGC

Jonathan – not sure on the logs, always struggle to get to grips with them. The tomcat7-stderr.2014-05-16.log contains a few “[…]This is very likely to create a memory leak.” entries even after the memory bump.

Rob

From: Jonathan Moules [mailto:jonathanmoules@anonymised.com]
Sent: 16 May 2014 09:40
To: Langford, Robert
Cc: geoserver-users@lists.sourceforge.net
Subject: Re: [Geoserver-users] Out Of Memory Error / Memory Leak

Hi Rob,

How much memory are you giving to your instance and what was it before?

My own 2.5-snapshot and 2.4.3 instances both show this line in a few places (and it’s in my 2.3.2 logs too):

2014-05-12 14:26:05,020 ERROR [geotools.map] - Call MapContent dispose() to prevent memory leaks

Any indications as to anything else with the word “memory” in the logs anywhere? (just guessing as to it being possibly culpable; I’m not a dev so don’t know).

Cheers,

Jonathan

On 16 May 2014 09:19, Langford, Robert <Robert.Langford@anonymised.com90…> wrote:

All,

Wondering if anyone can point us in the right direction to solve this java/memory issue. Yesterday afternoon our Tomcat went down, wasn’t picked up until this morning. Stop/started Tomcat and appeared to work again. Looking at the logs appeared to be a “java.lang.OutOfMemoryError: PermGen space” problem. We have spare memory on the server so we upped the JVM container settings (-Xmx & -XX:MaxPermSize), we then restarted Tomcat again.

Looking at the logs in more detail we found a few things that seemed worrying such as:

· IOException while loading persisted sessions: java.io.EOFException

· The web application [/geoserver] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.

· CSSEngine: exception property.syntax.error:org.w3c.css.sac.CSSParseException: Unexpected token: 0 (see LexicalUnits)

System details

· GeoServer: 2.3.3

· Apache Tomcat: 7.0.41

· JVM Version: 1.6.0_45-b06

· Windows Server 2008 R2; 64-bit; 4GB RAM; virtual

I’ve attached some logs (I’ve edited some for brevity); the 2014-05-15 logs relate to the initial OutOfMemoryError and the 2014-05-16 one includes info on other errors listed above on restarts. Brief timeline to help navigate the logs:

· Initial problems: 2014-05-15 15:45

· Stopped Tomcat: 2014-05-16 07:44

· Started Tomcat: 2014-05-16 07:50

· Stopped Tomcat: 2014-05-16 08:07

· Increase JVM memory options

· Started Tomcat: 2014-05-16 08:08

System has been working great for some time, recently has started to be accessed more, but no major changes that we’re aware of. Any pointers greatly appreciated.

Many thanks,

Rob



<br>DISCLAIMER: The information contained in this communication/message from [Robert.Langford@anonymised.com](mailto:Robert.Langford@anonymised.com) sent on Fri May 16 09:19:08 2014 is confidential. It is intended solely for the addressee(s) [geoserver-users@lists.sourceforge.net](mailto:geoserver-users@anonymised.comet)<br>



<br><br>



<br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>



<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act. <br>



<br>Please immediately contact the sender, [Robert.Langford@anonymised.com](mailto:Robert.Langford@anonymised.com) if you have received this message in error. <br>



<br><br>



<br>For the full disclaimer please access [http://www.salford.gov.uk/e-mail](http://www.salford.gov.uk/e-mail). Thank you.<br>


“Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.”
http://p.sf.net/sfu/SauceLabs


Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.



<br>DISCLAIMER: The information contained in this communication/message from Robert.Langford@anonymised.com..5490... sent on Fri May 16 09:51:06 2014 is confidential. It is intended solely for the addressee(s) geoserver-users@lists.sourceforge.net;jonathanmoules@anonymised.com<br><br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act.<br>Please immediately contact the sender, Robert.Langford@anonymised.com if you have received this message in error.<br><br>For the full disclaimer please access http://www.salford.gov.uk/e-mail. Thank you.<br>

On Fri, May 16, 2014 at 10:51 AM, Langford, Robert <
Robert.Langford@anonymised.com> wrote:

Jonathan – not sure on the logs, always struggle to get to grips with
them. The tomcat7-stderr.2014-05-16.log contains a few “[…]This is very
likely to create a memory leak.” entries even after the memory bump.

Unfortunately the person that wrote that warn message was thinking about
desktop applications with listeners
on the map, GeoServer is as far as I know unaffected by that "memory leak"
threat.
Jody, care to comment? (afaik you were the one to add that log message)

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Eric / Andrea / Jonathan – Thanks for your inputs and apologies for the delay in coming back to you all.

Everything now seems to be running ok, guess we’ll just keep an eye on it. Next probable action will be to upgrade Geoserver to at least the 2.4.x version as mentioned by Eric.

In the mean time could anyone comment if our new JVM options look reasonable (I’m aware they’re really dependant on our data and demand, but anything we’ve overlooked or that doesn’t add up)?:

· -Xmx2048M

· -Xms48

· -XX:SoftRefLRUPolicyMSPerMB=36000

· -XX:MaxPermSize=256m

· -XX:+UseParallelGC

Thanks all,

Rob

From: Eric Smets [mailto:eric.smets@anonymised.com]
Sent: 16 May 2014 10:44
To: Langford, Robert
Subject: Re: [Geoserver-users] Out Of Memory Error / Memory Leak

Hi,

A major memory leak has been solved in the geoserver 2.4 version.
Since we installed this version no issues have been seen.

Regards,

Eric Smets

Op 16/05/2014 10:19, Langford, Robert schreef:

All,

Wondering if anyone can point us in the right direction to solve this java/memory issue. Yesterday afternoon our Tomcat went down, wasn’t picked up until this morning. Stop/started Tomcat and appeared to work again. Looking at the logs appeared to be a “java.lang.OutOfMemoryError: PermGen space” problem. We have spare memory on the server so we upped the JVM container settings (-Xmx & -XX:MaxPermSize), we then restarted Tomcat again.

Looking at the logs in more detail we found a few things that seemed worrying such as:

· IOException while loading persisted sessions: java.io.EOFException

· The web application [/geoserver] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.

· CSSEngine: exception property.syntax.error:org.w3c.css.sac.CSSParseException: Unexpected token: 0 (see LexicalUnits)

System details

· GeoServer: 2.3.3

· Apache Tomcat: 7.0.41

· JVM Version: 1.6.0_45-b06

· Windows Server 2008 R2; 64-bit; 4GB RAM; virtual

I’ve attached some logs (I’ve edited some for brevity); the 2014-05-15 logs relate to the initial OutOfMemoryError and the 2014-05-16 one includes info on other errors listed above on restarts. Brief timeline to help navigate the logs:

· Initial problems: 2014-05-15 15:45

· Stopped Tomcat: 2014-05-16 07:44

· Started Tomcat: 2014-05-16 07:50

· Stopped Tomcat: 2014-05-16 08:07

· Increase JVM memory options

· Started Tomcat: 2014-05-16 08:08

System has been working great for some time, recently has started to be accessed more, but no major changes that we’re aware of. Any pointers greatly appreciated.

Many thanks,

Rob



<br>DISCLAIMER: The information contained in this communication/message from [Robert.Langford@anonymised.com](mailto:Robert.Langford@anonymised.com490...) sent on Fri May 16 09:19:08 2014 is confidential. It is intended solely for the addressee(s) [geoserver-users@anonymised.comt](mailto:geoserver-users@lists.sourceforge.net)<br>



<br><br>



<br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>



<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act. <br>



<br>Please immediately contact the sender, [Robert.Langford@anonymised.com](mailto:Robert.Langford@anonymised.com) if you have received this message in error. <br>



<br><br>



<br>For the full disclaimer please access [http://www.salford.gov.uk/e-mail](http://www.salford.gov.uk/e-mail). Thank you.<br>

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
[http://p.sf.net/sfu/SauceLabs](http://p.sf.net/sfu/SauceLabs)
_______________________________________________
Geoserver-users mailing list
[Geoserver-users@lists.sourceforge.net](mailto:Geoserver-users@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-users](https://lists.sourceforge.net/lists/listinfo/geoserver-users)
-- 
Eric Smets                                    [eric.smets@anonymised.com](mailto:eric.smets@anonymised.com)
FKS bvba - Formal and Knowledge Systems       [http://www.fks.be/](http://www.fks.be/)
Schampbergstraat 32                           Tel:  ++32-(0)11-21 49 11
B-3511 Hasselt                                Fax:  ++32-(0)11-22 04 19


<br>DISCLAIMER: The information contained in this communication/message from Robert.Langford@anonymised.com..5490... sent on Fri May 23 12:13:52 2014 is confidential. It is intended solely for the addressee(s) eric.smets@anonymised.com;andrea.aime@anonymised.com;geoserver-users@lists.sourceforge.net;jonathanmoules@anonymised.com<br><br>Access to this message by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful.<br>As a public body, Salford City Council may be required to disclose this email [or any response to it] under the Freedom of Information Act 2000, unless the information in it is covered by one of the exemptions in the Act.<br>Please immediately contact the sender, Robert.Langford@anonymised.com if you have received this message in error.<br><br>For the full disclaimer please access http://www.salford.gov.uk/e-mail. Thank you.<br>