[Geoserver-users] Possible Memory Leak causing server to not server pages

Jesus_Gomez · June 23, 2014, 9:08pm

Hello.

Installed Geoserver 2.5.1 as a WAR on tomcat 7.0.28-4+deb7u1 (Debian wheezy).

The server runs normally but i notice the following on catalina.out when i stop the server::

Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread named [Thread-6] but has failed to stop it. This is very likely to create a memory leak.
Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/geoserver] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@anonymised.com]) and a value of type [org.geoserver.ows.Request] (value [WMS 1.1.1 GetMap]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This cause a real problem to me when i try to add the following to /etc/tomcat/web.xml ::

CorsFilter org.apache.catalina.filters.CorsFilter CorsFilter /*

Because the server simply doesn’t show anything to the client and the new thing is that the same errors about “memory leaks”::

Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread named [Thread-5] but has failed to stop it. This is very likely to create a memory leak.
Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread named [Thread-6] but has failed to stop it. This is very likely to create a memory leak.
Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/geoserver] created a ThreadLocal with key of type [org.eclipse.emf.ecore.impl.EClassImpl$1] (value [org.eclipse.emf.ecore.impl.EClassImpl$1@anonymised.com]) and a value of type [java.util.HashSet] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/geoserver] created a ThreadLocal with key of type [org.eclipse.emf.ecore.xml.type.util.XMLTypeUtil.CharArrayThreadLocal] (value [org.eclipse.emf.ecore.xml.type.util.XMLTypeUtil$CharArrayThreadLocal@anonymised.com]) and a value of type [char] (value [[C@anonymised.com]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This time i don’t have to wait for stoping the server to see the errors.

So, why is this happening?.

Jesus_Gomez · June 25, 2014, 1:40pm

Hello, I haven't resolved this yet, but for the record, here is some
information I have collected that could be of use.

2014-06-23 16:38 GMT-04:30 Jesús Gómez <jgomo3@anonymised.com>:

Hello.

Installed Geoserver 2.5.1 as a WAR on tomcat 7.0.28-4+deb7u1 (Debian
wheezy).

The server runs normally but i notice the following on catalina.out when i
stop the server::

Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread
named [Thread-6] but has failed to stop it. This is very likely to create a
memory leak.
Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/geoserver] created a ThreadLocal with key of
type [java.lang.InheritableThreadLocal] (value
[java.lang.InheritableThreadLocal@anonymised.com]) and a value of type
[org.geoserver.ows.Request] (value [WMS 1.1.1 GetMap]) but failed to remove
it when the web application was stopped. Threads are going to be renewed
over time to try and avoid a probable memory leak.

What we are seeing is a mechanism of Tomcat that we could name
MemoryLeakProtection[1].

The first thing I've learned is that it is normally and expected that this
precise information about memory leak will be printed out on the logs
_when_ the application _stops_: "Hopefully tomcat 6.0.24 can detect the
leak when the application is stopped" as the documents explains.

By reading in that documentation the examples of different kinds of
ThreadLocal leaks, we can see the logs are really similar to the ones I've
posted. You can see:

Mar 16, 2010 11:47:24 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[test.MyThreadLocal] (value [test.MyThreadLocal@anonymised.com]) and a value of
type [test.MyCounter] (value [test.MyCounter@anonymised.com]) but failed to
remove it when the web application was stopped. To prevent a memory leak,
the ThreadLocal has been forcibly removed.

So this is a big hint about Geoserver causing memory leaks.

This cause a real problem to me when i try to add the following to
/etc/tomcat/web.xml ::

<filter>
  <filter-name>CorsFilter</filter-name>
  <filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>CorsFilter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>

...

Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread
named [Thread-5] but has failed to stop it. This is very likely to create a
memory leak.
Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
...

This time i don't have to wait for stoping the server to see the errors.

Second, now I'm sure that the memory leak is causing Geoserver to stop. The
fact I can cause an inevitable stop by enabling the CorsFilter doesn't
change the fact that there is a memory leak problem in Geoserver.

Of course, i think I can live with that time bomb if I disable the
CorsFilter, because I could be monitoring the tomcat and use the "Find
Leaks" button in low traffic periods and bypass the need of CorsFilter by
using a Proxy.

But:

1) It is a time bomb. I would not like to live with the Damocles sword on
my head.
2) This could be a bug if I've understand right the situation.

So, why is this happening?.

So, by far I don't know why this is happening, but it would be of great
help if you point me to where to look, give me some hints about how to hunt
this bug or about the reasons behind this problem.

Tank you!

P.D: Since this is a potential BUG, I'm also sending a copy to the
developers list.

[1] http://wiki.apache.org/tomcat/MemoryLeakProtection

jive · June 26, 2014, 12:15am

Your logs do not really provide enough detail to determine which thread is “left open”. We do have a number of background threads responsible for clearing caches and so on in order to keep our memory footprint low at runtime. Perhaps it is one of these that has not been stopped?

One example is the EPSG referencing code runs a thread to manage memory use. The code is over engineered so that multiple web applications can share the same EPSG database and is probably not designed to removed when a single web application shuts down.

Rather then send a copy to the developers list please open an issue with our issue tracker!

···

Jody Garnett

On Wed, Jun 25, 2014 at 11:40 PM, Jesús Gómez <jgomo3@anonymised.com> wrote:

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hello, I haven’t resolved this yet, but for the record, here is some information I have collected that could be of use.

2014-06-23 16:38 GMT-04:30 Jesús Gómez <jgomo3@anonymised.com>:

Hello.

Installed Geoserver 2.5.1 as a WAR on tomcat 7.0.28-4+deb7u1 (Debian wheezy).

The server runs normally but i notice the following on catalina.out when i stop the server::

Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread named [Thread-6] but has failed to stop it. This is very likely to create a memory leak.
Jun 23, 2014 10:54:44 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/geoserver] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@anonymised.com]) and a value of type [org.geoserver.ows.Request] (value [WMS 1.1.1 GetMap]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

What we are seeing is a mechanism of Tomcat that we could name MemoryLeakProtection[1].

The first thing I’ve learned is that it is normally and expected that this precise information about memory leak will be printed out on the logs when the application stops: “Hopefully tomcat 6.0.24 can detect the leak when the application is stopped” as the documents explains.

By reading in that documentation the examples of different kinds of ThreadLocal leaks, we can see the logs are really similar to the ones I’ve posted. You can see:

Mar 16, 2010 11:47:24 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [test.MyThreadLocal] (value [test.MyThreadLocal@anonymised.com]) and a value of type [test.MyCounter] (value [test.MyCounter@anonymised.com]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.

So this is a big hint about Geoserver causing memory leaks.

This cause a real problem to me when i try to add the following to /etc/tomcat/web.xml ::
CorsFilter org.apache.catalina.filters.CorsFilter CorsFilter /*
…

Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/geoserver] appears to have started a thread named [Thread-5] but has failed to stop it. This is very likely to create a memory leak.
Jun 23, 2014 11:04:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

…

This time i don’t have to wait for stoping the server to see the errors.

Second, now I’m sure that the memory leak is causing Geoserver to stop. The fact I can cause an inevitable stop by enabling the CorsFilter doesn’t change the fact that there is a memory leak problem in Geoserver.

Of course, i think I can live with that time bomb if I disable the CorsFilter, because I could be monitoring the tomcat and use the “Find Leaks” button in low traffic periods and bypass the need of CorsFilter by using a Proxy.

But:

It is a time bomb. I would not like to live with the Damocles sword on my head.
This could be a bug if I’ve understand right the situation.

So, why is this happening?.

So, by far I don’t know why this is happening, but it would be of great help if you point me to where to look, give me some hints about how to hunt this bug or about the reasons behind this problem.

Tank you!

P.D: Since this is a potential BUG, I’m also sending a copy to the developers list.

[1] http://wiki.apache.org/tomcat/MemoryLeakProtection