Hello,
We have a geoserver 2.2 running in production environment on linux running openjdk 1.6.
After a while the geoserver stops functionning due to memory exhaustion, this comes from the
fact that more than 10.000 threads are in the following type of status. Which causes the geoserver
and other tomcat applications to hang.
Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
"Thread-24008" daemon prio=10 tid=0x0000000005713000 nid=0x3258 waiting on condition [0x00007f321c491000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.geoserver.security.file.FileWatcher.run(FileWatcher.java:122)
The functions that are enabled are wfs/wms and some complex SLD defintions. This happens after a while without using
specific functionalities.
Does someone have an idea how this issue can be solved/tackeled.
Best regards,
Eric Smets
On Fri, Feb 22, 2013 at 6:02 PM, Eric Smets <eric.smets@anonymised.com> wrote:
After a while the geoserver stops functionning due to memory exhaustion, this
comes from the
fact that more than 10.000 threads are in the following type of status. Which
causes the geoserver
and other tomcat applications to hang.
Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):
“Thread-24008” daemon prio=10 tid=0x0000000005713000 nid=0x3258 waiting on condition [0x00007f321c491000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.geoserver.security.file.FileWatcher.run(FileWatcher.java:122)
Hmm… this is part of the new security subsystem.
It seems they get started multiple times, and the old watchers are not cleaned up.
I’m not sure what might cause this, but a reasonable candidate could be configuration
reloads, are you issuing those commands frequently?
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
On Sat, Feb 23, 2013 at 12:31 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:
I’m not sure what might cause this, but a reasonable candidate could be configuration
reloads, are you issuing those commands frequently?
Hum… I’ve tried to reload/clear the configuration, and edit users and roles, but
the threads watching the files keep on being just 2.
Not sure what can cause them to become thousands…
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
Interesting, looking at the code in GeoServerSecurityManager I found the following snippet
/*
- list of file watchers
- TODO: we should probably manage these better rather than just throwing them in a
- list, repeated loads will cause this list to fill up with threads
*/
protected List fileWatchers = new ArrayList();
I think it would be better to register the watchers in a Map using the canonical file name as a key. Upon reload, an existing watcher should be terminated.
Andrea, I have no idea why you have always 2 threads but looking on the Java code, this issue should be fixed.
Should I open a JIRA issue
Christian
···
2013/2/23 Andrea Aime <andrea.aime@anonymised.com.>
On Sat, Feb 23, 2013 at 12:31 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:
I’m not sure what might cause this, but a reasonable candidate could be configuration
reloads, are you issuing those commands frequently?
Hum… I’ve tried to reload/clear the configuration, and edit users and roles, but
the threads watching the files keep on being just 2.
Not sure what can cause them to become thousands…
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
On Sat, Feb 23, 2013 at 4:48 PM, Christian Mueller <christian.mueller@anonymised.com> wrote:
Interesting, looking at the code in GeoServerSecurityManager I found the following snippet
/*
- list of file watchers
- TODO: we should probably manage these better rather than just throwing them in a
- list, repeated loads will cause this list to fill up with threads
*/
protected List fileWatchers = new ArrayList();
I think it would be better to register the watchers in a Map using the canonical file name as a key. Upon reload, an existing watcher should be terminated.
Right, except that “reload” does not actually go through that code. It seems it’s triggered only by a application context event of spring context reload,
which at runtime should never happen.
Andrea, I have no idea why you have always 2 threads but looking on the Java code, this issue should be fixed.
Right, but before fixing it, it would be also know what’s causing it.
If we lack understanding of how things work, we might just end up making a change that introduces another bug.
Should I open a JIRA issue
I think you should.
Cheers
Andrea
–
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it