Hi Justin,
Ok. For us, our patch is sufficient to get things going without having
to periodically restart tomcat. We'll use that version until the
official fix makes it into 2.1 or such.
If someone is interested, we just copied
main.java.org.vfny.geoserver.global.GeoserverDataDirectory.findGeoServerDataDir()
to
org.geoserver.catalog.ResourcePool
and then changed it a bit so the size of the datastore cache can be set
via a system or Java environment variable named "DATASTORE_CACHE_SIZE".
The code we've added to ResourcePool.java is shown below. Nothing fancy.
static int DATASTORE_CACHE_SIZE_DEFAULT = 500;
public ResourcePool(Catalog catalog) {
...
dataStoreCache = new DataStoreCache(getDataStoreCacheSize());
...
}
public static int getDataStoreCacheSize() {
return getEnvVariable("DATASTORE_CACHE_SIZE"
, DATASTORE_CACHE_SIZE_DEFAULT);
}
public static int getEnvVariable(String varName, int defaultSize) {
final String typeStrs = { "Java environment variable "
, "Servlet context parameter "
, "System environment variable " };
int intVal = -1;
// Loop over variable access methods
for (int j = 0; j < typeStrs.length; j++) {
String value = null;
String typeStr = typeStrs[j];
// Lookup section
switch (j) {
case 0:
value = System.getProperty(varName);
break;
// case 1:
// value = servContext.getInitParameter(varName);
// break;
case 2:
value = System.getenv(varName);
break;
}
if (value == null || value.equalsIgnoreCase("")) {
LOGGER.finer("Found " + typeStr + varName + " to be unset");
continue;
}
// Verify section
try {
intVal = Integer.parseInt(value);
}
catch(Exception e) {
LOGGER.fine("Invalid value for setting " + varName + ": "
+ value + " is not an integer");
continue;
}
}
if (intVal < 0) {
intVal = defaultSize;
}
return intVal;
}
class DataStoreCache extends LRUMap {
public DataStoreCache() {
this(DATASTORE_CACHE_SIZE_DEFAULT);
}
public DataStoreCache(int maxSize) {
super(maxSize);
}
...
}
Greetings, Egon
-----Original Message-----
From: Justin Deoliveira <jdeolive@anonymised.com>
To: geoserver-devel@lists.sourceforge.net
Date: Wed, 21 Jul 2010 17:12:39 -0600
Subject: Re: [Geoserver-devel] memory leak in 2.0.2 and 2.0.x nightly build
Ok, it is good to confirm that is the issue.
It actually is not the datastores not disposing properly that is the
problem. It is that every time a feature type or a datastore is
diposed,
an event latches on to it that rebuilds the internal object we use to
model the wfs schema. When that happens memory leaks.
The fix to prevent leaking in this way is unfortunately not trivial. I
hope to implement it soon but it may be something that will have to
wait
for 2.1.
-Justin
On 10-07-21 4:49 PM, egona wrote:
> Hi Justin,
>
> We have run some tests with different-sized datastore caches. In
short,
> we can confirm that the size of the cache plays a role: to prevent
the
> memory leak, it should be equal to or greater than the number of
> datastores.
>
> In our test case, with 191 datastores, everything works well when the
> cache size is set to: 1200, 800, and 400. The heap at the end of
those
> tests was 120 MB. When set to 150, however, the heap was stuck at
1000 MB.
>
> We haven't investigated the exact cause, but we suspect that the
datastore
> (org.geotools.jdbc.JDBCDataStore) that is evicted from the cache
doesn't
> dispose itself properly.
>
>
> Greetings, Egon
>
>
>
>
> -----Original Message-----
> From: Justin Deoliveira<jdeolive@anonymised.com>
> To: egona<egon.amade@anonymised.com>
> Cc: Robert van Seeters<robgis@anonymised.com>,
geoserver-devel@lists.sourceforge.net
> Date: Tue, 20 Jul 2010 11:15:22 -0600
> Subject: Re: [Geoserver-devel] memory leak in 2.0.2 and 2.0.x nightly
build
>
>> Hi Egon,
>>
>> So yeah, this makes me think it is indeed an issue with the size of
the
>> datastore cache, which is by default 100 entries big.
>>
>> So what we need to do is make this configurable the same way the
>> feature
>> type cache size is. I have created an issue in the tracker.
>>
>> http://jira.codehaus.org/browse/GEOS-4068
>>
>> -Justin
>>
>> On 10-07-20 7:17 AM, egona wrote:
>>> Hi Justin,
>>>
>>> We conducted 11 runs of the same test. The amount of heap used at
the
>>> end of each test, after requesting a GC, is shown below:
>>>
>>> 1. 100 workspaces/datastores, 562 layers: 86 MB
>>> 2. 101 workspaces/datastores, 563 layers: 68 MB
>>> 3. 101 workspaces/datastores, 605 layers: 70 MB
>>> 4. 101 workspaces/datastores, 567 layers: 69 MB
>>> 5. 101 workspaces/datastores, 565 layers: 69 MB
>>> 6. 101 workspaces/datastores, 565 layers: 69 MB
>>> 7. 105 workspaces/datastores, 617 layers: 103 MB
>>> 8. 110 workspaces/datastores, 666 layers: 110 MB
>>> 9. 120 workspaces/datastores, 756 layers: 250 MB
>>> 10 140 workspaces/datastores, 867 layers: 687 MB
>>> 11 191 workspaces/datastores, 1114 layers: 1414 MB
>>>
>>> The only difference between the tests is the number of workspaces,
>> datastores
>>> and layers that are registered in geoserver. The test only affects
23
>>> workspaces.
>>>
>>> Each workspace has its own datastore. A datastore belongs to one
>> workspace
>>> (a datastore is not shared among multiple workspaces). All
datastores
>> are
>>> attached to the same JNDI data source.
>>>
>>> We observed the following during these tests:
>>>
>>> o The JDBC errors (DataStore.dispose() is not explicitly called)
only
>>> seem to appear when the number of datastores is 100+.
>>>
>>> o The following do not seem to play a role in generating the memory
>> leak:
>>> o quality of the spatial data
>>> o size of the map images
>>> o feature type cache size (when set larger than the number of
>> features)
>>>
>>> o Clicking on the following in the geoserver admin does not reduce
>> the
>>> leaked memory:
>>>
>>> Free locks
>>> Free memory
>>> Free JAI memory
>>> Clear resource cache
>>> Reload configuration and catalog
>>>
>>>
>>> So, we think it is linked to number of workspaces, datastores, or
>> layers.
>>> It is maybe interesting to note that no memory leaks were found
when
>>> the 191 workspaces were spread accross two geoservers (running in
the
>>> same tomcat): one with 100 workspaces and one with 91 workspaces.
>>>
>>> Unfortunately, non-disclosure agreements prohibit us to supply a
heap
>>> dump 
>>>
>>> We have now setup a development environment for geoserver 2.0.x to
>> study
>>> and tinker the code. Do you have any tips for us? I am thinking
about
>>> adding debug info to the caches in
>> org.geoserver.catalog.ResourcePool.
>>>
>>>
>>> Greetings,
>>>
>>> Egon
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Justin Deoliveira<jdeolive@anonymised.com>
>>> To: egona<egon.amade@anonymised.com>
>>> Cc: Robert van Seeters<robgis@anonymised.com>,
>> geoserver-devel@lists.sourceforge.net
>>> Date: Mon, 19 Jul 2010 09:22:03 -0600
>>> Subject: Re: [Geoserver-devel] memory leak in 2.0.2 and 2.0.x
nightly
>> build
>>>
>>>> How many datastores are in play? Unfortunately the size of the
>>>> workspace
>>>> cache (which is currently unsettable :() leads to issues as well.
>>>>
>>>> Any chance you can generate a heap dump for me to look at? Just
test
>>>> the
>>>> server to memory exhaustion and then run:
>>>>
>>>> jmap -dump:live,file=<...> <pid>
>>>>
>>>> Or alternatively just set the flag for the server to generate a
heap
>>>> dump on oom. Also if you could limit the size of the heap as to
>>>> generate
>>>> a reasonable sized file that would be helpful as well.
>>>>
>>>> Thanks. Hopefully we can nail down this issue for you.
>>>>
>>>>
>>>> On 10-07-17 4:38 AM, egona wrote:
>>>>> Hi Justin,
>>>>>
>>>>> A little more info.
>>>>>
>>>>> Heap after forcing/requesting a GC (via VisualVM) at the end of
>> each
>>>> test:
>>>>>
>>>>> Test 1: 100 workspaces, 562 layers: 86 MB
>>>>> Test 2: 120 workspaces, 756 layers: 250 MB (jdbc errors)
>>>>> Test 3: 140 workspaces, 867 layers: 687 MB (jdbc errors)
>>>>> Test 4: 191 workspaces, 1114 layers: 1414 MB (jdbc errors)
>>>>>
>>>>> Note that the test (WMS/WFS requests) only affects 23 workspaces.
>>>> These
>>>>> are always the same workspaces. The only difference between the
>> tests
>>>> is
>>>>> the number of workspaces/layers registered in geoserver.
>>>>>
>>>>> For all tests, the feature type cache size is set to 4000. We
will
>>>> run
>>>>> some more tests where we adjust/lower that size. Hopefully, you
can
>>>>> find what's going wrong here.
>>>>>
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Egon
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Robert van Seeters<robgis@anonymised.com>
>>>>> To: Justin Deoliveira<jdeolive@anonymised.com>
>>>>> Cc: geoserver-devel@lists.sourceforge.net
>>>>> Date: Sat, 17 Jul 2010 11:28:38 +0200
>>>>> Subject: Re: [Geoserver-devel] memory leak in 2.0.2 and 2.0.x
>> nightly
>>>> build
>>>>>
>>>>>> hi Justin,
>>>>>> we already did that before executing the tests. We set the
feature
>>>>>> type cache size to 4000.
>>>>>> Too big maybe?
>>>>>>
>>>>>> ciao, Robert (colleague of Egon)
>>>>>>
>>>>>>
>>>>>> 2010/7/17 Justin Deoliveira<jdeolive@anonymised.com>:
>>>>>>> Ok, that number of feature types will bring about yet another
>>>> problem
>>>>>>> that will cause a memory leak, not related to the DFT one.
>>>>>>>
>>>>>>> The only way to get around it is to make sure the "feature type
>>>> cache
>>>>>>> size" (on the global settings page) setting is greater than the
>>>>>> number
>>>>>>> of feature types in the catalog.
>>>>>>>
>>>>>>> Try increasing them to 563 and 1115 respectively and let me
know
>> if
>>>>>> that
>>>>>>> helps.
>>>>>>>
>>>>>>> On 10-07-16 4:57 PM, egona wrote:
>>>>>>>> Hi Justin,
>>>>>>>>
>>>>>>>> Today we ran a test using two setups:
>>>>>>>>
>>>>>>>> Setup 1: 100 workspaces, 100 datastores, 562 layers
>>>>>>>> Setup 2: 191 workspaces, 191 datastores, 1114 layers
>>>>>>>>
>>>>>>>> In both cases, the same batch of WMS and WFS requests were
sent
>> to
>>>>>> geoserver.
>>>>>>>>
>>>>>>>> At the end of the test, the heap of setup 1 was at 82 MB,
>> whereas
>>>>>> the
>>>>>>>> heap of setup 2 was at 1440 MB. Another difference is that the
>>>>>> geoserver
>>>>>>>> log of setup 2 contains a lot of the following messages:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010-07-16 19:29:51,927 ERROR [geotools.jdbc] - There's code
>> using
>>>>>> JDBC based datastore and not
>>>>>>>> disposing them. This may lead to temporary loss of database
>>>>>> connections. Please make sure all
>>>>>>>> data access code calls DataStore.dispose() before freeing all
>>>>>> references to it
>>>>>>>>
>>>>>>>>
>>>>>>>> Have you got any clue if this is caused by bad spatial data, a
>>>>>> configuration
>>>>>>>> error of one of the layers, or perhaps a limit by geoserver?
>>>>>>>>
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> Egon
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Justin Deoliveira<jdeolive@anonymised.com>
>>>>>>>> To: geoserver-devel@lists.sourceforge.net
>>>>>>>> Date: Thu, 15 Jul 2010 10:32:41 -0600
>>>>>>>> Subject: Re: [Geoserver-devel] memory leak in 2.0.2 and 2.0.x
>>>>>> nightly build
>>>>>>>>
>>>>>>>>> Hi Egon,
>>>>>>>>>
>>>>>>>>> This is a known issue that is ongoing right now. It is
>> definitely
>>>>>>>>> related to GEOS-3534.
>>>>>>>>>
>>>>>>>>> What would help is to know about the request pattern you are
>>>>>> executing
>>>>>>>>> that shows the leak. The leak is known to occur during
>>>>>>>>> DescribeFeatureType and Transaction requests. Any info about
>> how
>>>>>> your
>>>>>>>>> data is set up and what sort of requests you are making will
>>>> help.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> -Justin
>>>>>>>>>
>>>>>>>>> On 10-07-15 10:25 AM, egona wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> This is my first post. Don't know if this should have been
>>>> posted
>>>>>> in
>>>>>>>>> the
>>>>>>>>>> user forum.
>>>>>>>>>>
>>>>>>>>>> We are struggling with a major memory leak.
>>>>>>>>>>
>>>>>>>>>> Our platform:
>>>>>>>>>> o Geoserver 2.0.2, 821 layers, 151 workspaces/datastores
>>>>>>>>>> o Win2003 32-bit 4 GB, Java 1.6.0_21, Tomcat 6.0.26, JVM
>>>>>>>>> minmx=memmx=1420 MB
>>>>>>>>>>
>>>>>>>>>> We are only using WMS and WFS.
>>>>>>>>>>
>>>>>>>>>> The JVM runs out of heap space very fast, forcing us to
reboot
>>>>>> Tomcat
>>>>>>>>>> every 2 hours.
>>>>>>>>>>
>>>>>>>>>> VisualVM reports that the top consumers (933 MB in total)
are:
>>>>>>>>>>
>>>>>>>>>> java.lang.String: 251 MB
>>>>>>>>>> char: 235 MB
>>>>>>>>>> org.apache.xerces.dom.AttrNSImpl: 217 MB
>>>>>>>>>> java.util.HashMap$Entry 92 MB
>>>>>>>>>> org.apache.xerces.dom.TextImpl: 70 MB
>>>>>>>>>> org.eclipse.xsd.impl.XSDElementDeclarationImpl: 68 MB
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The AttrNSImpl looks suspicious. Geoserver 2.0.2 uses Xerces
>>>>>> 2.6.2.
>>>>>>>>> The
>>>>>>>>>> current version is Xerces 2.10.0 (released June 19 2010).
>>>>>> Presumably,
>>>>>>>>>> it contains a fix for an AttrNSImpl related leak (see
>>>>>>>>>>
http://web.archiveorange.com/archive/v/4o5nETE1HwzVF6W9n09z),
>>>>>> amongst
>>>>>>>>>> many other fixes (including memory leaks). However, the
above
>>>>>> numbers
>>>>>>>>>> were generated with geoserver 2.0.2 using xerces 2.10.0 (we
>>>>>> bluntly
>>>>>>>>>> replaced the Xerces jar).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is anyone familiar with this behavior? Is Xerces the culprit
>> or
>>>>>> are
>>>>>>>>> other
>>>>>>>>>> objects unintentionally holding references to Xerces
objects?
>>>>>>>>>>
>>>>>>>>>> Is this related to http://jira.codehaus.org/browse/GEOS-3534
>> ??
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We also tried the nightly 2.0.x build from July 12 2010.
>>>>>>>>> Unfortunately,
>>>>>>>>>> the heapspace gets exhausted in a similar way.
>>>>>>>>>>
>>>>>>>>>> Any help appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Greetings,
>>>>>>>>>>
>>>>>>>>>> Egon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>
-----------------------------------------------------------------------
>>>>>>>>> -------
>>>>>>>>>> This SF.net email is sponsored by Sprint
>>>>>>>>>> What will you do first with EVO, the first 4G phone?
>>>>>>>>>> Visit sprint.com/first --
http://p.sf.net/sfu/sprint-com-first
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Geoserver-devel mailing list
>>>>>>>>>> Geoserver-devel@lists.sourceforge.net
>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Justin Deoliveira
>>>>>>>>> OpenGeo - http://opengeo.org
>>>>>>>>> Enterprise support for open source geospatial.
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>
-----------------------------------------------------------------------
>>>>>>>>> -------
>>>>>>>>> This SF.net email is sponsored by Sprint
>>>>>>>>> What will you do first with EVO, the first 4G phone?
>>>>>>>>> Visit sprint.com/first --
http://p.sf.net/sfu/sprint-com-first
>>>>>>>>> _______________________________________________
>>>>>>>>> Geoserver-devel mailing list
>>>>>>>>> Geoserver-devel@lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
-----------------------------------------------------------------------
>>>>>> -------
>>>>>>>> This SF.net email is sponsored by Sprint
>>>>>>>> What will you do first with EVO, the first 4G phone?
>>>>>>>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>>>>>>>> _______________________________________________
>>>>>>>> Geoserver-devel mailing list
>>>>>>>> Geoserver-devel@lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Justin Deoliveira
>>>>>>> OpenGeo - http://opengeo.org
>>>>>>> Enterprise support for open source geospatial.
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
-----------------------------------------------------------------------
>>>>>> -------
>>>>>>> This SF.net email is sponsored by Sprint
>>>>>>> What will you do first with EVO, the first 4G phone?
>>>>>>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>>>>>>> _______________________________________________
>>>>>>> Geoserver-devel mailing list
>>>>>>> Geoserver-devel@lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -------------------------------------
>>>>>> Geodan IT b.v.
>>>>>> Buitenhaven 27-A
>>>>>> 5211 TP 's-Hertogenbosch (NL)
>>>>>> -------------------------------------
>>>>>> Tel: +31 (0)73 - 692 5151
>>>>>> Fax: +31 (0)20 - 5711 333
>>>>>> -------------------------------------
>>>>>> Postadres / mailing address
>>>>>> President Kennedylaan 1
>>>>>> 1079 MB Amsterdam (NL)
>>>>>> -------------------------------------
>>>>>> Website: www.geodan.nl
>>>>>> Disclaimer: www.geodan.nl/disclaimer
>>>>>> -------------------------------------
>>>>>>
>>>>>>
>>>>
>>
-----------------------------------------------------------------------
>>>>>> -------
>>>>>> This SF.net email is sponsored by Sprint
>>>>>> What will you do first with EVO, the first 4G phone?
>>>>>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>>>>>> _______________________________________________
>>>>>> Geoserver-devel mailing list
>>>>>> Geoserver-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Justin Deoliveira
>>>> OpenGeo - http://opengeo.org
>>>> Enterprise support for open source geospatial.
>>>
>>>
>>
>>
>> --
>> Justin Deoliveira
>> OpenGeo - http://opengeo.org
>> Enterprise support for open source geospatial.
>
>
>
>
-----------------------------------------------------------------------
-------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> Geoserver-devel mailing list
> Geoserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-devel
--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
-----------------------------------------------------------------------
-------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel