[Geoserver-users] Memory Leak

GeoServer GeoServer User
1

Hello,

I am trying to run a Web Feature Service with GeoServer but I am experiencing a memory leak. I have a Java program that sends new features to GeoServer in an http post that then get added to a PostGIS database. The total amount of data that gets sent to the server is very small, usually less than a couple of dozen of features per http post. Also each feature is only made up of five fields (a position, a time stamp, and a few numbers) so the amount of data that I am expecting GeoServer to handle is very small. After about a day of running the program on tomcat I found that the server was using up much more memory than it had the day before. I looked at the heap and found that the class java.util.HashMap had more instances then any other class and was taking up more memory then any other class. Does anyone have any idea what could be causing this memory leak? I am running GeoServer version 1.6.4 and tomcat version 6.0.18.

Brett

2

Hi Brett,

there's a usual suspect, xml paser. But I may be wrong and the leak laying out
in another place. That inspection of yours does not tell who creates the
hashmaps, does it?

I'm tempted to hit geoserver like that but running under a profiler for a
while but that'd be constrained by higher priorities. Wonder if you could
manage to make the same longevity testing with Yourkit java profiler (there's
a trial version and we have geotools specific licences) so we can analyze a
profiler's snapshot?

best regards,

Gabriel

On Friday 05 September 2008 05:06:40 pm Levasseur, Brett wrote:

Hello,

I am trying to run a Web Feature Service with GeoServer but I am
experiencing a memory leak. I have a Java program that sends new features
to GeoServer in an http post that then get added to a PostGIS database. The
total amount of data that gets sent to the server is very small, usually
less than a couple of dozen of features per http post. Also each feature is
only made up of five fields (a position, a time stamp, and a few numbers)
so the amount of data that I am expecting GeoServer to handle is very
small. After about a day of running the program on tomcat I found that the
server was using up much more memory than it had the day before. I looked
at the heap and found that the class java.util.HashMap had more instances
then any other class and was taking up more memory then any other class.
Does anyone have any idea what could be causing this memory leak? I am
running GeoServer version 1.6.4 and tomcat version 6.0.18.

Brett

3

Can you explain a bit more about what with the XML Parser causes it to leak memory? I am also seeing some memory leaks and it appears that it is related to the XML Parser. Any hints on how to track this down, or where to look for typical problems in the code would be greatly appreciated. TNX David

Gabriel Roldán wrote:

··· 
-- 

David R Robison
Open Roads Consulting, Inc.
708 S. Battlefield Blvd., Chesapeake, VA 23322
phone: (757) 546-3401
e-mail: [drrobison@anonymised.com](mailto:drrobison@anonymised.com)
web: [http://openroadsconsulting.com](http://openroadsconsulting.com)
blog: [http://therobe.blogspot.com](http://therobe.blogspot.com)
book: [http://www.xulonpress.com/book_detail.php?id=2579](http://www.xulonpress.com/book_detail.php?id=2579)

This e-mail communication (including any attachments) may contain confidential and/or privileged material intended solely for the individual or entity to which it is addressed.  If you are not the intended recipient, you should immediately stop reading this message and delete it from all computers that it resides on. Any unauthorized reading, distribution, copying or other use of this communication (or its attachments) is strictly prohibited.  If you have received this communication in error, please notify us immediately.
4

David R Robison ha scritto:

Can you explain a bit more about what with the XML Parser causes it to leak memory? I am also seeing some memory leaks and it appears that it is related to the XML Parser. Any hints on how to track this down, or where to look for typical problems in the code would be greatly appreciated. TNX David

The only way I know is to run a stress test and keep GeoServer in
a memory profiler to see where the maps are getting allocated.
This has already been reproted by someone else too, I kept GeoServer
running for 2+ hours with 40 clients threads hitting it with WFS requests, no luck, the memory profiler showed no leaks.
So I guess there is something specific to your requests or your data
that makes this happen, but I have no idea of what.

Have you tried upgrading to 1.7.x and see if the bug occurs there
too?

Cheers
Andrea

5

hi david,

no i can't know without a profiler, that's what i was saying before.

gabriel
On Tuesday 09 September 2008 09:37:41 am David R Robison wrote:

Can you explain a bit more about what with the XML Parser causes it to
leak memory? I am also seeing some memory leaks and it appears that it
is related to the XML Parser. Any hints on how to track this down, or
where to look for typical problems in the code would be greatly
appreciated. TNX David

Gabriel Roldán wrote:
> Hi Brett,
>
> there's a usual suspect, xml paser. But I may be wrong and the leak
> laying out in another place. That inspection of yours does not tell who
> creates the hashmaps, does it?
>
> I'm tempted to hit geoserver like that but running under a profiler for a
> while but that'd be constrained by higher priorities. Wonder if you could
> manage to make the same longevity testing with Yourkit java profiler
> (there's a trial version and we have geotools specific licences) so we
> can analyze a profiler's snapshot?
>
> best regards,
>
> Gabriel
>
> On Friday 05 September 2008 05:06:40 pm Levasseur, Brett wrote:
>> Hello,
>>
>> I am trying to run a Web Feature Service with GeoServer but I am
>> experiencing a memory leak. I have a Java program that sends new
>> features to GeoServer in an http post that then get added to a PostGIS
>> database. The total amount of data that gets sent to the server is very
>> small, usually less than a couple of dozen of features per http post.
>> Also each feature is only made up of five fields (a position, a time
>> stamp, and a few numbers) so the amount of data that I am expecting
>> GeoServer to handle is very small. After about a day of running the
>> program on tomcat I found that the server was using up much more memory
>> than it had the day before. I looked at the heap and found that the
>> class java.util.HashMap had more instances then any other class and was
>> taking up more memory then any other class. Does anyone have any idea
>> what could be causing this memory leak? I am running GeoServer version
>> 1.6.4 and tomcat version 6.0.18.
>>
>> Brett
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge Build the coolest Linux based applications with Moblin SDK &
> win great prizes Grand prize is a trip for two to an Open Source event
> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

6

Gabriel,

I will check out using Yourkit like you suggested. Until then I used jhat to go through a dump of the heap from one of my trials. I checked the first 150 instances of the class java.util.HashMap$Entry and found that among them 147 had the same setup with a key being an instance of the class javax.xml.namespace.QName and a value being an instance of the class org.eclipse.xsd.impl.XSDElementDecleration. These instances of the HashMap class are referenced by instances of the class org.geotools.xml.impl.SchemaIndexImpl.

I will know more once I have had the chance to collect more data, however if the appearance of the classes I have mentioned above proves to be the pattern would this fall under the category of the xml parser?

Brett Levasseur

-----Original Message-----
From: Gabriel Roldán [mailto:groldan@anonymised.com]
Sent: Tuesday, September 09, 2008 10:47 AM
To: David R Robison
Cc: geoserver-users@lists.sourceforge.net; Levasseur, Brett
Subject: Re: [Geoserver-users] Memory Leak

hi david,

no i can't know without a profiler, that's what i was saying before.

gabriel
On Tuesday 09 September 2008 09:37:41 am David R Robison wrote:

Can you explain a bit more about what with the XML Parser causes it to
leak memory? I am also seeing some memory leaks and it appears that it
is related to the XML Parser. Any hints on how to track this down, or
where to look for typical problems in the code would be greatly
appreciated. TNX David

Gabriel Roldán wrote:
> Hi Brett,
>
> there's a usual suspect, xml paser. But I may be wrong and the leak
> laying out in another place. That inspection of yours does not tell who
> creates the hashmaps, does it?
>
> I'm tempted to hit geoserver like that but running under a profiler for a
> while but that'd be constrained by higher priorities. Wonder if you could
> manage to make the same longevity testing with Yourkit java profiler
> (there's a trial version and we have geotools specific licences) so we
> can analyze a profiler's snapshot?
>
> best regards,
>
> Gabriel
>
> On Friday 05 September 2008 05:06:40 pm Levasseur, Brett wrote:
>> Hello,
>>
>> I am trying to run a Web Feature Service with GeoServer but I am
>> experiencing a memory leak. I have a Java program that sends new
>> features to GeoServer in an http post that then get added to a PostGIS
>> database. The total amount of data that gets sent to the server is very
>> small, usually less than a couple of dozen of features per http post.
>> Also each feature is only made up of five fields (a position, a time
>> stamp, and a few numbers) so the amount of data that I am expecting
>> GeoServer to handle is very small. After about a day of running the
>> program on tomcat I found that the server was using up much more memory
>> than it had the day before. I looked at the heap and found that the
>> class java.util.HashMap had more instances then any other class and was
>> taking up more memory then any other class. Does anyone have any idea
>> what could be causing this memory leak? I am running GeoServer version
>> 1.6.4 and tomcat version 6.0.18.
>>
>> Brett
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge Build the coolest Linux based applications with Moblin SDK &
> win great prizes Grand prize is a trip for two to an Open Source event
> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

7

On Tuesday 09 September 2008 04:37:03 pm Levasseur, Brett wrote:

Gabriel,

I will check out using Yourkit like you suggested. Until then I used jhat
to go through a dump of the heap from one of my trials. I checked the first
150 instances of the class java.util.HashMap$Entry and found that among
them 147 had the same setup with a key being an instance of the class
javax.xml.namespace.QName and a value being an instance of the class
org.eclipse.xsd.impl.XSDElementDecleration. These instances of the HashMap
class are referenced by instances of the class
org.geotools.xml.impl.SchemaIndexImpl.

I will know more once I have had the chance to collect more data, however
if the appearance of the classes I have mentioned above proves to be the
pattern would this fall under the category of the xml parser?

yes, or rather under the category of the whole xml subsystem, so called gt-xsd
extensions. These are xml schema aware parsers that use java binding classes
and the eclipse emf framework. I wonder if we are certain that the leak is
due to stale HashMap instances, and I would suspect of the xsd caches, but to
have more than guessing you may want to contact Justin, cc'ed here, whose the
expert on this.

keep us posted.

Gabriel

Brett Levasseur

-----Original Message-----
From: Gabriel Roldán [mailto:groldan@anonymised.com]
Sent: Tuesday, September 09, 2008 10:47 AM
To: David R Robison
Cc: geoserver-users@lists.sourceforge.net; Levasseur, Brett
Subject: Re: [Geoserver-users] Memory Leak

hi david,

no i can't know without a profiler, that's what i was saying before.

gabriel

On Tuesday 09 September 2008 09:37:41 am David R Robison wrote:
> Can you explain a bit more about what with the XML Parser causes it to
> leak memory? I am also seeing some memory leaks and it appears that it
> is related to the XML Parser. Any hints on how to track this down, or
> where to look for typical problems in the code would be greatly
> appreciated. TNX David
>
> Gabriel Roldán wrote:
> > Hi Brett,
> >
> > there's a usual suspect, xml paser. But I may be wrong and the leak
> > laying out in another place. That inspection of yours does not tell who
> > creates the hashmaps, does it?
> >
> > I'm tempted to hit geoserver like that but running under a profiler for
> > a while but that'd be constrained by higher priorities. Wonder if you
> > could manage to make the same longevity testing with Yourkit java
> > profiler (there's a trial version and we have geotools specific
> > licences) so we can analyze a profiler's snapshot?
> >
> > best regards,
> >
> > Gabriel
> >
> > On Friday 05 September 2008 05:06:40 pm Levasseur, Brett wrote:
> >> Hello,
> >>
> >> I am trying to run a Web Feature Service with GeoServer but I am
> >> experiencing a memory leak. I have a Java program that sends new
> >> features to GeoServer in an http post that then get added to a PostGIS
> >> database. The total amount of data that gets sent to the server is
> >> very small, usually less than a couple of dozen of features per http
> >> post. Also each feature is only made up of five fields (a position, a
> >> time stamp, and a few numbers) so the amount of data that I am
> >> expecting GeoServer to handle is very small. After about a day of
> >> running the program on tomcat I found that the server was using up
> >> much more memory than it had the day before. I looked at the heap and
> >> found that the class java.util.HashMap had more instances then any
> >> other class and was taking up more memory then any other class. Does
> >> anyone have any idea what could be causing this memory leak? I am
> >> running GeoServer version 1.6.4 and tomcat version 6.0.18.
> >>
> >> Brett
> >
> > -----------------------------------------------------------------------
> >-- This SF.Net email is sponsored by the Moblin Your Move Developer's
> > challenge Build the coolest Linux based applications with Moblin SDK &
> > win great prizes Grand prize is a trip for two to an Open Source event
> > anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Geoserver-users mailing list
> > Geoserver-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-users

8

Gabriel & Justin,

I have used Yourkit to collect some more information. I made two snapshots of the heap while running my tests. In Yourkit I used the Inspections feature and ran the tests under 'Possible leaks'. The 'Lost SWT Controls' test returned nothing. The 'Objects Retained by Inner Class Back References' test did return a few classes. Between the first and the second snapshot the classes that the test returned and the number of instances were the same for all but one. In my first snapshot the class org.geotools.xml.impl.SchemaIndexImpl appeared 470 times taking up 15% of the heaps retained size. In my second snapshot this class appeared 940 times taking up 25% of the heaps retained size. All of the instances of this class were made up of an array of org.eclipse.xsd.XSDSchemaImpl objects and three HashMap objects.

Under the Memory feature of Yourkit I found that the HashMap class was taking up the majority of the memory in both snapshots. I then used the 'Merged paths' feature of Yourkit to determine what objects were retaining the memory held by HashMap. I found that the one class that held more of this memory than any other was again org.geotools.xml.impl.SchemaIndexImpl. The memory for this class was being held by org.geotools.xml.impl.SchemaIndex$SchemaAdapter whose memory was held by org.eclipse.emf.notify.Adapter whose memory was held by org.eclipse.emf.common.notify.impl.BasicNotifierImpl$EAdapterList whose memory was held by org.eclipse.xsd.impl.XSDSchemaImpl whose memory was held by instances of multiple other classes.

From this information I now believe that while the HashMap is taking up the majority of memory it is most likely because of the class org.geotools.xmlimpl.SchemaIndexImpl or one of the classes that retain a hold on that class that I listed above.

The WFS insert transaction statements and getFeature requests that I send the server use the schemas for wfs, ogc, gml, and the regular xml. Also I created another schema to represent my own data. Is it possible that this memory leak is occurring because of my schema document? Currently my instance of GeoServer does not have access to my schema document.

Brett

-----Original Message-----
From: Gabriel Roldán [mailto:groldan@anonymised.com]
Sent: Tuesday, September 09, 2008 4:05 PM
To: Levasseur, Brett; Justin Deoliveira
Cc: David R Robison; geoserver-users@lists.sourceforge.net
Subject: Re: [Geoserver-users] Memory Leak

On Tuesday 09 September 2008 04:37:03 pm Levasseur, Brett wrote:

Gabriel,

I will check out using Yourkit like you suggested. Until then I used jhat
to go through a dump of the heap from one of my trials. I checked the first
150 instances of the class java.util.HashMap$Entry and found that among
them 147 had the same setup with a key being an instance of the class
javax.xml.namespace.QName and a value being an instance of the class
org.eclipse.xsd.impl.XSDElementDecleration. These instances of the HashMap
class are referenced by instances of the class
org.geotools.xml.impl.SchemaIndexImpl.

I will know more once I have had the chance to collect more data, however
if the appearance of the classes I have mentioned above proves to be the
pattern would this fall under the category of the xml parser?

yes, or rather under the category of the whole xml subsystem, so called gt-xsd
extensions. These are xml schema aware parsers that use java binding classes
and the eclipse emf framework. I wonder if we are certain that the leak is
due to stale HashMap instances, and I would suspect of the xsd caches, but to
have more than guessing you may want to contact Justin, cc'ed here, whose the
expert on this.

keep us posted.

Gabriel

Brett Levasseur

-----Original Message-----
From: Gabriel Roldán [mailto:groldan@anonymised.com]
Sent: Tuesday, September 09, 2008 10:47 AM
To: David R Robison
Cc: geoserver-users@lists.sourceforge.net; Levasseur, Brett
Subject: Re: [Geoserver-users] Memory Leak

hi david,

no i can't know without a profiler, that's what i was saying before.

gabriel

On Tuesday 09 September 2008 09:37:41 am David R Robison wrote:
> Can you explain a bit more about what with the XML Parser causes it to
> leak memory? I am also seeing some memory leaks and it appears that it
> is related to the XML Parser. Any hints on how to track this down, or
> where to look for typical problems in the code would be greatly
> appreciated. TNX David
>
> Gabriel Roldán wrote:
> > Hi Brett,
> >
> > there's a usual suspect, xml paser. But I may be wrong and the leak
> > laying out in another place. That inspection of yours does not tell who
> > creates the hashmaps, does it?
> >
> > I'm tempted to hit geoserver like that but running under a profiler for
> > a while but that'd be constrained by higher priorities. Wonder if you
> > could manage to make the same longevity testing with Yourkit java
> > profiler (there's a trial version and we have geotools specific
> > licences) so we can analyze a profiler's snapshot?
> >
> > best regards,
> >
> > Gabriel
> >
> > On Friday 05 September 2008 05:06:40 pm Levasseur, Brett wrote:
> >> Hello,
> >>
> >> I am trying to run a Web Feature Service with GeoServer but I am
> >> experiencing a memory leak. I have a Java program that sends new
> >> features to GeoServer in an http post that then get added to a PostGIS
> >> database. The total amount of data that gets sent to the server is
> >> very small, usually less than a couple of dozen of features per http
> >> post. Also each feature is only made up of five fields (a position, a
> >> time stamp, and a few numbers) so the amount of data that I am
> >> expecting GeoServer to handle is very small. After about a day of
> >> running the program on tomcat I found that the server was using up
> >> much more memory than it had the day before. I looked at the heap and
> >> found that the class java.util.HashMap had more instances then any
> >> other class and was taking up more memory then any other class. Does
> >> anyone have any idea what could be causing this memory leak? I am
> >> running GeoServer version 1.6.4 and tomcat version 6.0.18.
> >>
> >> Brett
> >
> > -----------------------------------------------------------------------
> >-- This SF.Net email is sponsored by the Moblin Your Move Developer's
> > challenge Build the coolest Linux based applications with Moblin SDK &
> > win great prizes Grand prize is a trip for two to an Open Source event
> > anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Geoserver-users mailing list
> > Geoserver-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-users