[Geoserver-devel] [jira] Created: (GEOS-3873) GeoServer binary distribution might use 100% constantly due to a bug in the JVM

GeoServer binary distribution might use 100% constantly due to a bug in the JVM
-------------------------------------------------------------------------------

                 Key: GEOS-3873
                 URL: http://jira.codehaus.org/browse/GEOS-3873
             Project: GeoServer
          Issue Type: Bug
            Reporter: Andrea Aime
            Assignee: Andrea Aime
             Fix For: 2.0.2

See: http://jira.codehaus.org/browse/JETTY-937

We should try upgrading the Jetty we embed in GeoServer.

Btw, noticed the problem on demo.opengeo.org

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hello Andrea,

i think i race exactly this behavior with 100% constantly due to a bug in
the JVM described in GEOS-3873 issue track.

I run GeoServer 1.7.5 with JDK 1.6.0_06 and for a week i had this problem.
The java process had 100% percent cpu consumption and could only be bing
back to normal state by a restart.

I had read the Jetty issue track: http://jira.codehaus.org/browse/JETTY-937
and it seems to going on right my behavior.

Question is, how can i use the upgrade.patch distributed in GEOS-3873 issue
track?
A upgrad of Jetty from version 6.1.8 to 6.1.24 will fix this bug, but i
don't know how i have to handle with this patch discrition.

Thank for reply and best wishes,

Lg 2StepForward

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6286892.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Tue, Apr 19, 2011 at 1:31 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hello Andrea,

i think i race exactly this behavior with 100% constantly due to a bug in
the JVM described in GEOS-3873 issue track.

I run GeoServer 1.7.5 with JDK 1.6.0_06 and for a week i had this problem.
The java process had 100% percent cpu consumption and could only be bing
back to normal state by a restart.

I had read the Jetty issue track: http://jira.codehaus.org/browse/JETTY-937
and it seems to going on right my behavior.

Question is, how can i use the upgrade.patch distributed in GEOS-3873 issue
track?
A upgrad of Jetty from version 6.1.8 to 6.1.24 will fix this bug, but i
don't know how i have to handle with this patch discrition.

Ah... difficult to say, you're using very old software that has not been
in a supported state for well over than one year.
We also just released what is likely the last of the 2.0.x series and
concentrating
only on the 2.1.x series now.

The JDK also is very old and buggy compared to the latest version 1.6.0_24.

If you need to stay on the 1.7.x series maybe you can just download the 1.7.7
war release and deploy it into tomcat instead?

I don't see any easy "click the button" or "change a file" solution here.

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

Hello Andrea,

yes is right. It's an old one, but i can't update to newer one at this time.
I had upgrade the JDK to last 1.6.0_24 version and since the version
1.6.0_18 this bug should be fixed by sun, but i don't know if it only a
problem of jdk or is the jetty involved by himself.

Jetty solve this problem by himself upon the version of 6.1.17 with a
workaround.
I could try to implement the new jetty 6.1.24 into the geoserver 1.7.5 but i
don't know if there is a explicit binding from geoserver to jetty or is it
possible to use the new jetty jar's in the lib directory?

It will be very helpfull if you could tell me on short way if there are
heavy bindings between geoserver and jetty or only light bindings maybe
references by start.jar file.

Thank a lot for reply,

Best wishes 2StepForward

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6287163.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Tue, Apr 19, 2011 at 2:54 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hello Andrea,

yes is right. It's an old one, but i can't update to newer one at this time.
I had upgrade the JDK to last 1.6.0_24 version and since the version
1.6.0_18 this bug should be fixed by sun, but i don't know if it only a
problem of jdk or is the jetty involved by himself.

Jetty solve this problem by himself upon the version of 6.1.17 with a
workaround.
I could try to implement the new jetty 6.1.24 into the geoserver 1.7.5 but i
don't know if there is a explicit binding from geoserver to jetty or is it
possible to use the new jetty jar's in the lib directory?

Hmm... we just removed all the libraries and extra files we don't need.
Replacing the jars might work, or not, I'm not sure.

It will be very helpfull if you could tell me on short way if there are
heavy bindings between geoserver and jetty or only light bindings maybe
references by start.jar file.

Besides the careful file selection to get the lightest possible distribution
we made no other changes to Jetty (that I know of)

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

Hi Andrea,

first thanks for helping me.
Last question is, if i upgrade to a higher version of GS, there i will also
use the jetty 6.1.8 version?

So i will try to implement the new jetty in the old version. If i had done
so, i could give you my expiriences.

Thank for help and have a nice day,

Best wishes 2StepForward

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6287309.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

Hello,

after a long time to search where the 100% percent cpu consumtion come from
i had found it out.

Our backend is a Oracle 10g database which is bounded by Oracle NG Plugin to
geoserver.
After a long period of WFS-Requests the java heap runs into a extensive GC.

The printout of jmap show full memory in Edge and old Generations:

./jmap -heap 7881
Attaching to process ID 7881, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize = 629145600 (600.0MB)
   NewSize = 1048576 (1.0MB)
   MaxNewSize = 4294901760 (4095.9375MB)
   OldSize = 4194304 (4.0MB)
   NewRatio = 2
   SurvivorRatio = 8
   PermSize = 16777216 (16.0MB)
   MaxPermSize = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 137101312 (130.75MB)
   used = 137101312 (130.75MB)
   free = 0 (0.0MB)
   100.0% used

From Space:

   capacity = 34340864 (32.75MB)
   used = 0 (0.0MB)
   free = 34340864 (32.75MB)
   0.0% used
To Space:
   capacity = 36306944 (34.625MB)
   used = 0 (0.0MB)
   free = 36306944 (34.625MB)
   0.0% used
PS Old Generation
   capacity = 419430400 (400.0MB)
   used = 419430400 (400.0MB)
   free = 0 (0.0MB)
   100.0% used
PS Perm Generation
   capacity = 46137344 (44.0MB)
   used = 46061712 (43.92787170410156MB)
   free = 75632 (0.0721282958984375MB)
   99.83607205477628% used

After the GC had run the Old generation will only reduced to
99.99998664855957% used and new will stay unchanged by 100.0% used.

It looks like that the Oracle NG Plugin has a memory leak. I expect that
there a many active referenced
objects in memory which are not needed to run the aktive programm state.

If i print the thread info, there is one thread (not the specific GC
threads) that use a lot of time to make
PSMarkSweep GC:

thread information for 7888:
  PID (TID) (PPID) : 7881 (7888) (1)
  Process status : Running, Multi Threaded
  Running Time : Tue May 24 08:34:06 2011 (02:37:43)
  CPU Time : 00:39:48
  CPU usage : 25.2%
  Process flags : Forked but no Exec
  User:Group : geoserver(1001):geoserver(1002) (-M)
  Priority : 0
  Last System Function: - (-)
  CPU ID : 1
  Instruction Pointer : 0xb77d02f0
  Stack Pointer : 0x8a33bdf4
  Stack Bottom : 0xbfff7b30
  Additional Info : Environment (-e), Stack Trace (-S), Registers (-R),
Signals (-g)

  ---------------- Stack Trace --------------
  #0 0xb75c9118 in instanceKlass::oop_adjust_pointers () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #1 0xb7821728 in PSMarkSweepDecorator::adjust_pointers () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #2 0xb7821041 in PSMarkSweep::mark_sweep_phase3 () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #3 0xb781ff52 in PSMarkSweep::invoke_no_policy () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #4 0xb782a82e in PSScavenge::invoke () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #5 0xb77f1639 in ParallelScavengeHeap::failed_mem_allocate () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #6 0xb7922a1c in VM_ParallelGCFailedAllocation::doit () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #7 0xb79310e6 in VM_Operation::evaluate () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #8 0xb7930593 in VMThread::evaluate_operation () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #9 0xb7930800 in VMThread::loop () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #10 0xb79302f0 in VMThread::run () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #11 0xb77dfa6e in java_start () from
/usr/java/jdk1.6.0_24/jre/lib/i386/server/libjvm.so
  #12 0xb7f4b34b in start_thread () from /lib/libpthread.so.0
  #13 0xb7ed565e in clone () from /lib/libc.so.6

It's a reproducable case.
I had test it with POSTGIS and MYSQL but there the memory management by GC
is running clean.
Only by Oracle NG the memory is running full after about 140.000
WFS-Requests.

Is there a known memory leak or a bug in this plugin.
It will help a lot. I try to repair this error since a long time.

Thanks to all and greetings,
2StepForward

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6398297.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Tue, May 24, 2011 at 2:14 PM, 2StepForward <kwegezeder@anonymised.com22…> wrote:

Hello,

after a long time to search where the 100% percent cpu consumtion come from
i had found it out.

You don’t say which version of GeoServer you’re using?
GS 2.0.2 has a known memory leak in WFS.

In any case, upgrade to 2.1.0, try again, if the memory leak is still there
provide data and a request script to make it reproducable and attach
it to a bug report on jira.codehaus.org

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Hello Andrea,

I use the Version 1.6.5. Very old but I could not change at the moment to a
newer one.

So my hope is, that a known bug (memory leak) exists, since this is a older
version.
And second hope is, i could change only the plugin, because otherwise
everything is going well.

I have tried so search a jira bugreport of any kind of bug who expect like
this behavior.

Greets,
2StepForward

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6398400.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Tue, May 24, 2011 at 2:50 PM, 2StepForward <kwegezeder@anonymised.com22…> wrote:

Hello Andrea,

I use the Version 1.6.5. Very old but I could not change at the moment to a
newer one.

So my hope is, that a known bug (memory leak) exists, since this is a older
version.
And second hope is, i could change only the plugin, because otherwise
everything is going well.

I have tried so search a jira bugreport of any kind of bug who expect like
this behavior.

The leak I was talking about is very ancient, would not be surprised if it was
in 1.6.x as well.
However the fix is only in 2.1.0, if you cannot upgrade I can’t help you

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Thanks for your reply,

can you tell me at least which jira or codehouse bug id this fix has.
So maybe i can solve this problem by myself with new compiling the oracle
plugin.

Thanks and greetings

--
View this message in context: http://osgeo-org.1803224.n2.nabble.com/jira-Created-GEOS-3873-GeoServer-binary-distribution-might-use-100-constantly-due-to-a-bug-in-the-JVM-tp6128411p6398496.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.