[GeoNetwork-users] Java updates

Hendrix_Rik · April 20, 2010, 12:50pm

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)
http://www.i-sup2010.org
---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

(attachments)

hs_err_pid3248.log (64 KB)

Timo_Proescholdt · April 20, 2010, 2:41pm

Hi,

make sure you use the right JVM for your architecture.
I had the same issues and the problem turned out to be the wrong JVM
for a 64bit machine.

If you use 64bit linux I'd be almost sure this is the reason for your problem.

best
Timo

On 4/20/2010 2:50 PM, Hendrix Rik wrote:

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)
http://www.i-sup2010.org
---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

sppigot · April 21, 2010, 4:12am

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880 ! net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [rik.hendrix@anonymised.com]
Sent: Tuesday, 20 April 2010 10:50 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)

---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

sppigot · April 21, 2010, 9:47am

Not quite as nicely as I thought - the fix below is still not reliable - I'll see what else I can find out.
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Wednesday, 21 April 2010 2:12 PM
To: rik.hendrix@anonymised.com; geonetwork-users@lists.sourceforge.net; geonetwork-devel@lists.sourceforge.net
Subject: [ExternalEmail] Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880 ! net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [rik.hendrix@anonymised.com]
Sent: Tuesday, 20 April 2010 10:50 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)

---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

sppigot · April 21, 2010, 10:50am

Apologies for all the followups: the fix suggested in the original devcomments link seems fine actually - I was tripping another issue that has been fixed elsewhere. I've committed the fix as part of 2.4.3 so it should be ok to use jdks > 1.6.0_17/6u17 for 2.4.3.

Cheers,
Simon
________________________________________
From: Pigot, Simon (CMAR, Hobart)
Sent: Wednesday, 21 April 2010 7:47 PM
To: Pigot, Simon (CMAR, Hobart); rik.hendrix@anonymised.com; geonetwork-users@anonymised.comsts.sourceforge.net; geonetwork-devel@lists.sourceforge.net
Subject: RE: [GeoNetwork-users] Java updates

Not quite as nicely as I thought - the fix below is still not reliable - I'll see what else I can find out.
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Wednesday, 21 April 2010 2:12 PM
To: rik.hendrix@anonymised.com; geonetwork-users@lists.sourceforge.net; geonetwork-devel@lists.sourceforge.net
Subject: [ExternalEmail] Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880 ! net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [rik.hendrix@anonymised.com]
Sent: Tuesday, 20 April 2010 10:50 PM
To: geonetwork-users@lists.sourceforge.net
Subject: [GeoNetwork-users] Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)

---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Frank1 · May 9, 2010, 4:12am

it seems that the submitted fix is not working for us. We are using Geonetwork 2.4.3 and have tried to upgrade to JDK 1.6.0_20 Linux 64 Bit. Geonetwork still crashes after a while with an out-of-swap-space exception.

After downgrading to JDK 1.6.0_16 Linux 64 Bit things work correctly again.

best regards
Frank Lieber

sppigot · May 9, 2010, 11:54am

Hi Frank,

Just tested with and without the fix here - also using jdk 1.6.0_20 & Linux 64bit - actually been using GeoNetwork successfully now for some weeks with this fix.

Are you using jetty? If so can you check that your start-geonetwork.sh script has -XX:CompileCommandFile=hotspot_compiler and that a file called hotspot_compiler exists in the jetty directory? (this is working and setup by default with 2.4.3 - but maybe you have customised this?)

If using tomcat then you'll need to make sure that the -XX:Compile.... argument above is in the JAVA_OPTS environment variable (or wherever else you are supplying the jvm args to tomcat) and the hotspot_compiler file is available wherever tomcat needs it (this could vary depending on which way you are starting up tomcat I guess).

Cheers,
Simon
________________________________________
From: FrankL [liefra@anonymised.com]
Sent: Sunday, 9 May 2010 2:12 PM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

it seems that the submitted fix is not working for us. We are using Geonetwork 2.4.3 and have tried to upgrade to JDK 1.6.0_20 Linux 64 Bit. Geonetwork still crashes after a while with an out-of-swap-space exception.

After downgrading to JDK 1.6.0_16 Linux 64 Bit things work correctly again.

best regards
Frank Lieber

On 21/04/10 17:50, Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com> wrote:

Apologies for all the followups: the fix suggested in the original devcomments link seems fine actually - I was tripping another issue that has been fixed elsewhere. I've committed the fix as part of 2.4.3 so it should be ok to use jdks > 1.6.0_17/6u17 for 2.4.3.

Cheers,
Simon
________________________________________
From: Pigot, Simon (CMAR, Hobart)
Sent: Wednesday, 21 April 2010 7:47 PM
To: Pigot, Simon (CMAR, Hobart); rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com…447…>; geonetwork-users@lists.sourceforge.net<mailto:geonetwork-users@anonymised.comists.sourceforge.net>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net>
Subject: RE: [GeoNetwork-users] Java updates

Not quite as nicely as I thought - the fix below is still not reliable - I'll see what else I can find out.
________________________________________
From: Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com> [Simon.Pigot@anonymised.com92...<mailto:Simon.Pigot@anonymised.com>]
Sent: Wednesday, 21 April 2010 2:12 PM
To: rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com>; geonetwork-users@anonymised.comists.sourceforge.net<mailto:geonetwork-users@lists.sourceforge.net>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comnet>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880 ! net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com>]
Sent: Tuesday, 20 April 2010 10:50 PM
To: geonetwork-users@lists.sourceforge.net<mailto:geonetwork-users@anonymised.comurceforge.net>
Subject: [GeoNetwork-users] Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)

---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Frank1 · May 9, 2010, 1:38pm

Hi Simon,

thanks a lot for your prompt and detailed response. We are running GN2.4.3 on tomcat 6.0.26. Your hint about the the missing hotspot_compiler argument was absolutely correct.

I just have added -XX:CompileCommandFile=/path/to/hotspot_compiler to the tomcat JAVA_OPTS environment variable and GN is running now on JDK 1.6.0_20 with normal CPU and without crashing - really great!

Thanks again and Cheers
Frank

Friis-Christensen_An · May 10, 2010, 7:36am

Dear Simon, all,

We have the same problem with a new JVM and W2008 64 bit.

Your suggestion solves the problem all right, although we simply added this line to the tomcat JAVA_OPTS:

-XX:CompileCommand=exclude,net/sf/saxon/event/ReceivingContentHandler.startElement

However, my question is now: Does this affect the performance of GeoNetwork? The command seems to disable JIT compiling of that method so does this not mean that performance risk to be degraded? Would you recommend to run GeoNetwork like this in a production environment?

Thanks for any input.

Best regards,
Anders

From: FrankL [mailto:liefra@anonymised.com]
Sent: Sunday, May 09, 2010 3:39 PM
To: Simon.Pigot@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Simon,

thanks a lot for your prompt and detailed response. We are running GN2.4.3 on tomcat 6.0.26. Your hint about the the missing hotspot_compiler argument was absolutely correct.

I just have added -XX:CompileCommandFile=/path/to/hotspot_compiler to the tomcat JAVA_OPTS environment variable and GN is running now on JDK 1.6.0_20 with normal CPU and without crashing - really great!

Thanks again and Cheers
Frank

On 09/05/10 18:54, Simon.Pigot@anonymised.com wrote:

Hi Frank,

Just tested with and without the fix here - also using jdk 1.6.0_20 & Linux 64bit - actually been using GeoNetwork successfully now for some weeks with this fix.

Are you using jetty? If so can you check that your start-geonetwork.sh script has -XX:CompileCommandFile=hotspot_compiler and that a file called hotspot_compiler exists in the jetty directory? (this is working and setup by default with 2.4.3 - but maybe you have customised this?)

If using tomcat then you'll need to make sure that the -XX:Compile.... argument above is in the JAVA_OPTS environment variable (or wherever else you are supplying the jvm args to tomcat) and the hotspot_compiler file is available wherever tomcat needs it (this could vary depending on which way you are starting up tomcat I guess).

Cheers,
Simon
________________________________________
From: FrankL [[liefra@anonymised.com](mailto:liefra@anonymised.com31...)]
Sent: Sunday, 9 May 2010 2:12 PM
To: [geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comts.sourceforge.net)
Subject: Re: [GeoNetwork-devel] [GeoNetwork-users]  Java updates

it seems that the submitted fix is not working for us. We are using Geonetwork 2.4.3 and have tried to upgrade to JDK 1.6.0_20 Linux 64 Bit. Geonetwork still crashes after a while with an out-of-swap-space exception.

After downgrading to JDK 1.6.0_16 Linux 64 Bit things work correctly again.

best regards
Frank Lieber

On 21/04/10 17:50, [Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)[<mailto:Simon.Pigot@anonymised.com>](mailto:Simon.Pigot@anonymised.com) wrote:

Apologies for all the followups: the fix suggested in the original devcomments link seems fine actually - I was tripping another issue that has been fixed elsewhere. I've committed the fix as part of 2.4.3 so it should be ok to use jdks > 1.6.0_17/6u17 for 2.4.3.

Cheers,
Simon
________________________________________
From: Pigot, Simon (CMAR, Hobart)
Sent: Wednesday, 21 April 2010 7:47 PM
To: Pigot, Simon (CMAR, Hobart); [rik.hendrix@anonymised.com](mailto:rik.hendrix@anonymised.com)[<mailto:rik.hendrix@anonymised.com>](mailto:rik.hendrix@anonymised.com); [geonetwork-users@anonymised.comt](mailto:geonetwork-users@lists.sourceforge.net)[<mailto:geonetwork-users@lists.sourceforge.net>](mailto:geonetwork-users@anonymised.com.sourceforge.net); [geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.com.sourceforge.net)[<mailto:geonetwork-devel@lists.sourceforge.net>](mailto:geonetwork-devel@lists.sourceforge.net)
Subject: RE: [GeoNetwork-users]  Java updates

Not quite as nicely as I thought - the fix below is still not reliable - I'll see what else I can find out.
________________________________________
From: [Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com...)[<mailto:Simon.Pigot@anonymised.com>](mailto:Simon.Pigot@anonymised.com) [[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)[<mailto:Simon.Pigot@anonymised.com>](mailto:Simon.Pigot@anonymised.com)]
Sent: Wednesday, 21 April 2010 2:12 PM
To: [rik.hendrix@anonymised.com](mailto:rik.hendrix@anonymised.com.)[<mailto:rik.hendrix@anonymised.com>](mailto:rik.hendrix@anonymised.com); [geonetwork-users@lists.sourceforge.net](mailto:geonetwork-users@anonymised.comge.net)[<mailto:geonetwork-users@lists.sourceforge.net>](mailto:geonetwork-users@lists.sourceforge.net); [geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)[<mailto:geonetwork-devel@anonymised.coms.sourceforge.net>](mailto:geonetwork-devel@lists.sourceforge.net)
Subject: [ExternalEmail] Re: [GeoNetwork-devel] [GeoNetwork-users]  Java updates

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880  !   net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

[http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm](http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm)

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [[rik.hendrix@anonymised.com](mailto:rik.hendrix@anonymised.com)[<mailto:rik.hendrix@anonymised.com>](mailto:rik.hendrix@anonymised.com)]
Sent: Tuesday, 20 April 2010 10:50 PM
To: [geonetwork-users@lists.sourceforge.net](mailto:geonetwork-users@anonymised.comts.sourceforge.net)[<mailto:geonetwork-users@lists.sourceforge.net>](mailto:geonetwork-users@lists.sourceforge.net)
Subject: [GeoNetwork-users]  Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
#  Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and  the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)
[http://www.i-sup2010.org](http://www.i-sup2010.org)
---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only.  Access to this e-mail by anyone other than the addressee(s) is unauthorized.  If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it.  In such case, you should destroy this message and notify the sender immediately.  If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments.  As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comourceforge.net)[<mailto:GeoNetwork-devel@lists.sourceforge.net>](mailto:GeoNetwork-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geonetwork-devel](https://lists.sourceforge.net/lists/listinfo/geonetwork-devel)
GeoNetwork OpenSource is maintained at [http://sourceforge.net/projects/geonetwork](http://sourceforge.net/projects/geonetwork)

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comourceforge.net)[<mailto:GeoNetwork-devel@lists.sourceforge.net>](mailto:GeoNetwork-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geonetwork-devel](https://lists.sourceforge.net/lists/listinfo/geonetwork-devel)
GeoNetwork OpenSource is maintained at [http://sourceforge.net/projects/geonetwork](http://sourceforge.net/projects/geonetwork)

sppigot · May 14, 2010, 5:34am

Hi Anders,

This 'fix' is obviously just a temporary fix until the underlying problem/feature in these latest Sun JVM versions is found and resolved.

My own limited testing with a profiler shows this particular method is pretty heavily used eg. it is invoked ~80,000 times to show the main page - more than enough to attract the attention of the JIT compiler and thus trigger the bug! Despite this, the profiler shows little or no difference between the time taken for stylesheet transformations with the compiler exclusion and without the compiler exclusion! My own feeling is that there is more work to be done to see what effect the exclusion has. Anyone else done any work on this?

Cheers,
Simon
________________________________________
From: Friis-Christensen, Anders [andfr@anonymised.com]
Sent: Monday, 10 May 2010 5:36 PM
To: geonetwork-devel@lists.sourceforge.net; Pigot, Simon (CMAR, Hobart)
Subject: RE: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Dear Simon, all,

We have the same problem with a new JVM and W2008 64 bit.

Your suggestion solves the problem all right, although we simply added this line to the tomcat JAVA_OPTS:

-XX:CompileCommand=exclude,net/sf/saxon/event/ReceivingContentHandler.startElement

However, my question is now: Does this affect the performance of GeoNetwork? The command seems to disable JIT compiling of that method so does this not mean that performance risk to be degraded? Would you recommend to run GeoNetwork like this in a production environment?

Thanks for any input.

Best regards,
Anders

________________________________
From: FrankL [mailto:liefra@anonymised.com]
Sent: Sunday, May 09, 2010 3:39 PM
To: Simon.Pigot@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Simon,

thanks a lot for your prompt and detailed response. We are running GN2.4.3 on tomcat 6.0.26. Your hint about the the missing hotspot_compiler argument was absolutely correct.

I just have added -XX:CompileCommandFile=/path/to/hotspot_compiler to the tomcat JAVA_OPTS environment variable and GN is running now on JDK 1.6.0_20 with normal CPU and without crashing - really great!

Thanks again and Cheers
Frank

On 09/05/10 18:54, Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com> wrote:

Hi Frank,

Just tested with and without the fix here - also using jdk 1.6.0_20 & Linux 64bit - actually been using GeoNetwork successfully now for some weeks with this fix.

Are you using jetty? If so can you check that your start-geonetwork.sh script has -XX:CompileCommandFile=hotspot_compiler and that a file called hotspot_compiler exists in the jetty directory? (this is working and setup by default with 2.4.3 - but maybe you have customised this?)

If using tomcat then you'll need to make sure that the -XX:Compile.... argument above is in the JAVA_OPTS environment variable (or wherever else you are supplying the jvm args to tomcat) and the hotspot_compiler file is available wherever tomcat needs it (this could vary depending on which way you are starting up tomcat I guess).

Cheers,
Simon
________________________________________
From: FrankL [liefra@anonymised.com<mailto:liefra@anonymised.com>]
Sent: Sunday, 9 May 2010 2:12 PM
To: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net>
Subject: Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

it seems that the submitted fix is not working for us. We are using Geonetwork 2.4.3 and have tried to upgrade to JDK 1.6.0_20 Linux 64 Bit. Geonetwork still crashes after a while with an out-of-swap-space exception.

After downgrading to JDK 1.6.0_16 Linux 64 Bit things work correctly again.

best regards
Frank Lieber

On 21/04/10 17:50, Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com> wrote:

Apologies for all the followups: the fix suggested in the original devcomments link seems fine actually - I was tripping another issue that has been fixed elsewhere. I've committed the fix as part of 2.4.3 so it should be ok to use jdks > 1.6.0_17/6u17 for 2.4.3.

Cheers,
Simon
________________________________________
From: Pigot, Simon (CMAR, Hobart)
Sent: Wednesday, 21 April 2010 7:47 PM
To: Pigot, Simon (CMAR, Hobart); rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com…447…><mailto:rik.hendrix@anonymised.com><mailto:rik.hendrix@anonymised.com>; geonetwork-users@lists.sourceforge.net<mailto:geonetwork-users@anonymised.come.net><mailto:geonetwork-users@lists.sourceforge.net><mailto:geonetwork-users@lists.sourceforge.net>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.comorge.net><mailto:geonetwork-devel@lists.sourceforge.net>
Subject: RE: [GeoNetwork-users] Java updates

Not quite as nicely as I thought - the fix below is still not reliable - I'll see what else I can find out.
________________________________________
From: Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com> [Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com2...>]
Sent: Wednesday, 21 April 2010 2:12 PM
To: rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com><mailto:rik.hendrix@…447…><mailto:rik.hendrix@anonymised.com>; geonetwork-users@anonymised.come.net<mailto:geonetwork-users@lists.sourceforge.net><mailto:geonetwork-users@lists.sourceforge.net><mailto:geonetwork-users@lists.sourceforge.net>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comrge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] [GeoNetwork-users] Java updates

Hi Rik,

Not actually a Geonetwork problem or memory leak in GeoNetwork code - I checked into this this morning as 2.4.3 is about to be released. Pretty much skyrocketing memory usage the moment anything XSLT is done when running under jdk 1.6 20. Looked into the hs error file and found:

Current CompileTask:
C2:880 ! net.sf.saxon.event.ReceivingContentHandler.startElement(......)

Turning that into google produced a link to something that makes everything settle down nicely:

http://www.devcomments.com/Saxon-JVM-crash-Out-of-swap-space-OOM-error-withsolution-at89036.htm

I'll commit this for 2.4.3 after a bit more testing etc.

Cheers,
Simon

________________________________________
From: Hendrix Rik [rik.hendrix@anonymised.com<mailto:rik.hendrix@anonymised.com><mailto:rik.hendrix@anonymised.com><mailto:rik.hendrix@anonymised.com>]
Sent: Tuesday, 20 April 2010 10:50 PM
To: geonetwork-users@lists.sourceforge.net<mailto:geonetwork-users@anonymised.comurceforge.net><mailto:geonetwork-users@lists.sourceforge.net><mailto:geonetwork-users@lists.sourceforge.net>
Subject: [GeoNetwork-users] Java updates

Dear Geonetwork users,

On a test server, I'm running Geonetwork on JRE version 6u17 (=1.6.0_17) for several months without stability problems.

Last week I upgraded JRE to 6u19. Geonetwork started crashing every few minutes.
Did the same upgrade on another server, same result.
Downgraded to 6u17, problem solved.
This week I upgraded to 6u20, same problem as 6u19.

Apparently there is a memory leak:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1073741832 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:215), pid=3248, tid=2335914912
# Error: Chunk::new
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode linux-x86 )

For more details see attachment.

When changing Xmx from 512m to 1024m, Geonetwork stays up and running a few minutes longer, but still crashes after a while.

Did someone notice this behavior already?

Kind regards
Rik Hendrix

Uniting expertise from different fields of technology in a pleasant and stimulating environment enhances innovation and the development of novel methods for sustainable production.
Have a look at the international congress 'Innovation for Sustainable Production 2010'
April, 18-21, 2010 - Bruges (Belgium)

---
This e-mail, any attachments and the information it contains are confidential and meant only for the use of the addressee(s) only. Access to this e-mail by anyone other than the addressee(s) is unauthorized. If you are not the intended addressee (or responsible for delivery of the message to such person), you may not use, copy, distribute or deliver to anyone this message (or any part of its contents) or take any action in reliance on it. In such case, you should destroy this message and notify the sender immediately. If you have received this e-mail in error, please notify us immediately by e-mail or telephone and delete the e-mail from any computer.
All reasonable precautions have been taken to ensure no viruses are present in this e-mail and its attachments. As our company cannot accept responsibility for any loss or damage arising from the use of this e-mail or attachments we recommend that you subject these to your virus checking procedures prior to use.

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork