[Geoserver-users] JVM goes up in smoke

[cmaul]

Hello,

Geoserver is 2.25
Apache 2.2.21
Tomcat 7.0.27
GDAL 1.9.2
ECW SDK 4.3
Visual C++ 2010 redistributables.

Windows2008R2 virtual machine with 32GB RAM.

When cutting tiles the Java Vm disappears without a trace after happily
working between 4 - 15 minutes.
There is nothing suspicious in the geoserver log, tomcat.log, apache or the
event viewer.

The screenshot of the visualvm shows a busy server but also a large portion
of the heap unused.
I wouldn't call this a server that is stressed.

Java VisualVM screenshot
<http://services.land.vic.gov.au/memory_20130318.jpg&gt;

Because the tiles are cut from ecw there are a lot of components involved.
Where could I find an indicator of what it is that's making the JVM
collapse?

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context: http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

[Andrea_Aime4]

On Mon, Mar 18, 2013 at 7:34 AM, cmaul <Christian.Maul@anonymised.com> wrote:

Because the tiles are cut from ecw there are a lot of components involved.
Where could I find an indicator of what it is that’s making the JVM
collapse?

Welcome to the wondering world of native code integration… a minor glitch
in GDAL or the ECW is enough to take down the entire JVM, there is no
protection against SEGFAULT errors in native code.

That said, I’m not sure how to help… maybe it’s the specific ECW version,
I believe GDAL is supposed to be built against version 3.x… but I also may
be quite off the mark.

On the bright side GDAL is working on a plan to have these readers work
in a separate process, in order to proctect itself from such catastrophic failures.
But I don’t know if/when that will be available, you might want to check on the
GDAL mailing list.

Cheers
Andre

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

[Brad_Bode]

I have the same exact issue, which I can duplicate every time. I am working on posting the information I have gained. But the short answer is that I am using Windows R2 on a VM, tomcat, apache, and GeoServer.

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: cmaul Christian.Maul@anonymised.com
To: geoserver-users@lists.sourceforge.net
Sent: Sunday, March 17, 2013 11:34 PM
Subject: [Geoserver-users] JVM goes up in smoke

Hello,

Geoserver is 2.25
Apache 2.2.21
Tomcat 7.0.27
GDAL 1.9.2
ECW SDK 4.3
Visual C++ 2010 redistributables.

Windows2008R2 virtual machine with 32GB RAM.

When cutting tiles the Java Vm disappears without a trace after happily
working between 4 - 15 minutes.
There is nothing suspicious in the geoserver log, tomcat.log, apache or the
event viewer.

The screenshot of the visualvm shows a busy server but also a large portion
of the heap unused.
I wouldn’t call this a server that is stressed.

Java VisualVM screenshot
http://services.land.vic.gov.au/memory_20130318.jpg

Because the tiles are cut from ecw there are a lot of components involved.
Where could I find an indicator of what it is that’s making the JVM
collapse?

Cheers

Christian

---

---

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813

View this message in context: http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

---

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Simone_Giannecchini3]

Dear All,
thanks for reporting these issues. We are running some additional
stress test to try and reproduce this issue in
order to fix it.

We'll keep you posted.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Mon, Mar 18, 2013 at 4:17 PM, Brad Bode <cabaal@anonymised.com> wrote:

I have the same exact issue, which I can duplicate every time. I am working
on posting the information I have gained. But the short answer is that I am
using Windows R2 on a VM, tomcat, apache, and GeoServer.

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: cmaul <Christian.Maul@anonymised.com>
To: geoserver-users@lists.sourceforge.net
Sent: Sunday, March 17, 2013 11:34 PM
Subject: [Geoserver-users] JVM goes up in smoke

Hello,

Geoserver is 2.25
Apache 2.2.21
Tomcat 7.0.27
GDAL 1.9.2
ECW SDK 4.3
Visual C++ 2010 redistributables.

Windows2008R2 virtual machine with 32GB RAM.

When cutting tiles the Java Vm disappears without a trace after happily
working between 4 - 15 minutes.
There is nothing suspicious in the geoserver log, tomcat.log, apache or the
event viewer.

The screenshot of the visualvm shows a busy server but also a large portion
of the heap unused.
I wouldn't call this a server that is stressed.

Java VisualVM screenshot
<http://services.land.vic.gov.au/memory_20130318.jpg&gt;

Because the tiles are cut from ecw there are a lot of components involved.
Where could I find an indicator of what it is that's making the JVM
collapse?

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[cmaul]

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context: http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

[Simone_Giannecchini3]

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what's going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com> wrote:

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context: http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[rouault]

Le lundi 18 mars 2013 10:26:44, Andrea Aime a écrit :

On Mon, Mar 18, 2013 at 7:34 AM, cmaul <Christian.Maul@anonymised.com>wrote:
> Because the tiles are cut from ecw there are a lot of components
> involved. Where could I find an indicator of what it is that's making
> the JVM collapse?

The ECW SDK can consume a lot of RAM (and there was a bug in old SDK 3.3 where
in some configurations, the 1/4 RAM threshold was not respected). You should
look at the ECW_CACHE_MAXMEM configuration option / environement variable
documented in http://gdal.org/frmt_ecw.html

Welcome to the wondering world of native code integration... a minor glitch
in GDAL or the ECW is enough to take down the entire JVM, there is no
protection against SEGFAULT errors in native code.

That said, I'm not sure how to help... maybe it's the specific ECW version,
I believe GDAL is supposed to be built against version 3.x... but I also
may be quite off the mark.

On the bright side GDAL is working on a plan to have these readers work
in a separate process, in order to proctect itself from such catastrophic
failures.
But I don't know if/when that will be available, you might want to check on
the
GDAL mailing list.

GDAL 1.10 to be released in a couple of weeks :
http://gdal.org/gdal_api_proxy.html

Cheers
Andre

[Brad_Bode]

I mentioned that I was able to crash the JVM as well. I’ve been away for a bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One is zooming in and out using the mousewheel within our map (which is calling WMS). The other is simple jumping to our map page, then navigating away, then returning. We do this repeatedly until the JVM crashes within 5 minutes.

Notes

• Our dataset is very a very small shapefile of 1.5 mb.
• We are using Spring MVC as our front end and embedding a map into our page using Open Layers to call WMS in Geoserver.
• Attached you will find a zip of the logs. Stderr, stout, and a the JVM error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file. For starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image …
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say “done” as it does in other logs.

Comparing that to the closest timeframe in the stderr:

SEVERE: Servlet.service() for servlet [DataForge2] in context with path [/DataForge] threw exception [Request processing failed; nested exception is org.apache.tiles.impl.CannotRenderException: ServletException including path ‘/WEB-INF/layouts/default.jspx’.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I’m looking into the problem of how this is happening. The response was likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864, tid=1180

JRE version: 6.0_38-b05

Java VM: Java HotSpot™ Server VM (20.13-b02 mixed mode windows-x86 )

Problematic frame:

C [tcnative-1.dll+0x4bdf]

An error report file with more information is saved as:

C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

I assume this is from the previous failure since it occurs at the earliest point in the file.

What I don’t know
This seems to happen only when I use GeoServer WMS. However, I am doing more testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini simone.giannecchini@anonymised.com
To: cmaul <Christian.Maul@anonymised.com.>
Cc: geoserver-users@anonymised.comists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what’s going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com> wrote:

Simone,

I am a little bit further now. The ecws that didn’t work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I’ll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

---

---

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813

View this message in context: http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

---

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

---

Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel’s independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

dataforgeweb-stderr.2013-04-09.zip (945 KB)

[Simone_Giannecchini3]

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that's been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode <cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I've been away for a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes
- Our dataset is very a very small shapefile of 1.5 mb.
- We are using Spring MVC as our front end and embedding a map into our page
using Open Layers to call WMS in Geoserver.
- Attached you will find a zip of the logs. Stderr, stout, and a the JVM
error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file. For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image ...
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say "done" as it does in other logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception is
org.apache.tiles.impl.CannotRenderException: ServletException including path
'/WEB-INF/layouts/default.jspx'.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at
org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I'm looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,
tid=1180
#
# JRE version: 6.0_38-b05
# Java VM: Java HotSpot(TM) Server VM (20.13-b02 mixed mode windows-x86 )
# Problematic frame:
# C [tcnative-1.dll+0x4bdf]
#
# An error report file with more information is saved as:
# C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I assume this is from the previous failure since it occurs at the earliest
point in the file.

What I don't know
This seems to happen only when I use GeoServer WMS. However, I am doing more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: cmaul <Christian.Maul@anonymised.com>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what's going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Andrea_Aime4]

On Wed, Apr 10, 2013 at 10:16 AM, Simone Giannecchini <
simone.giannecchini@anonymised.com> wrote:

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that's been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Indeed it's not possible, if it was, we would leverage it to stop processing
the WMS requests as soon as the client dropped the connection.

That said, maybe there are bits of code that are first calling close on the
output
stream, and them something else calling flush on it.

That said, I'm appalled that Tomcat developers can close an issue that
brings the JVM up in smoke
so casually, calling close and flush in sequence on any output stream
should result at most in
an IOException, not in a JVM crash

Cheers
Andrea

--

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Ben_Caradoc-Davies]

Nice find, Simone. Please re-open/resubmit that tomcat issue as pestering the developers is the only way it will get fixed! A link to this thread (including Andrea's rather pointed synopsis) should give them encouragement. The report in this thread is a much clearer description than the original report.

Kind regards,
Ben.

On 10/04/13 16:16, Simone Giannecchini wrote:

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that's been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th& 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode<cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I've been away for a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes
- Our dataset is very a very small shapefile of 1.5 mb.
- We are using Spring MVC as our front end and embedding a map into our page
using Open Layers to call WMS in Geoserver.
- Attached you will find a zip of the logs. Stderr, stout, and a the JVM
error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file. For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image ...
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say "done" as it does in other logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception is
org.apache.tiles.impl.CannotRenderException: ServletException including path
'/WEB-INF/layouts/default.jspx'.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at
org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I'm looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,
tid=1180
#
# JRE version: 6.0_38-b05
# Java VM: Java HotSpot(TM) Server VM (20.13-b02 mixed mode windows-x86 )
# Problematic frame:
# C [tcnative-1.dll+0x4bdf]
#
# An error report file with more information is saved as:
# C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I assume this is from the previous failure since it occurs at the earliest
point in the file.

What I don't know
This seems to happen only when I use GeoServer WMS. However, I am doing more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini<simone.giannecchini@anonymised.com>
To: cmaul<Christian.Maul@anonymised.com>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what's going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 19, 2013 at 11:59 PM, cmaul<Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis& visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer
CSIRO Earth Science and Resource Engineering
Australian Resources Research Centre

[Brad_Bode]

Thank you for the quick response. I did some cursory searching and could not find answers to the following:

1. What is the purpose of Tomcat Native? It seems it is intended to make the server faster and more compatible with other server tech. If that’s so, removing native support will make my server slower, which isn’t desirable. Since GeoServer is a must for our application I suppose I will have to. Luckily we don’t have many concurrent users to worry about.

2. How do you remove tomcat native? Is it as simple as removing the “tcnative-1.dll” file?

I appreciate the help. I realize this isn’t your problem. And as Andrea pointed out, I am amazed that this issue has not received more notice. If something as simple as a connection closing prematurely can crash the entire VM, then it should receive major attention.

Is there anything I can do to help?

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini simone.giannecchini@anonymised.com
To: Brad Bode <cabaal@…54…>
Cc: “geoserver-users@lists.sourceforge.net” geoserver-users@anonymised.comt
Sent: Wednesday, April 10, 2013 1:16 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that’s been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode <cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I’ve been away for a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes

• Our dataset is very a very small shapefile of 1.5 mb.
• We are using Spring MVC as our front end and embedding a map into our page
  using Open Layers to call WMS in Geoserver.
• Attached you will find a zip of the logs. Stderr, stout, and a the JVM
  error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file. For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image …
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say “done” as it does in other logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception is
org.apache.tiles.impl.CannotRenderException: ServletException including path
‘/WEB-INF/layouts/default.jspx’.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at
org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at
javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I’m looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,

tid=1180

JRE version: 6.0_38-b05

Java VM: Java HotSpot™ Server VM (20.13-b02 mixed mode windows-x86 )

Problematic frame:

C [tcnative-1.dll+0x4bdf]

An error report file with more information is saved as:

C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

I assume this is from the previous failure since it occurs at the earliest
point in the file.

What I don’t know
This seems to happen only when I use GeoServer WMS. However, I am doing more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: cmaul <Christian.Maul@anonymised.com>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what’s going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn’t work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I’ll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

---

---

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813

View this message in context:
http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

---

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

---

Geoserver-users mailing list
Geoserver-users@anonymised.coms.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

---

Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel’s independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2

---

Geoserver-users mailing list
Geoserver-users@anonymised.coms.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Simone_Giannecchini3]

Ciao Brad,
please, read below...

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 7:00 PM, Brad Bode <cabaal@anonymised.com> wrote:

Thank you for the quick response. I did some cursory searching and could
not find answers to the following:

1) What is the purpose of Tomcat Native? It seems it is intended to make the
server faster and more compatible with other server tech. If that's so,
removing native support will make my server slower, which isn't desirable.
Since GeoServer is a must for our application I suppose I will have to.
Luckily we don't have many concurrent users to worry about.

It should speed up tomcat handling of output stream. In my experience
it does not make a noticeable difference.
Others may have different opinions, as such I would like to hear them.

2) How do you remove tomcat native? Is it as simple as removing the
"tcnative-1.dll" file?

It should be.
Check the doc (here as an instance for more info
http://tomcat.apache.org/tomcat-7.0-doc/apr.html)

I appreciate the help. I realize this isn't your problem. And as Andrea
pointed out, I am amazed that this issue has not received more notice. If
something as simple as a connection closing prematurely can crash the entire
VM, then it should receive major attention.

Well, this has been around for a while. Believe there are very
disappointing bugs around, I guess one does what one can

Is there anything I can do to help?

As Andrea pointed out we could try to spend even more time to check
if/how we can work this around. He already tried in the past,
and I tried myself too. I guess the best thing to do would be fixing
the apr code directly (I did not check if it is available or not).

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: Brad Bode <cabaal@anonymised.com>
Cc: "geoserver-users@lists.sourceforge.net"
<geoserver-users@lists.sourceforge.net>
Sent: Wednesday, April 10, 2013 1:16 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that's been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode <cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I've been away for a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One
is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes
- Our dataset is very a very small shapefile of 1.5 mb.
- We are using Spring MVC as our front end and embedding a map into our
page
using Open Layers to call WMS in Geoserver.
- Attached you will find a zip of the logs. Stderr, stout, and a the JVM
error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file.
For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image ...
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say "done" as it does in other
logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception
is
org.apache.tiles.impl.CannotRenderException: ServletException including
path
'/WEB-INF/layouts/default.jspx'.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at

org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at

javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I'm looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,
tid=1180
#
# JRE version: 6.0_38-b05
# Java VM: Java HotSpot(TM) Server VM (20.13-b02 mixed mode windows-x86 )
# Problematic frame:
# C [tcnative-1.dll+0x4bdf]
#
# An error report file with more information is saved as:
# C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I assume this is from the previous failure since it occurs at the earliest
point in the file.

What I don't know
This seems to happen only when I use GeoServer WMS. However, I am doing
more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: cmaul <Christian.Maul@anonymised.com>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what's going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which
went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context:

http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Brad_Bode]

Per your requests I removed the Tomcat Native DLL and observed the following:

1. Most important, I could not get it to crash using the same repeatable process as before
2. It did not appear significantly or functionally slower.

Thank you for your help. Though I’m a bit disappointed that Tomcat has this issue, I am nonetheless happy to have a work around that meets my needs.

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini <simone.giannecchini@anonymised.com.>
To: Brad Bode cabaal@anonymised.com
Cc: “geoserver-users@lists.sourceforge.net” geoserver-users@anonymised.comeforge.net
Sent: Wednesday, April 10, 2013 10:47 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
please, read below…

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Apr 10, 2013 at 7:00 PM, Brad Bode <cabaal@anonymised.com> wrote:

Thank you for the quick response. I did some cursory searching and could
not find answers to the following:

1. What is the purpose of Tomcat Native? It seems it is intended to make the
   server faster and more compatible with other server tech. If that’s so,
   removing native support will make my server slower, which isn’t desirable.
   Since GeoServer is a must for our application I suppose I will have to.
   Luckily we don’t have many concurrent users to worry about.

It should speed up tomcat handling of output stream. In my experience
it does not make a noticeable difference.
Others may have different opinions, as such I would like to hear them.

2. How do you remove tomcat native? Is it as simple as removing the
   “tcnative-1.dll” file?

It should be.
Check the doc (here as an instance for more info
http://tomcat.apache.org/tomcat-7.0-doc/apr.html)

I appreciate the help. I realize this isn’t your problem. And as Andrea
pointed out, I am amazed that this issue has not received more notice. If
something as simple as a connection closing prematurely can crash the entire
VM, then it should receive major attention.

Well, this has been around for a while. Believe there are very
disappointing bugs around, I guess one does what one can

Is there anything I can do to help?

As Andrea pointed out we could try to spend even more time to check
if/how we can work this around. He already tried in the past,
and I tried myself too. I guess the best thing to do would be fixing
the apr code directly (I did not check if it is available or not).

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: Brad Bode <cabaal@anonymised.com>
Cc: “geoserver-users@lists.sourceforge.net”
<geoserver-users@lists.sourceforge.net>
Sent: Wednesday, April 10, 2013 1:16 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that’s been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode <cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I’ve been away for a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One
is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes

• Our dataset is very a very small shapefile of 1.5 mb.
• We are using Spring MVC as our front end and embedding a map into our
  page
  using Open Layers to call WMS in Geoserver.
• Attached you will find a zip of the logs. Stderr, stout, and a the JVM
  error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file.
For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image …
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say “done” as it does in other
logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception
is
org.apache.tiles.impl.CannotRenderException: ServletException including
path
‘/WEB-INF/layouts/default.jspx’.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at

org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at

javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I’m looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,

tid=1180

JRE version: 6.0_38-b05

Java VM: Java HotSpot™ Server VM (20.13-b02 mixed mode windows-x86 )

Problematic frame:

C [tcnative-1.dll+0x4bdf]

An error report file with more information is saved as:

C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

I assume this is from the previous failure since it occurs at the earliest
point in the file.

What I don’t know
This seems to happen only when I use GeoServer WMS. However, I am doing
more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

---

Brad A. Bode
Principal
Software Systems
Foundry Engineering

---

From: Simone Giannecchini <simone.giannecchini@anonymised.com…1107…>
To: cmaul <Christian.Maul@anonymised.com…4794…>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what’s going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn’t work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which
went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I’ll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff on
our FTP server. The 11 files in question are between 8 and 14 GB and have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

---

---

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813

View this message in context:

http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

---

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

---

Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel’s independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2

---

Geoserver-users mailing list
Geoserver-users@anonymised.comsourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Simone_Giannecchini3]

Ciao Brad,
please, read below...

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Thu, Apr 11, 2013 at 1:40 AM, Brad Bode <cabaal@anonymised.com> wrote:

Per your requests I removed the Tomcat Native DLL and observed the
following:

1) Most important, I could not get it to crash using the same repeatable
process as before

cool

2) It did not appear significantly or functionally slower.

this is expected

Thank you for your help. Though I'm a bit disappointed that Tomcat has this
issue, I am nonetheless happy to have a work around that meets my needs.

Life isn't perfect and software does not make an exception, unfortunately.
Let's put it this way, this defect is probably something we can live
with (at least for the moment )

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: Brad Bode <cabaal@anonymised.com>
Cc: "geoserver-users@lists.sourceforge.net"
<geoserver-users@lists.sourceforge.net>
Sent: Wednesday, April 10, 2013 10:47 AM

Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
please, read below...

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 7:00 PM, Brad Bode <cabaal@anonymised.com> wrote:

Thank you for the quick response. I did some cursory searching and could
not find answers to the following:

1) What is the purpose of Tomcat Native? It seems it is intended to make
the
server faster and more compatible with other server tech. If that's so,
removing native support will make my server slower, which isn't desirable.
Since GeoServer is a must for our application I suppose I will have to.
Luckily we don't have many concurrent users to worry about.

It should speed up tomcat handling of output stream. In my experience
it does not make a noticeable difference.
Others may have different opinions, as such I would like to hear them.

2) How do you remove tomcat native? Is it as simple as removing the
"tcnative-1.dll" file?

It should be.
Check the doc (here as an instance for more info
http://tomcat.apache.org/tomcat-7.0-doc/apr.html)

I appreciate the help. I realize this isn't your problem. And as Andrea
pointed out, I am amazed that this issue has not received more notice. If
something as simple as a connection closing prematurely can crash the
entire
VM, then it should receive major attention.

Well, this has been around for a while. Believe there are very
disappointing bugs around, I guess one does what one can

Is there anything I can do to help?

As Andrea pointed out we could try to spend even more time to check
if/how we can work this around. He already tried in the past,
and I tried myself too. I guess the best thing to do would be fixing
the apr code directly (I did not check if it is available or not).

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: Brad Bode <cabaal@anonymised.com>
Cc: "geoserver-users@lists.sourceforge.net"
<geoserver-users@lists.sourceforge.net>
Sent: Wednesday, April 10, 2013 1:16 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Ciao Brad,
thanks for getting back to us.
I believe this problem is generated by the tomcat native connector.
Can you confirm that by uninstalling it the problem goes away?

Looking at this link http://bit.ly/Zm91zN it looks like we are
flushing a stream that's been closed.
This happens frequently when talking to clients as they drop the
connection while we are sending an image. I will talk to andrea to see
if we can improve this but I kind of remember it would be almost
impossible to know whether or not a stream has been dropped by a
client.

Long story short, short term fix should be disabling tomcat native.

Let me know if this helps.

Regards,
Simone Giannecchini

GeoServer training in Milan, 6th & 7th June 2013! Visit
http://geoserver.geo-solutions.it for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Wed, Apr 10, 2013 at 1:04 AM, Brad Bode <cabaal@anonymised.com> wrote:

I mentioned that I was able to crash the JVM as well. I've been away for
a
bit, but after returning I was able to duplicate the issue consistently.

Duplicating the issue
I have two users issue http requests to our web page simultaneously. One
is
zooming in and out using the mousewheel within our map (which is calling
WMS). The other is simple jumping to our map page, then navigating away,
then returning. We do this repeatedly until the JVM crashes within 5
minutes.

Notes
- Our dataset is very a very small shapefile of 1.5 mb.
- We are using Spring MVC as our front end and embedding a map into our
page
using Open Layers to call WMS in Geoserver.
- Attached you will find a zip of the logs. Stderr, stout, and a the JVM
error mentioned in the top of the stdout log.

Log File Comparison
I think it may be relevant to look at the last actions in the log file.
For
starters, the stdout file has the following at the end:

09 Apr 15:24:22 DEBUG [wms.map] - Writing png image ...
09 Apr 15:24:22 DEBUG [geotools.image] - Encoded input image for png
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Getting a writer
09 Apr 15:24:22 DEBUG [geotools.image] - Setting write parameters for
this
writer
09 Apr 15:24:22 DEBUG [geotools.image] - Writer is native
09 Apr 15:24:22 DEBUG [geotools.image] - About to write png image

Note that the write.png.image does not say "done" as it does in other
logs.

Comparing that to the closest timeframe in the stderr:
SEVERE: Servlet.service() for servlet [DataForge2] in context with path
[/DataForge] threw exception [Request processing failed; nested exception
is
org.apache.tiles.impl.CannotRenderException: ServletException including
path
'/WEB-INF/layouts/default.jspx'.] with root cause
java.lang.IllegalStateException: getOutputStream() has already been
called
for this response
at org.apache.catalina.connector.Response.getWriter(Response.java:639)
at

org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)
at

javax.servlet.ServletResponseWrapper.getWriter(ServletResponseWrapper.java:105)

I'm looking into the problem of how this is happening. The response was
likely closed prematurely.

And finally, in the top of the stdout:

2013-04-09 15:21:33 Commons Daemon procrun stdout initialized
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x5acc4bdf, pid=2864,
tid=1180
#
# JRE version: 6.0_38-b05
# Java VM: Java HotSpot(TM) Server VM (20.13-b02 mixed mode windows-x86 )
# Problematic frame:
# C [tcnative-1.dll+0x4bdf]
#
# An error report file with more information is saved as:
# C:\Program Files (x86)\DataForge\webserver\bin\hs_err_pid2864.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I assume this is from the previous failure since it occurs at the
earliest
point in the file.

What I don't know
This seems to happen only when I use GeoServer WMS. However, I am doing
more
testing.

Do you have any suggestions as to how to proceed?

Do you see anything in the logs that looks suspicious to you?

Thank you

________________________________
Brad A. Bode
Principal
Software Systems
Foundry Engineering

________________________________
From: Simone Giannecchini <simone.giannecchini@anonymised.com>
To: cmaul <Christian.Maul@anonymised.com>
Cc: geoserver-users@lists.sourceforge.net
Sent: Friday, March 29, 2013 10:27 AM
Subject: Re: [Geoserver-users] JVM goes up in smoke

Dear Christian,
going back to this, I have run some (relatively basic) tests against
ecw using a 2gb dataset (which is by accident a raster covergin
australia :)).
I tested on windows with both 2.2.x and master and I was not able to
get any strange behavior.

Thinking about a way to get some hints on what's going on on your
installation, I guess one thing you could do would be reproduce the
problematic situation and then
troubleshoot GeoServer following this instructions
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html
to get a feeling about where the problem is.

Let us know if you make any progress.

Regards,
Simone Giannecchini

Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Tue, Mar 19, 2013 at 11:59 PM, cmaul <Christian.Maul@anonymised.com>
wrote:

Simone,

I am a little bit further now. The ecws that didn't work were actually a
layergroup of 11 images.
These images were in EPSG:28355 and reprojected to EPSG:990913.

I have done a test (4 threads/ 170000 images) to tile one image which
went
fine. Speed was 163 minutes or 17.2 tiles per second, which is o.k. as
well. The next thing I will try is to have a layergroup and NOT to
reproject, i.e. to tile in EPSG:28355. I'll tell you how that works (or
not).

Simone, if you want any ECW files for testing, I am happy to put stuff
on
our FTP server. The 11 files in question are between 8 and 14 GB and
have
a
total of 128GB. So, a subset of 4 or 5 neighbouring ecws perhaps.

Cheers

Christian

-----
____________________________

Dr Christian Maul
Project Manager

Information Services Branch
Department of Sustainability and Environment
Level13, Marland House, 570 Bourke Street
Melbourne 3000

PO Box 500, East Melbourne Vic 3002

Telephone: +61-3-8636 2325
Telefax: +61-3-8636 2813
--
View this message in context:

http://osgeo-org.1560.n6.nabble.com/JVM-goes-up-in-smoke-tp5041006p5041499.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete
for recognition, cash, and the chance to get your game on Steam.
$5K grand prize plus 10 genre and skill prizes. Submit your demo
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users