[Geoserver-users] Recent issue - java.lang.OutOfMemoryError: Java heap space

Hi List,
I’ve been getting a lot of OutOfMemoryError errors in my logs over the past few days, to the extent that all three instances of my live service died yesterday afternoon!

GeoServer 2.4.3
Windows Server 2008 R2 -64bit. - The machine has ~30GB RAM, but each tomcat instance never uses more than 1.5GB.
Tomcat 6

Environmental variables for each instance:

set JAVA_OPTS= -Xmx5G -Xms2G -XX:MaxPermSize=256m
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4

Example of the error:

2014-01-29 12:12:34,082 ERROR [geotools.rendering] - Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferByte.(Unknown Source)
at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown Source)
at sun.awt.image.BufferedImageGraphicsConfig.createCompatibleImage(Unknown Source)
at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source)
at org.geotools.renderer.lite.DelayedBackbufferGraphic.init(DelayedBackbufferGraphic.java:71)
at org.geotools.renderer.lite.StreamingRenderer$PaintShapeRequest.execute(StreamingRenderer.java:3445)
at org.geotools.renderer.lite.StreamingRenderer$PainterThread.run(StreamingRenderer.java:3657)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-01-29 12:12:34,082 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Rendering process failed
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:507)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:251)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:123)

This hasn’t happened before. The only change I’ve made to the systems recently was:

  • Increase all Oracle stores “Fetch size” from 1000 to 10,000
  • Increase all Oracle stores “min connections” from 5 to 20.

There are two Oracle stores.

A complete set of logs is here - http://maps.warwickshire.gov.uk/shr/logs.zip

Anyone have any ideas why?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

On Wed, Jan 29, 2014 at 3:54 PM, Jonathan Moules <
jonathanmoules@anonymised.com> wrote:

Hi List,
I've been getting a lot of OutOfMemoryError errors in my logs over the
past few days, to the extent that all three instances of my live service
died yesterday afternoon!

GeoServer 2.4.3
Windows Server 2008 R2 -64bit. - The machine has ~30GB RAM, but each
tomcat instance never uses more than 1.5GB.
Tomcat 6

Environmental variables for each instance:

set JAVA_OPTS= -Xmx5G -Xms2G -XX:MaxPermSize=256m
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:ParallelGCThreads=4

Example of the error:

2014-01-29 12:12:34,082 ERROR [geotools.rendering] - Java heap space

java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferByte.<init>(Unknown Source)
at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.<init>(Unknown Source)
at
sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown
Source)
at
sun.awt.image.BufferedImageGraphicsConfig.createCompatibleImage(Unknown
Source)
at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source)
at
org.geotools.renderer.lite.DelayedBackbufferGraphic.init(DelayedBackbufferGraphic.java:71)
at
org.geotools.renderer.lite.StreamingRenderer$PaintShapeRequest.execute(StreamingRenderer.java:3445)
at
org.geotools.renderer.lite.StreamingRenderer$PainterThread.run(StreamingRenderer.java:3657)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-01-29 12:12:34,082 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Rendering process failed
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:507)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:251)
at
org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:123)

This hasn't happened before. The only change I've made to the systems
recently was:
- Increase all Oracle stores "Fetch size" from 1000 to 10,000

This parameter increases 10 times the memory used to keep the records
retrieved from the database
(but reduces by the same factor the number of round trips).

If your instances were already close to being OOM, this might as well have
been the drop that made the
bucket overflow.

See also
http://docs.geoserver.org/latest/en/user/production/troubleshooting.html#jmap
for tools to analyze OOM situations

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Hi Andrea,
After installing three versions of the JDK (apparently it needs to be the exact same as the JRE or you get lots of errors), I’ve got the following:

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1073741824 (1024.0MB)
NewSize = 21757952 (20.75MB)
MaxNewSize = 87228416 (83.1875MB)
OldSize = 65404928 (62.375MB)
NewRatio = 7
SurvivorRatio = 8
PermSize = 134217728 (128.0MB)
MaxPermSize = 536870912 (512.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 73110712 (69.72380828857422MB)
free = 5401416 (5.151191711425781MB)
93.12027818173519% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 66148256 (63.083892822265625MB)
free = 3647584 (3.478607177734375MB)
94.77392348884976% used
From Space:
capacity = 8716288 (8.3125MB)
used = 6962456 (6.639915466308594MB)
free = 1753832 (1.6725845336914062MB)
79.87868230145676% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 830111392 (791.6559143066406MB)
free = 156402016 (149.15658569335938MB)
84.14598172395037% used
Perm Generation:
capacity = 185602048 (177.00390625MB)
used = 110888984 (105.7519760131836MB)
free = 74713064 (71.2519302368164MB)
59.745560566228235% used

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

The second instance:

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 56468240 (53.85231018066406MB)
free = 22043888 (21.022689819335938MB)
71.92295182726419% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 56468240 (53.85231018066406MB)
free = 13327600 (12.710189819335938MB)
80.90487914465963% used
From Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 982866344 (937.3343887329102MB)
free = 3647064 (3.4781112670898438MB)
99.63030771093179% used
Perm Generation:
capacity = 180719616 (172.34765625MB)
used = 108201840 (103.18931579589844MB)
free = 72517776 (69.15834045410156MB)
59.87276998198137% used


Third instance:

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 65007392 (61.995880126953125MB)
free = 13504736 (12.879119873046875MB)
82.79917212280886% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 65007392 (61.995880126953125MB)
free = 4788448 (4.566619873046875MB)
93.13935042546949% used
From Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 974120624 (928.9938201904297MB)
free = 12392784 (11.818679809570312MB)
98.7437794661986% used
Perm Generation:
capacity = 186138624 (177.515625MB)
used = 111186864 (106.03605651855469MB)
free = 74951760 (71.47956848144531MB)
59.73336517196989% used

I’m not clear what any of that says though. The sections have different names to the ones on the manual page.

The -histo command one doesn’t work though:

“10736: Insufficient memory or insufficient privileges to attach
The -F option can be used when the target process is not responding”
(running it as admin so shouldn’t be any privilege issues).

10x more memory usage is fine - but how do I give GeoServer more access to memory? It’s not using much of the 5G I’m trying to give it currently.

Cheers,
Jonathan

On 29 January 2014 15:14, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Jan 29, 2014 at 3:54 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
I’ve been getting a lot of OutOfMemoryError errors in my logs over the past few days, to the extent that all three instances of my live service died yesterday afternoon!

GeoServer 2.4.3
Windows Server 2008 R2 -64bit. - The machine has ~30GB RAM, but each tomcat instance never uses more than 1.5GB.
Tomcat 6

Environmental variables for each instance:

set JAVA_OPTS= -Xmx5G -Xms2G -XX:MaxPermSize=256m
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4

Example of the error:

2014-01-29 12:12:34,082 ERROR [geotools.rendering] - Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferByte.(Unknown Source)
at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown Source)
at sun.awt.image.BufferedImageGraphicsConfig.createCompatibleImage(Unknown Source)
at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source)
at org.geotools.renderer.lite.DelayedBackbufferGraphic.init(DelayedBackbufferGraphic.java:71)
at org.geotools.renderer.lite.StreamingRenderer$PaintShapeRequest.execute(StreamingRenderer.java:3445)
at org.geotools.renderer.lite.StreamingRenderer$PainterThread.run(StreamingRenderer.java:3657)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-01-29 12:12:34,082 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Rendering process failed
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:507)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:251)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:123)

This hasn’t happened before. The only change I’ve made to the systems recently was:

  • Increase all Oracle stores “Fetch size” from 1000 to 10,000

This parameter increases 10 times the memory used to keep the records retrieved from the database
(but reduces by the same factor the number of round trips).

If your instances were already close to being OOM, this might as well have been the drop that made the
bucket overflow.

See also http://docs.geoserver.org/latest/en/user/production/troubleshooting.html#jmap
for tools to analyze OOM situations

Cheers
Andrea

== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Hi List,
Further to this, does anyone have any thoughts as to what jmap is reporting? As noted, I thought I was giving much more memory to the system than it seems its using.

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 29 January 2014 16:52, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Andrea,
After installing three versions of the JDK (apparently it needs to be the exact same as the JRE or you get lots of errors), I’ve got the following:

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1073741824 (1024.0MB)
NewSize = 21757952 (20.75MB)
MaxNewSize = 87228416 (83.1875MB)
OldSize = 65404928 (62.375MB)
NewRatio = 7
SurvivorRatio = 8
PermSize = 134217728 (128.0MB)
MaxPermSize = 536870912 (512.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 73110712 (69.72380828857422MB)
free = 5401416 (5.151191711425781MB)
93.12027818173519% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 66148256 (63.083892822265625MB)
free = 3647584 (3.478607177734375MB)
94.77392348884976% used
From Space:
capacity = 8716288 (8.3125MB)
used = 6962456 (6.639915466308594MB)
free = 1753832 (1.6725845336914062MB)
79.87868230145676% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 830111392 (791.6559143066406MB)
free = 156402016 (149.15658569335938MB)
84.14598172395037% used
Perm Generation:
capacity = 185602048 (177.00390625MB)
used = 110888984 (105.7519760131836MB)
free = 74713064 (71.2519302368164MB)
59.745560566228235% used


The second instance:

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 56468240 (53.85231018066406MB)
free = 22043888 (21.022689819335938MB)
71.92295182726419% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 56468240 (53.85231018066406MB)
free = 13327600 (12.710189819335938MB)
80.90487914465963% used
From Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 982866344 (937.3343887329102MB)
free = 3647064 (3.4781112670898438MB)
99.63030771093179% used
Perm Generation:
capacity = 180719616 (172.34765625MB)
used = 108201840 (103.18931579589844MB)
free = 72517776 (69.15834045410156MB)
59.87276998198137% used


Third instance:

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 78512128 (74.875MB)
used = 65007392 (61.995880126953125MB)
free = 13504736 (12.879119873046875MB)
82.79917212280886% used
Eden Space:
capacity = 69795840 (66.5625MB)
used = 65007392 (61.995880126953125MB)
free = 4788448 (4.566619873046875MB)
93.13935042546949% used
From Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
To Space:
capacity = 8716288 (8.3125MB)
used = 0 (0.0MB)
free = 8716288 (8.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 986513408 (940.8125MB)
used = 974120624 (928.9938201904297MB)
free = 12392784 (11.818679809570312MB)
98.7437794661986% used
Perm Generation:
capacity = 186138624 (177.515625MB)
used = 111186864 (106.03605651855469MB)
free = 74951760 (71.47956848144531MB)
59.73336517196989% used

I’m not clear what any of that says though. The sections have different names to the ones on the manual page.

The -histo command one doesn’t work though:

“10736: Insufficient memory or insufficient privileges to attach
The -F option can be used when the target process is not responding”
(running it as admin so shouldn’t be any privilege issues).

10x more memory usage is fine - but how do I give GeoServer more access to memory? It’s not using much of the 5G I’m trying to give it currently.

Cheers,
Jonathan

On 29 January 2014 15:14, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Jan 29, 2014 at 3:54 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
I’ve been getting a lot of OutOfMemoryError errors in my logs over the past few days, to the extent that all three instances of my live service died yesterday afternoon!

GeoServer 2.4.3
Windows Server 2008 R2 -64bit. - The machine has ~30GB RAM, but each tomcat instance never uses more than 1.5GB.
Tomcat 6

Environmental variables for each instance:

set JAVA_OPTS= -Xmx5G -Xms2G -XX:MaxPermSize=256m
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4

Example of the error:

2014-01-29 12:12:34,082 ERROR [geotools.rendering] - Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.awt.image.DataBufferByte.(Unknown Source)
at java.awt.image.ComponentSampleModel.createDataBuffer(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.(Unknown Source)
at sun.awt.image.ByteInterleavedRaster.createCompatibleWritableRaster(Unknown Source)
at sun.awt.image.BufferedImageGraphicsConfig.createCompatibleImage(Unknown Source)
at java.awt.GraphicsConfiguration.createCompatibleImage(Unknown Source)
at org.geotools.renderer.lite.DelayedBackbufferGraphic.init(DelayedBackbufferGraphic.java:71)
at org.geotools.renderer.lite.StreamingRenderer$PaintShapeRequest.execute(StreamingRenderer.java:3445)
at org.geotools.renderer.lite.StreamingRenderer$PainterThread.run(StreamingRenderer.java:3657)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-01-29 12:12:34,082 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Rendering process failed
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:507)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:251)
at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:123)

This hasn’t happened before. The only change I’ve made to the systems recently was:

  • Increase all Oracle stores “Fetch size” from 1000 to 10,000

This parameter increases 10 times the memory used to keep the records retrieved from the database
(but reduces by the same factor the number of round trips).

If your instances were already close to being OOM, this might as well have been the drop that made the
bucket overflow.

See also http://docs.geoserver.org/latest/en/user/production/troubleshooting.html#jmap
for tools to analyze OOM situations

Cheers
Andrea

== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it