[Geoserver-devel] OOM with big image

Andrea_Aime3 · January 15, 2007, 2:58pm

Got the BlueMarble images from geotorrents http://www.geotorrent.org/details.php?id=61, then extracted a big single tile geotiff image
using gdal_transalte:
gdal_translate -of GTiff -projwin -140 85 -40 0 world-topo-bathy-200408-3x86400x43200.ecw na_2004_08.tiff

The result is a geotiff of 1.4GB, 24000x20000 pixels.
I've then tried to configure it in geoserver, but got an OOM when creating the coverage (creating the coverage set worked fine):

java.lang.OutOfMemoryError: Java heap space
  at java.lang.AbstractStringBuilder.(AbstractStringBuilder.java:45)
  at java.lang.StringBuffer.(StringBuffer.java:79)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFFieldNode.initialize(TIFFFieldNode.java:165)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFFieldNode.getFirstChild(TIFFFieldNode.java:210)
  at com.sun.media.imageio.plugins.tiff.TIFFField.createFromMetadataNode(TIFFField.java:470)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.parseIFD(TIFFImageMetadata.java:1576)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.mergeNativeTree(TIFFImageMetadata.java:1600)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.mergeTree(TIFFImageMetadata.java:1624)
  at javax.imageio.metadata.IIOMetadata.setFromTree(IIOMetadata.java:742)
  at com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.getImageMetadata(TIFFImageReader.java:833)
  at org.geotools.gce.geotiff.GeoTiffReader.getHRInfo(GeoTiffReader.java:256)
  at org.geotools.gce.geotiff.GeoTiffReader.(GeoTiffReader.java:201)
  at org.geotools.gce.geotiff.GeoTiffFormat.getReader(GeoTiffFormat.java:252)
  at org.geotools.gce.geotiff.GeoTiffFormat.getReader(GeoTiffFormat.java:212)
  at
... (cut here, the relevant part is above anyways)

Man, what in the coverage _metadata_ can be so big to occupy the standard 64MB heap space? Setting the heap to 128m solved the problem.
Now, I know when serving big images a bigger heap is suggested, yet
I'm wondering, may it be that the heap was already full with the tile
cache?

The JAI parameters configuration in the Geoserver setting suggest the
default value is 209715200, which I suppose is something around 200MB,
right?
I would suggest to have a setting that is proportial to the max heap
size instead, that is, use something like 50%, so that If I start geoserver with 64mb the tile cache won't eat all the heap memory.
That is, assuming the tile cache can do that, that is, allocate
X megabytes and keep them no matter what causing an OOM.

Cheers
Andrea

Simone3 · January 15, 2007, 3:25pm

Ciao Andrea,
please,read below....

On 1/15/07, Andrea Aime <aaime@anonymised.com> wrote:

Got the BlueMarble images from geotorrents
http://www.geotorrent.org/details.php?id=61, then extracted a big single
tile geotiff image
using gdal_transalte:
gdal_translate -of GTiff -projwin -140 85 -40 0
world-topo-bathy-200408-3x86400x43200.ecw na_2004_08.tiff

The result is a geotiff of 1.4GB, 24000x20000 pixels.
I've then tried to configure it in geoserver, but got an OOM when
creating the coverage (creating the coverage set worked fine):

java.lang.OutOfMemoryError: Java heap space
       at java.lang.AbstractStringBuilder.(AbstractStringBuilder.java:45)
       at java.lang.StringBuffer.(StringBuffer.java:79)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFFieldNode.initialize(TIFFFieldNode.java:165)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFFieldNode.getFirstChild(TIFFFieldNode.java:210)
       at
com.sun.media.imageio.plugins.tiff.TIFFField.createFromMetadataNode(TIFFField.java:470)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.parseIFD(TIFFImageMetadata.java:1576)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.mergeNativeTree(TIFFImageMetadata.java:1600)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFImageMetadata.mergeTree(TIFFImageMetadata.java:1624)
       at javax.imageio.metadata.IIOMetadata.setFromTree(IIOMetadata.java:742)
       at
com.sun.media.imageioimpl.plugins.tiff.TIFFImageReader.getImageMetadata(TIFFImageReader.java:833)
       at org.geotools.gce.geotiff.GeoTiffReader.getHRInfo(GeoTiffReader.java:256)
       at org.geotools.gce.geotiff.GeoTiffReader.(GeoTiffReader.java:201)
       at org.geotools.gce.geotiff.GeoTiffFormat.getReader(GeoTiffFormat.java:252)
       at org.geotools.gce.geotiff.GeoTiffFormat.getReader(GeoTiffFormat.java:212)
       at
... (cut here, the relevant part is above anyways)

Man, what in the coverage _metadata_ can be so big to occupy the
standard 64MB heap space? Setting the heap to 128m solved the problem.
Now, I know when serving big images a bigger heap is suggested, yet
I'm wondering, may it be that the heap was already full with the tile
cache?

Well, in theory it could be, in practice, I do not think so or least
this is a mixed effect.

Metadata in ImageIO are pretty big as ou noticed, I would say too big
in some cases. I can give you an explanation much probably, having
seen how you created this geotiff, you geotiff is a planar striped
tiff which means it come in stripes which usually are large
image_width*(8 rows) . You are going to have a lot of this stripes and
all of them are going to be described using the metadata. Having this
info plus all the metadata plus some tiles already in cache plus a
small amount of memory availaible means OOM errors.

As an aside, when create a tiff always do that by using -co
"TILED=YES" option in gdal-translate.

The JAI parameters configuration in the Geoserver setting suggest the
default value is 209715200, which I suppose is something around 200MB,
right?

Yeah.

I would suggest to have a setting that is proportial to the max heap
size instead, that is, use something like 50%, so that If I start
geoserver with 64mb the tile cache won't eat all the heap memory.
That is, assuming the tile cache can do that, that is, allocate
X megabytes and keep them no matter what causing an OOM.

The GridCoverage DefaulProcessor already does kind of this. If we gave
JAI too much tile memory it lowers it accordingly to heap memory
(thanks to Martin ). However I think your idea might work quite
well, we could borrow some code from the DefaultProcessor and
implement your idea.

Ciao,
Simone.

Andrea_Aime3 · January 15, 2007, 3:44pm

Simone Giannecchini ha scritto:

Ciao Andrea,
please,read below....

Man, what in the coverage _metadata_ can be so big to occupy the
standard 64MB heap space? Setting the heap to 128m solved the problem.
Now, I know when serving big images a bigger heap is suggested, yet
I'm wondering, may it be that the heap was already full with the tile
cache?

Well, in theory it could be, in practice, I do not think so or least
this is a mixed effect.

Metadata in ImageIO are pretty big as ou noticed, I would say too big
in some cases. I can give you an explanation much probably, having
seen how you created this geotiff, you geotiff is a planar striped
tiff which means it come in stripes which usually are large
image_width*(8 rows) . You are going to have a lot of this stripes and
all of them are going to be described using the metadata. Having this
info plus all the metadata plus some tiles already in cache plus a
small amount of memory availaible means OOM errors.

Ah, I see. Did not know about the stripes metadata issue, interesting.

As an aside, when create a tiff always do that by using -co
"TILED=YES" option in gdal-translate.

Ha, good hint for the user docs in fact.
I've embedded overviews in the big image and now performance has improved, but it's still non instant
Man, I had to use a 1.4GB image to notice non-instant response times
with overvies, a 120MB one was not big enough. Granted, my PC is very
powerful and RAID 0 disks do help, but anyways, performance work
on big images seems to be really bearing fruits here
Congratulations

Now on to adding tiles as well :-p

I would suggest to have a setting that is proportial to the max heap
size instead, that is, use something like 50%, so that If I start
geoserver with 64mb the tile cache won't eat all the heap memory.
That is, assuming the tile cache can do that, that is, allocate
X megabytes and keep them no matter what causing an OOM.

The GridCoverage DefaulProcessor already does kind of this. If we gave
JAI too much tile memory it lowers it accordingly to heap memory
(thanks to Martin ). However I think your idea might work quite
well, we could borrow some code from the DefaultProcessor and
implement your idea.

Opened a Jira issue about this:
http://jira.codehaus.org/browse/GEOS-852

Cheers
Andrea