[Geoserver-users] serving huge raster datasets with GeoServer

Pekka_Hurskainen · January 9, 2009, 9:49am

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
       e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
  TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=100.007484672
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.7884416006343e-299
    STATISTICS_STDDEV=34.047496023691
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=121.8462572288
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.8047999164279e-299
    STATISTICS_STDDEV=37.348685345109
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=135.154277504
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.8111374963437e-299
    STATISTICS_STDDEV=40.211793313468
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

Alessio_Fabiani · January 9, 2009, 10:46am

Hi Pekka,
GeoServer can handle ECW files but of course you need to buy a valid
license in order to use the ECW Decoder in a server fashion
architecture.

Anyway, imho you can still successfully use your GeoTIFF files with
good performances by adopting few expedients like:
1. Applying overviews to your GeoTIFFs
2. Tilize your GeoTIFFs
4. Serve the maps as GIF files and applying palettes if the number of
colors is limited or serve the maps as PNG-8

About the compressed vs. uncompressed questions, if you have a good
server and use the hints above you shouldn't notice particular leaks
of performances.

Mr.Sid are of course much slower than the other formats.

-------------------------------------------------------
Eng. Alessio Fabiani
Vice-President /CTO GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584 980933
fax: +39 0584 983027
mob: +39 349 8227000

http://www.geo-solutions.it

-------------------------------------------------------

On Fri, Jan 9, 2009 at 10:49 AM, Pekka Hurskainen
<pekka.hurskainen@anonymised.com> wrote:

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
      e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=100.007484672
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.7884416006343e-299
   STATISTICS_STDDEV=34.047496023691
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=121.8462572288
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8047999164279e-299
   STATISTICS_STDDEV=37.348685345109
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=135.154277504
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8111374963437e-299
   STATISTICS_STDDEV=40.211793313468
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Roger_Bedell · January 9, 2009, 4:13pm

Pekka,
We're doing something similar, and I've gotten reasonable results with GeoTIFF, Jpeg compressed, YCBCR, followed by creating overviews

Gdal_Translate -co compress=jpeg -co photometric=YCBCR -co tiled=yes infile outfile

then

GdalAddo -r gauss --config INTERLEAVE_OVERVIEW PIXEL --config COMPRESS_OVERVIEW JPEG filename 2 4 8 16 32 64

Then using the image mosaic in GeoServer.

Sizes approach that of the original mrSid, but way better performance.
If you look in the Gdal GeoTIFF docs, it mentions how YCBCR makes for better compression (2-3x) than normal RGB.

As far as performance, I've seen in the past that a fast processor doing a simple (jpeg) decompression can often outrun accessing an uncompressed image from disk.

Roger

--------------------------------------------------
From: "Pekka Hurskainen" <pekka.hurskainen@anonymised.com>
Sent: Friday, January 09, 2009 10:49 AM
To: <geoserver-users@lists.sourceforge.net>
Subject: [Geoserver-users] serving huge raster datasets with GeoServer

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
      e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=100.007484672
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.7884416006343e-299
   STATISTICS_STDDEV=34.047496023691
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=121.8462572288
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8047999164279e-299
   STATISTICS_STDDEV=37.348685345109
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=135.154277504
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8111374963437e-299
   STATISTICS_STDDEV=40.211793313468
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Arne_Kepp · January 10, 2009, 8:26am

As others already have pointed out, you should look into using cached tiles, probably stored as JPEGs so that you don't have to do any more work before sending them to the client. Assuming you can get your data into GeoServer, the only requirement is additional disk space.

GeoServer 1.7.1 includes GeoWebCache; Just configure your layer, restart GeoServer and go to http://localhost:8080/geoserver/gwc

You should find your layer in the list, using PNG by default. Just append &format=image/jpeg to the URL.

If you need projections other than EPSG:4326 and EPSG:900913 you can add them by creating a configuration file <GEOSERVER DATA DIR>/gwc/geowebcache.xml

-Arne

Pekka Hurskainen wrote:

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
       e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
  TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=100.007484672
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.7884416006343e-299
    STATISTICS_STDDEV=34.047496023691
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=121.8462572288
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.8047999164279e-299
    STATISTICS_STDDEV=37.348685345109
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
  Overviews: 1024x1024
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=135.154277504
    STATISTICS_MEDIAN=0
    STATISTICS_MODE=1.8111374963437e-299
    STATISTICS_STDDEV=40.211793313468
    LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers

Pekka_Hurskainen · January 19, 2009, 1:22pm

Thank you so much for your valuable advice and sorry for keeping quiet
so long. I have been quite busy trying to implement what you have
suggested and after a lot of set-backs I think I have finally managed
to get there.

What I have done so far:

1) Compressed the original GeoTIFF files into JPEG, YCM

On Sat, Jan 10, 2009 at 10:26 AM, Arne Kepp <ak@anonymised.com> wrote:

As others already have pointed out, you should look into using cached tiles,
probably stored as JPEGs so that you don't have to do any more work before
sending them to the client. Assuming you can get your data into GeoServer,
the only requirement is additional disk space.

GeoServer 1.7.1 includes GeoWebCache; Just configure your layer, restart
GeoServer and go to http://localhost:8080/geoserver/gwc

You should find your layer in the list, using PNG by default. Just append
&format=image/jpeg to the URL.

If you need projections other than EPSG:4326 and EPSG:900913 you can add
them by creating a configuration file <GEOSERVER DATA
DIR>/gwc/geowebcache.xml

-Arne

Pekka Hurskainen wrote:

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or
MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
      e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=100.007484672
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.7884416006343e-299
   STATISTICS_STDDEV=34.047496023691
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=121.8462572288
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8047999164279e-299
   STATISTICS_STDDEV=37.348685345109
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=135.154277504
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8111374963437e-299
   STATISTICS_STDDEV=40.211793313468
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers

--
Pekka Hurskainen
+358-44-3390018
pekka.hurskainen@anonymised.com

Pekka_Hurskainen · January 19, 2009, 1:50pm

Whoops, pressed the send button accidently too soon, sorry for that.
To continue where I left:

What I have done so far:

1) Compressed the original GeoTIFF files into JPEG, YCBCR tiles and
created the overlays as Roger suggested
2) Created a mosaic with the ImageMosaic plugin and loaded the data
into GeoServer as Roger suggested
3) Installed and configured GeoWebCache (.war file) as suggested by
Arne, which took a lot of time and trials & errors, and a lot of
studying
4) Installed native JAI libraries for JRE, since I got error messages
from Tomcat of not having it
5) Configured Apache Tomcat to deal with memory issues
(Java.lang.outofmemoryerror: Java Heap space) as instructed in
http://geoserver.org/display/GEOSDOC/2.6+GeoServer+in+Production+Environment

I used to get a lot of trouble with points 4 and 5 when trying to view
my mosaic through the gwc demo page in my web browser, with always the
same problem of some tiles at some scales not showing up at all and
some tiles getting all red and showing the
"Java.lang.outofmemoryerror: Java Heap space" error. I did everything
as instructed, but still got the same problems. Finally I realised I
must empty my web browser's cache, and after that everything worked
smoothly and fast!

My only question, at least for now, is can I get GeoWebCache on
GeoServer to work with ArcGIS 9.3 as a tiled WMS service? I could
connect to GeoServer and get my mosaic as a normal WMS layer, but when
I defined http://dbs:8080/geoserver/gwc/service/wms? to the server URL
in ArcCatalog, I can get a list of all my layers but nothing can be
seen either on the preview tab nor on ArcMap. So I guess this is not
implemented? It is not a big thing, since I can still use the normal
un-cached WMS service for ArcMap use, but getting GWC to work would be
better of course.

Thanks again for your help! I really appreciate how active and helpful
the GeoServer community is.

-Pekka

On Mon, Jan 19, 2009 at 3:22 PM, Pekka Hurskainen
<pekka.hurskainen@anonymised.com> wrote:

Thank you so much for your valuable advice and sorry for keeping quiet
so long. I have been quite busy trying to implement what you have
suggested and after a lot of set-backs I think I have finally managed
to get there.

What I have done so far:

1) Compressed the original GeoTIFF files into JPEG, YCM

On Sat, Jan 10, 2009 at 10:26 AM, Arne Kepp <ak@anonymised.com> wrote:

As others already have pointed out, you should look into using cached tiles,
probably stored as JPEGs so that you don't have to do any more work before
sending them to the client. Assuming you can get your data into GeoServer,
the only requirement is additional disk space.

GeoServer 1.7.1 includes GeoWebCache; Just configure your layer, restart
GeoServer and go to http://localhost:8080/geoserver/gwc

You should find your layer in the list, using PNG by default. Just append
&format=image/jpeg to the URL.

If you need projections other than EPSG:4326 and EPSG:900913 you can add
them by creating a configuration file <GEOSERVER DATA
DIR>/gwc/geowebcache.xml

-Arne

Pekka Hurskainen wrote:

Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or
MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
      e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=100.007484672
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.7884416006343e-299
   STATISTICS_STDDEV=34.047496023691
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=121.8462572288
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8047999164279e-299
   STATISTICS_STDDEV=37.348685345109
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=135.154277504
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8111374963437e-299
   STATISTICS_STDDEV=40.211793313468
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers

Arne_Kepp · January 19, 2009, 2:04pm

Sorry about that, you are right that current version of GeoWebCache / GeoServer will cache error messages too (it cannot tell them apart). We recently did some work on this, I must double check whether everything is in place to detect this.

I am guessing that ArcCatalog is using the getcapabilities document. With the integrated version of GeoServer this may actually work, thanks to a hack / patch by Jody Garnett. The freestanding version (WAR) of GWC is currently not able to provide a getcapabilities response.

If you can configure the layers individually in arccatalog then you should be able to get past this. I'm not sure exactly when we'll have a proper getcapabilties document, it would require users to provide a somewhat more extensive configuration to satisfy the WMS standard.

Anyway, glad it's smooth and fast at least

-Arne

Pekka Hurskainen wrote:

Whoops, pressed the send button accidently too soon, sorry for that.
To continue where I left:

What I have done so far:

1) Compressed the original GeoTIFF files into JPEG, YCBCR tiles and
created the overlays as Roger suggested
2) Created a mosaic with the ImageMosaic plugin and loaded the data
into GeoServer as Roger suggested
3) Installed and configured GeoWebCache (.war file) as suggested by
Arne, which took a lot of time and trials & errors, and a lot of
studying
4) Installed native JAI libraries for JRE, since I got error messages
from Tomcat of not having it
5) Configured Apache Tomcat to deal with memory issues
(Java.lang.outofmemoryerror: Java Heap space) as instructed in
http://geoserver.org/display/GEOSDOC/2.6+GeoServer+in+Production+Environment

I used to get a lot of trouble with points 4 and 5 when trying to view
my mosaic through the gwc demo page in my web browser, with always the
same problem of some tiles at some scales not showing up at all and
some tiles getting all red and showing the
"Java.lang.outofmemoryerror: Java Heap space" error. I did everything
as instructed, but still got the same problems. Finally I realised I
must empty my web browser's cache, and after that everything worked
smoothly and fast!

My only question, at least for now, is can I get GeoWebCache on
GeoServer to work with ArcGIS 9.3 as a tiled WMS service? I could
connect to GeoServer and get my mosaic as a normal WMS layer, but when
I defined http://dbs:8080/geoserver/gwc/service/wms? to the server URL
in ArcCatalog, I can get a list of all my layers but nothing can be
seen either on the preview tab nor on ArcMap. So I guess this is not
implemented? It is not a big thing, since I can still use the normal
un-cached WMS service for ArcMap use, but getting GWC to work would be
better of course.

Thanks again for your help! I really appreciate how active and helpful
the GeoServer community is.

-Pekka

On Mon, Jan 19, 2009 at 3:22 PM, Pekka Hurskainen
<pekka.hurskainen@anonymised.com> wrote:


Thank you so much for your valuable advice and sorry for keeping quiet
so long. I have been quite busy trying to implement what you have
suggested and after a lot of set-backs I think I have finally managed
to get there.

What I have done so far:

1) Compressed the original GeoTIFF files into JPEG, YCM

On Sat, Jan 10, 2009 at 10:26 AM, Arne Kepp <ak@anonymised.com> wrote:


As others already have pointed out, you should look into using cached tiles,
probably stored as JPEGs so that you don't have to do any more work before
sending them to the client. Assuming you can get your data into GeoServer,
the only requirement is additional disk space.

GeoServer 1.7.1 includes GeoWebCache; Just configure your layer, restart
GeoServer and go to http://localhost:8080/geoserver/gwc

You should find your layer in the list, using PNG by default. Just append
&format=image/jpeg to the URL.

If you need projections other than EPSG:4326 and EPSG:900913 you can add
them by creating a configuration file <GEOSERVER DATA
DIR>/gwc/geowebcache.xml

-Arne

Pekka Hurskainen wrote:


Dear friends,

I am new to GeoServer and just started to explore the coverage serving
capabilities, and would like to have your suggestions and ideas on how
to proceed. I have already read the posts in GeoServer User
documentation, mainly
http://geoserver.org/display/GEOSDOC/High+performance+coverage+serving
but I still have some questions, especially since that text was
written half a year ago.

We have currently 8.6 gb of aerial photography in GeoTIFF format,
which we want to put online with GeoServer. In total there are 77
individual mosaic tiles processed and mosaiced with OrthoVista
software, each having the same filesize of 118 mb. We have also the
same dataset in ECW format, while compressed the total filesize drops
to 248 mb but the visual quality is still acceptable for our purposes.

Our aim is to share these coverages as a background layer, and on top
of it will be overlaid vector layers: building footprints, road
network, and some administrative boundaries. The data will be
displayed with OpenLayers on custom-made clients, and even on web
browsers. The imagery will be not used for analysis, so even heavily
compressed imagery (ECW, JPEG2000) would suffice, but then there is
the question of how much time the decompressing of imagery will take
upon every request?

The GeoServer will we running in a closed Intranet, which is quite
slow Wireless network since we are based on a developing country. What
is needed is as fast display of the imagery as possible, so the ECW
would perhaps be the optimal choice. But I am not sure if it is
possible to serve ECW with GeoServer, even with the ImageIO-ext GDAL
extension?

Some general questions:
- Is it better to use compressed or decompressed imagery with these
kind of huge datasets?
- LZ77 compression vs uncompressed TIFF images
- Individual mosaic tiles vs. mosaicked single image
- Can GeoServer read pyramid layers (.rrd) made with ArcGIS or Erdas
Imagine, or do I have to create them with gdal?
- If ECW is not possible, any suggestions for other formats? JPG2000 or
MrSID?

Below is some info about our original, uncompressed GeoTIFF imagery:

C:\gdal\bin>gdalinfo e:\tif\520_9320.tif
Driver: GTiff/GeoTIFF
Files: e:\tif\520_9320.tif
      e:\tif\520_9320.aux
Size is 6250, 6250
Coordinate System is `'
Origin = (520000.000000000000000,9321000.000000000000000)
Pixel Size = (0.160000000000000,-0.160000000000000)
Metadata:
TIFFTAG_SOFTWARE=OrthoVista
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 520000.000, 9321000.000)
Lower Left ( 520000.000, 9320000.000)
Upper Right ( 521000.000, 9321000.000)
Lower Right ( 521000.000, 9320000.000)
Center ( 520500.000, 9320500.000)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=100.007, StdDev=34.047
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=100.007484672
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.7884416006343e-299
   STATISTICS_STDDEV=34.047496023691
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=121.846, StdDev=37.349
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=121.8462572288
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8047999164279e-299
   STATISTICS_STDDEV=37.348685345109
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Min=0.000 Max=255.000
Minimum=0.000, Maximum=255.000, Mean=135.154, StdDev=40.212
Overviews: 1024x1024
Metadata:
   STATISTICS_MINIMUM=0
   STATISTICS_MAXIMUM=255
   STATISTICS_MEAN=135.154277504
   STATISTICS_MEDIAN=0
   STATISTICS_MODE=1.8111374963437e-299
   STATISTICS_STDDEV=40.211793313468
   LAYER_TYPE=athematic
<GDALRasterAttributeTable/>

Many thanks in advance for any advice!

Pekka Hurskainen
SMOLE
Zanzibar, Tanzania
pekka.hurskainen@anonymised.com

------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Arne Kepp
OpenGeo - http://opengeo.org
Expert service straight from the developers