[Geoserver-users] Huge raster dataset - FGDB & ArcSDE

GeoServer GeoServer User
1

My questions relate to Serving huge raster datasets with GeoServer, but as I’m using a different configuration, I’ll start a new thread.

I was planning to use GeoServer, GeoWebCache and OpenLayers to serve ESRI data as WMS, WFS and WCS services to our web clients. Not too many problems for vector data, but well for raster data.

The data being read and modified by other (ESRI) applications, and as it’s huge (2TB uncompressed) they have to reside on an ESRI-compatible storage, not in GeoServer (other than cache).

Currently the raster data is stored in three FGDB’s :

  1. Non colormapped 3 bands RGB, 680 GB uncompressed, has to grow to 1.3 TB, pyramid level 10, jpeg compressed
  2. Non colormapped 1 band B&W, 230 GB uncompressed, pyramid level 10, jpeg compressed
  3. Colormapped 1 band unsigned byte, three tables of 140 GB in total, pyramids level 10, 8 and 7, LZ77 compressed.

As ESRI has been working since 2006 on an open API for FGDB, I don’t expect it for tomorrow. So we’d have to change our storage anyway.

The vector data being on an arcsde-oracle server, I thought I could maybe put the raster data as well in this DB and serve them from there. But serving arcsde colormapped rasters is planned for GeoServer 1.7.3 release, and I’m not sure yet if this is a good solution for serving huge data.

My questions :

  1. Any solution for working directly with the FGDB files ?
  2. Could arcsde be a good solution, when it will get developped ?
  3. Is it really planned for version 1.7.3, or could it be moved to a later release ?
  4. Should I go the GDAL way ? or another one ?

I really hope I can use GeoServer to serve my data, but if it wouldn’t fit my needs, would another project do it ? MapServer, MapFish ?

Alain

View this message in context: Huge raster dataset - FGDB & ArcSDE
Sent from the GeoServer - User mailing list archive at Nabble.com.

2

Hi Alain,

To answer the question about timelines. Yes, there is currently funded work going on to support color mapped rasters in SDE. As for releasing it as part of 1.7.3.. its is possible that a preliminary version could get into 1.7.3, but we are not sure yet as implementation is proceeding as we speak. So I would say either 1.7.3 or 1.7.4 definitely.

acangi wrote:

My questions relate to Serving huge raster datasets with GeoServer <http://www.nabble.com/serving-huge-raster-datasets-with-GeoServer-td21369212.html&gt;, but as I'm using a different configuration, I'll start a new thread.

I was planning to use GeoServer, GeoWebCache and OpenLayers to serve ESRI data as WMS, WFS and WCS services to our web clients. Not too many problems for vector data, but well for raster data.

The data being read and modified by other (ESRI) applications, and as it's huge (2TB uncompressed) they have to reside on an ESRI-compatible storage, not in GeoServer (other than cache).

Currently the raster data is stored in three FGDB's :

   1. Non colormapped 3 bands RGB, 680 GB uncompressed, has to grow to
      1.3 TB, pyramid level 10, jpeg compressed
   2. Non colormapped 1 band B&W, 230 GB uncompressed, pyramid level 10,
      jpeg compressed
   3. Colormapped 1 band unsigned byte, three tables of 140 GB in total,
      pyramids level 10, 8 and 7, LZ77 compressed.

As ESRI has been working since 2006 on an open API for FGDB, I don't expect it for tomorrow. So we'd have to change our storage anyway.

The vector data being on an arcsde-oracle server, I thought I could maybe put the raster data as well in this DB and serve them from there. But serving arcsde colormapped rasters <http://jira.codehaus.org/browse/GEOS-2567&gt; is planned for GeoServer 1.7.3 release, and I'm not sure yet if this is a good solution for serving huge data.

My questions :

   1. Any solution for working directly with the FGDB files ?
   2. Could arcsde be a good solution, when it will get developped ?
   3. Is it really planned for version 1.7.3, or could it be moved to a
      later release ?
   4. Should I go the GDAL way ? or another one ?

I really hope I can use GeoServer to serve my data, but if it wouldn't fit my needs, would another project do it ? MapServer, MapFish ?

Alain

------------------------------------------------------------------------
View this message in context: Huge raster dataset - FGDB & ArcSDE <http://www.nabble.com/Huge-raster-dataset---FGDB---ArcSDE-tp21498819p21498819.html&gt;
Sent from the GeoServer - User mailing list archive <http://www.nabble.com/GeoServer---User-f1194.html&gt; at Nabble.com.

------------------------------------------------------------------------

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword

------------------------------------------------------------------------

_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

Ciao Alain,
I hardly believe you'll find decent support for FGBD in any OS
project, especially when raster is involved.

That said, once complete support for ArcSDE raster will be implement
it will surely be capable of handling the amount of data you are
talking about, especially if GeoWebcache will be part of the game. An
alternative, since you are using oracle anyway, could be using oracle
georaster, which is supported by gdal, but which is not supported yet
by geoserver. Of course other paths are viable, like
extending/improving the imagemosaic-jdbc plugin which is already part
of geoserver but that would require a bit of funding. I guess some
more investigation and scoping is required to come up with a good
solution for your problem.

Ciao,
Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Fri, Jan 16, 2009 at 2:00 PM, acangi <aca@anonymised.com> wrote:

My questions relate to Serving huge raster datasets with GeoServer, but as
I'm using a different configuration, I'll start a new thread.

I was planning to use GeoServer, GeoWebCache and OpenLayers to serve ESRI
data as WMS, WFS and WCS services to our web clients. Not too many problems
for vector data, but well for raster data.

The data being read and modified by other (ESRI) applications, and as it's
huge (2TB uncompressed) they have to reside on an ESRI-compatible storage,
not in GeoServer (other than cache).

Currently the raster data is stored in three FGDB's :

Non colormapped 3 bands RGB, 680 GB uncompressed, has to grow to 1.3 TB,
pyramid level 10, jpeg compressed
Non colormapped 1 band B&W, 230 GB uncompressed, pyramid level 10, jpeg
compressed
Colormapped 1 band unsigned byte, three tables of 140 GB in total, pyramids
level 10, 8 and 7, LZ77 compressed.

As ESRI has been working since 2006 on an open API for FGDB, I don't expect
it for tomorrow. So we'd have to change our storage anyway.

The vector data being on an arcsde-oracle server, I thought I could maybe
put the raster data as well in this DB and serve them from there. But
serving arcsde colormapped rasters is planned for GeoServer 1.7.3 release,
and I'm not sure yet if this is a good solution for serving huge data.

My questions :

Any solution for working directly with the FGDB files ?
Could arcsde be a good solution, when it will get developped ?
Is it really planned for version 1.7.3, or could it be moved to a later
release ?
Should I go the GDAL way ? or another one ?

I really hope I can use GeoServer to serve my data, but if it wouldn't fit
my needs, would another project do it ? MapServer, MapFish ?

Alain

________________________________
View this message in context: Huge raster dataset - FGDB & ArcSDE
Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

Hi Alain, I’m at MassGIS and we are sponsoring the work to have GeoServer be able to serve additional raster types from SDE.
Currently we’re serving 2 layers which you can play with at http://giswebservices.massgis.state.ma.us/geoserver

GISDATA.IMG_COQ2001 is a 3 band non colormapped RGB - about 300 GB, 12 pyramid levels
GISDATA.IMG_COQ2005 is a 4 band non colormapped RGB - about 400 GB, 12 pyramid levels

We’re running Oracle 10, SDE 9.2 SP5 and GeoServer 1.6.5

This is the list of JIRA tasks related to the new rasters: http://jira.codehaus.org/browse/GEOS-2567

Aleda Freeman
http://www.mass.gov/mgis - MassGIS
http://lyceum.massgis.state.ma.us - Webservices Wiki
MassGIS is the Commonwealth’s Office of Geographic and Environmental Information, located in the Executive Office of Energy and Environmental Affairs

5

Justin, Simone, Aleda,

Thanks for the hints. I'll be waiting eagerly for arcsde plugin
developments.

Alain

--
View this message in context: http://www.nabble.com/Huge-raster-dataset---FGDB---ArcSDE-tp21498819p21560198.html
Sent from the GeoServer - User mailing list archive at Nabble.com.