Hi Martin…
Thanks again for your informative reply.
I suppose it would be wise to take advantage of the otherwise disadvantageous fact that my experience with geoserver consists of only a simple demonstration. What I mean is, hopefully I can provide a fresh new look at how geoserver should address the issue of nD raster data layers. I’m very interested to see how you think my ideas might best fit into the geoserver architecture (that I am completely unfamiliar with).
Over the weekend I started thinking about how I might build a mini-nD-WMS/WCS from scratch. I figure that the database should be in charge of holding all the metadata, georefs, pseudocolor tables, and raw data (whether inside as an array of BLOBs, or outside in netCDFs or whatever). It should be able to respond to any request for raw data (a layer) given spatial and temporal extents, and resolutions and return an nD array of values. The db would then give the array along with styling info, like a pseudocolor table, back to mapserver, or geoserver (we should design it to work with both) to be rendered (or packaged as WCS) and sent back to the requester.
The db should also be organized such that it contains overviews or pyramids of the data for display at different scales. These pyramids would match those specified in a WMS-Tiled request. So the db would contain pre-cached images (or links to these images).
The way I see it, the (geo|map)server shouldn’t have to deal with files, but should be able to just take the data, reproject it, style it, package it as a WS, and respond back to the request. So, implementing nD rasters in GeoServer turns out to be an issue of building the db connections to a component in geoserver that relays the WS request and response. In the end it would be great, because it could be easily set up to provide a solution for serving large 2D data raster datasets as well.
So that’s my idea of how this nD problem could ideally be solved. How well do you think this solution would fit into GeoServer’s current architecture? How well is it aligned with what you’ve been doing with PostGrid?
I haven’t yet been able to look into the PostGrid code. I imagine I should be able to organize a file structure inside PostGrid that optimizes a response to a WMS-Tiled request to get that part functional fairly easily?
Oh, and I propose we consider renaming the PostGrid project to something like nDraster so it is more explicative of its function. What do you think? Maybe the change could be licensed under the GPL, too, so that it can use GeoServer code where necessary. Does that make sense?
I’ll get started setting up my database with PostGrid and making myself more familiar with everything. In the meantime I’m anxious to hear your thoughts on my proposed architecture, and to hear how easily you think we might be able to get it working.
I look forward to working with you to finally solve this nD problem.
I also welcome any feedback from other geoserver/geotools developers. … which brings up another question: is this development effort more appropriate for discussion on the geotools-devel list?
Thanks again.
Sam
On Dec 3, 2007 1:35 AM, Martin Desruisseaux <martin.desruisseaux@anonymised.com> wrote:
Hello Samuel
Samuel Hiatt a écrit :
But I hear it is a sin for hackers to re-write code, and so I am really
interested in teaming up with you…Sure that would be nice!
I am guessing that first GeoServer interprets the WMS request and then
asks PostGrid where to find the right netCDF archive. Coverageio-netcdf
goes and gets the appropriate subset of raw data and then gives it to
GeoTool’s ImageReader to color it appropriately and spit it back to the
client? Is that right?Yes with a few nuances. Just replace “GeoTools’s ImageReader” by “GeoTools’s
streaming renderer”. We can also add that the coloring is currently done
directly by PostGrid and Coverageio-netcdf. I admit that coloring should be
delegated to GeoServer if we want to leverage their styling support.Note: Current GeoServer coverage support is designed for 2D rasters only. It is
designed with “one file == one layer” in mind, which is not applicable to nD
data. PostGrid integration in GeoServer is currently a hack with embarassing
side effects. When configuring a layer in the GeoServer configuration interface,
we must specify a file, which make no-sense for nD data (there is many files,
not just one). We must also specify valid Envelope and CRS, which are totally
ignored by PostGrid since they are already declared in the database. But users
have to do that because GeoServer coverage support is currently designed like
that. The GeoServer guys are aware of that, but it may take a while before we
can fix that.And PostGrid is just a schema with some scripts to auto-create the db
and then also to respond to the GS request and spit back the correct file?Yes. The scripts to auto-create the DB are in SQL, tuned for PostgreSQL with
PostGIS (not sure it could run on other DB). The codes for responding to GS
request are in Java.I imagine that the OGC WKT geo-referencing standard addresses these
issues. Do you have something else in mind?Java’s IIOMetadata is quite close in spirit to “GML in JPEG 2000” from OGC, so
the later is our main reference. We can not strictly align with that OGC
specification however, because IIOMetadata imposes some constraints that prevent
that, and also because I’m not sure that we need all the complexity of “GML in
JPEG 2000”.So it looks like you are thinking of doing a similar trick, bypassing
the (geo|map)server and running your own WMS/WCS. It sure is a tempting
idea, isn’t it? It is quite obvious that both servers were designed
with only 2D rasters in mind.Yes we are very likely to do that soon (maybe a first prototype this week).
However GeoServer has an advantage that we don’t have: a great community. So we
will try to be careful to not present that as a GeoServer competitor, but rather
as a tools designed specifically for nD rasters on PostGrid that we could move
to GeoServer later if the community wants (and if it is technically doable - it
would require consequent changes in GeoServer code before we can do that and it
may take a while). The license of both projects (LGPL for SeaGIS, GPL for
GeoServer) allow code transferts from SeaGIS to GeoServer, but not the other way
around. So we have a little bit more flexibility by doing the work in SeaGIS for
now and moving to GeoServer later.So now that you know a bit more about where I am and what my objectives
are with this project I am curious to know your opinion regarding if it
is worth hopping onto the java/geotools/geoserver train so I can use
your code. I am hesitant to do so because I am more familiar with the
way mapserver does things and I have minimal experience with java, but
the idea still tempts me for the following reasons:
- I get to leverage your efforts in serving nD raster data
- I like Geoserver’s out-of-the-box integration with Google Earth, etc…
- I think that Geoserver really needs native nD raster support. The
geek in me wants to help make it happen.Sure any help to integrate that in GeoServer would be very welcome! In my
understanding, there is interest in the GeoServer community for PostGrid.
However in order to make that happen, there is some work that needs to be done
which are not directly nD-related:
The whole Coverage I/O stuff in GeoTools (not to be confused with Image I/O)
needs to be rewrite from scratch. I will take that job, but we planed it only
for March-May 2008.According Andrea Aime (the main GeoServer maintainer), the configuration
stuff in GeoServer needs to be rewriten too. Andrea said that doing any
work on top of current configuration framework is like working on a dead
body. This is the kind of stuff that force the users to declares a file
for a layer even if PostGrid will ignore it.So we are leading toward a mini-WMS or WCS server bypassing completly GeoServer
until the above is fixed, and try to merge back to GeoServer later.If you want to know a
bit more about the types of data we will be serving, check outThat sound like! I’m oceanographer (in theory), so this kind of data looks like
the one I like to work with.Martin