RE: [Geoserver-devel] Ingestion Engine

> Hi Alessio,
> what do you mean exactly by "...ingestion engine"
> and by "...At the end the user should be able
> to manage Geoserver data like a file system directory tree"???
> I'm asking this because it sounds like something we may need too,
> but maybe I'm misunderstanding...
>
We'd like to avoid as much as possible using the GeoServer web
interface for data management both for features and coverages in order
to ease administrator's job. We'd like to allow GeoServer to
automatically recognize the type of the files uploaded onto the data
directory and to dispatch them to the right service.
Moreover, once the administrator deletes a file or a subdir from data,
GeoServer should be able to remove automatically the feature/coverage
from the services.

Ok, so it was exactly what I thought, avoiding the Web Admin interface
and have a mechanism to automatically configure GeoServer with the
DataStores (and coverages for your WCS version) that are available.
We already have that, but maybe it doesn't work for coverages because
they're a little different from DataStores. The drawback is that it only
works with JBoss and its integrated Tomcat, but it works quite well.
To add a DataStore to GeoServer you only have to put an XML file inside
JBoss' deploy directory. The file must be formatted as for JBoss' SAR
specifications and basically contain the connection params of the DataStore.
This automatically adds to GeoServer the DataStore and all the FeatureTypes
it contains. To remove the DataStore you simply have to
remove the file. In a sense we effectively used JBoss' own
hot-autodeploy mechanism to do what you planned to do using Quartz.
But there's more than that...

Another open issue related to this one would be having the possibility
to handle nested wms layers reflecting an hypothetical file system
defined or user-defined structure, but at this moment we have no
suggestion on this topic.

I didn't looked inside the WCS stuff, so I can't exactly understand what
you are meaning with "...nested wms layers...", anyway our system is based
on a Meta engine we have developed, that let's you create a hierarchical
structure of DataStores, so that you can attach a DataStore "foo" directly
to the root one, and this will give it in GeoServer an id of "root:foo", or
you can attach it under another one, so it'll have an id of, say,
"root:foo:sub". Also typenames are treated this way, so the complete id for
a certain FeatureType might be something like "root:foo:sub:type1". I don't
know if this is something
near what are looking for...

We would like to hear something on these topics from the other active
GeoServer developers (david, chris, etc..) since we are talking about
major changes/improvements and everybody's opinion is very very
welcome.

We already talked a little about this with Chris and Dave, and about other
issues as well such as adding security, versioning, etc.,
and we're probably going to open a branch on GeoServer to add all our
functionalities, maybe only as experiments, but we're going to use our
branch in production, so it is going to be something really working (we
hope). We only should be able to remove the dependency from JBoss, but this
would require a certain deal of work, so we'd like to postpone it a little,
if possible, but we can even try to do that up front, if it is
deemed important by others.
I think it would be a good thing if we all can put together a common
approach, some form of coordination between our efforts, to not do the
same things twice.

What do you think???
   Ciao Paolo

On 7/15/05, P. Rizzi Ag.Mobilità Ambiente <paolo.rizzi@anonymised.com> wrote:

> > Hi Alessio,
> > what do you mean exactly by "...ingestion engine"
> > and by "...At the end the user should be able
> > to manage Geoserver data like a file system directory tree"???
> > I'm asking this because it sounds like something we may need too,
> > but maybe I'm misunderstanding...
> >
> We'd like to avoid as much as possible using the GeoServer web
> interface for data management both for features and coverages in order
> to ease administrator's job. We'd like to allow GeoServer to
> automatically recognize the type of the files uploaded onto the data
> directory and to dispatch them to the right service.
> Moreover, once the administrator deletes a file or a subdir from data,
> GeoServer should be able to remove automatically the feature/coverage
> from the services.
Ok, so it was exactly what I thought, avoiding the Web Admin interface
and have a mechanism to automatically configure GeoServer with the
DataStores (and coverages for your WCS version) that are available.
We already have that, but maybe it doesn't work for coverages because
they're a little different from DataStores. The drawback is that it only
works with JBoss and its integrated Tomcat, but it works quite well.
To add a DataStore to GeoServer you only have to put an XML file inside
JBoss' deploy directory. The file must be formatted as for JBoss' SAR
specifications and basically contain the connection params of the DataStore.
This automatically adds to GeoServer the DataStore and all the FeatureTypes
it contains. To remove the DataStore you simply have to
remove the file. In a sense we effectively used JBoss' own
hot-autodeploy mechanism to do what you planned to do using Quartz.
But there's more than that...

Ok, it sounds very good. I know the JBoss hot deploy.
It would be very helpful to take a look to your solution, so I can
better understand if it is reusable for coverages and if there is a
fast way to remove JBoss dependency.

Actually I have some difficulties to figure out how you deploy
DataStores like Shapefiles into GeoServer. I need more details about
that.

> Another open issue related to this one would be having the possibility
> to handle nested wms layers reflecting an hypothetical file system
> defined or user-defined structure, but at this moment we have no
> suggestion on this topic.
I didn't looked inside the WCS stuff, so I can't exactly understand what
you are meaning with "...nested wms layers...", anyway our system is based
on a Meta engine we have developed, that let's you create a hierarchical
structure of DataStores, so that you can attach a DataStore "foo" directly
to the root one, and this will give it in GeoServer an id of "root:foo", or
you can attach it under another one, so it'll have an id of, say,
"root:foo:sub". Also typenames are treated this way, so the complete id for
a certain FeatureType might be something like "root:foo:sub:type1". I don't
know if this is something
near what are looking for...

We use Nested Layers into the WMS in order to help the end user to
find faster the Map he desires, waiting for a catalog service coming
out.
With Nested Layers you can replicate the tree structure of your data
into the WMS GetCapabilities response, so you have a way to organize
the WMS data presentation.
Obviously it works only for the WMS because in the WFS and WCS does
not exist the concept of "layer".

> We would like to hear something on these topics from the other active
> GeoServer developers (david, chris, etc..) since we are talking about
> major changes/improvements and everybody's opinion is very very
> welcome.
We already talked a little about this with Chris and Dave, and about other
issues as well such as adding security, versioning, etc.,
and we're probably going to open a branch on GeoServer to add all our
functionalities, maybe only as experiments, but we're going to use our
branch in production, so it is going to be something really working (we
hope). We only should be able to remove the dependency from JBoss, but this
would require a certain deal of work, so we'd like to postpone it a little,
if possible, but we can even try to do that up front, if it is
deemed important by others.
I think it would be a good thing if we all can put together a common
approach, some form of coordination between our efforts, to not do the
same things twice.

What do you think???
  Ciao Paolo

We think that if you can commit some example code, we can start to
work on it and see what can be reused (hopefully all), because we need
the Ingestion Engine as soon as possible.

On monday we can speek about that in the IRC session if it's ok for
Chris, Jody and James.