RE: [Geoserver-devel] Ingestion Engine

GeoServer GeoServer Developer
,
1

Ok, it sounds very good. I know the JBoss hot deploy.
It would be very helpful to take a look to your solution, so I can
better understand if it is reusable for coverages and if there is a
fast way to remove JBoss dependency.

I know I have to put together some kind of good demo, I'll try to do it
as soon as I can. The problem with removing the JBoss dependency is that
you then have to implement your own deploy engine. What I'd do, instead,
is create some kind of intermediate abstraction, and then implement it
using JBoss. Then someone else not wanting to use JBoss would be free
to implement it using another system. I'd like to use this approach
in "every" part of what we're doing, so to be able to proceed fast
with something reliable, without preempting anyone to use a different
system.

Actually I have some difficulties to figure out how you deploy
DataStores like Shapefiles into GeoServer. I need more details about
that.

Here's an example that adds a shapefile and a PostGIS DataStore.
They can be saved as a single file inside JBoss deploy directory with
a "magic" name like "twodatastores-service.xml" (mind the new-lines).
-----------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE server
    PUBLIC "-//JBoss//DTD MBean Service 3.2//EN"
    "http://www.jboss.org/j2ee/dtd/jboss-service_3_2.dtd&quot;&gt;
<server>
  
<loader-repository>someuniquename:loader=someuniquename.ear</loader-reposito
ry>
  
  <mbean code="it.ama_mi.sis.framework.j2ee.SISModuleLoaderService"
    name="someuniquename:service=SISModule,name=datastoreid1">
    <attribute name="SISInstanceName">someuniquename</attribute>
    <attribute name="Id">datastoreid1</attribute>
    <attribute name="DataStoreProperties">
      url=file:/some/path/to/any/shapefile.shp
    </attribute>
  
<depends>someuniquename:service=SISMetaSpaceService</depends>
  </mbean>

  <mbean code="it.ama_mi.sis.framework.j2ee.SISModuleLoaderService"
  
name="someuniquename:service=SISModule,name=anydatastoreid1">
    <attribute name="SISInstanceName">someuniquename</attribute>
    <attribute name="Id">datastoreid2</attribute>
    <attribute name="DataStoreProperties">
      dbtype=postgis
      host=localhost
      port=5432
      database=somedb
      user=someuser
      passwd=somepassword
    </attribute>
  
<depends>someuniquename:service=SISMetaSpaceService</depends>
  </mbean>

</server>
-----------------------------------------

Obviosuly it doesn't work without all our stuff...
Actually I now have the possibility to easily package GeoServer with
our stuff and any demo service I like, so it might not be difficult
to put a working demo in the Wiki (obviously you'll need JBoss to see it).

We use Nested Layers into the WMS in order to help the end user to
find faster the Map he desires, waiting for a catalog service coming
out.
With Nested Layers you can replicate the tree structure of your data
into the WMS GetCapabilities response, so you have a way to organize
the WMS data presentation.
Obviously it works only for the WMS because in the WFS and WCS does
not exist the concept of "layer".

...I really didn't look into any of the above well, so I'm too far from
what you're saying. I hope the demo will help you understand...

We think that if you can commit some example code, we can start to
work on it and see what can be reused (hopefully all), because we need
the Ingestion Engine as soon as possible.

I know... It would be great if I'd manage to commit our branch very soon,
but I don't think it would be before a week or two, but I'll see if
I can do something faster.

On monday we can speek about that in the IRC session if it's ok for
Chris, Jody and James.

Yes, maybe the time has come for this discussion.

Anyway, could you use JBoss as an immediate solution or you are absolutely
with another container for some reason???
   Bye Paolo

2

Quoting "P. Rizzi Ag.Mobilità Ambiente" <paolo.rizzi@anonymised.com>:

> Ok, it sounds very good. I know the JBoss hot deploy.
> It would be very helpful to take a look to your solution, so I can
> better understand if it is reusable for coverages and if there is a
> fast way to remove JBoss dependency.
I know I have to put together some kind of good demo, I'll try to do
it
as soon as I can. The problem with removing the JBoss dependency is
that
you then have to implement your own deploy engine. What I'd do,
instead,
is create some kind of intermediate abstraction, and then implement
it
using JBoss. Then someone else not wanting to use JBoss would be free
to implement it using another system. I'd like to use this approach
in "every" part of what we're doing, so to be able to proceed fast
with something reliable, without preempting anyone to use a different
system.

This definitely sounds like the way to go.

> We use Nested Layers into the WMS in order to help the end user to
> find faster the Map he desires, waiting for a catalog service
coming
> out.
> With Nested Layers you can replicate the tree structure of your
data
> into the WMS GetCapabilities response, so you have a way to
organize
> the WMS data presentation.
> Obviously it works only for the WMS because in the WFS and WCS does
> not exist the concept of "layer".
...I really didn't look into any of the above well, so I'm too far
from
what you're saying. I hope the demo will help you understand...

Paolo - basically what he's talking about is the WMS capabilities
document. GeoServer just lists all the layers as if they were in the
WFS. But the WMS spec allows you to group different sets of layers
together, and to have them nested in all kinds of different ways,
inheriting properties from above and whatnot. We have no concept of
this in geoserver, as we have no concept of it in our organization of
data either. So I think Alessio is basically talking about using the
file structure of the data to represent the Capabilities response.

I think I'd be for it, but it'd take some good thinking to get it right.
My worry with the ingesting is that all the metadata about the layers
would just be crap, a bunch of generated junk that wouldn't help users
at all. Perhaps we could look into ways of deriving more of it, like
from companion xml file to shapefile, and perhaps there's an equivalent
to arcgrid? But I suppose if the nesting structure is done right that
could help - you'd establish a parent directory with a set of keywords,
and all children would inherit appropriately - even though the WFS
doesn't support the layer concept the children could still grab all
their keywords from the parent, for example.

Chris

> We think that if you can commit some example code, we can start to
> work on it and see what can be reused (hopefully all), because we
need
> the Ingestion Engine as soon as possible.
I know... It would be great if I'd manage to commit our branch very
soon,
but I don't think it would be before a week or two, but I'll see if
I can do something faster.

> On monday we can speek about that in the IRC session if it's ok for
> Chris, Jody and James.
Yes, maybe the time has come for this discussion.

Anyway, could you use JBoss as an immediate solution or you are
absolutely
with another container for some reason???
   Bye Paolo

-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration
Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

----------------------------------------------------------
This mail sent through IMP: https://webmail.limegroup.com/

3

I think I'd be for it, but it'd take some good thinking to get it right.
My worry with the ingesting is that all the metadata about the layers
would just be crap, a bunch of generated junk that wouldn't help users
at all. Perhaps we could look into ways of deriving more of it, like
from companion xml file to shapefile, and perhaps there's an equivalent
to arcgrid? But I suppose if the nesting structure is done right that
could help - you'd establish a parent directory with a set of keywords,
and all children would inherit appropriately - even though the WFS
doesn't support the layer concept the children could still grab all
their keywords from the parent, for example.

I admit I have to study it deeper, but have a pretty strong feeling that
using a OWC (OGC Web Context) document as the back format for configuring WMS
layers would make a lot more sense.
If we go 'ingesting' the changes on the filesystem, don't are we going to get
sticked to a filesystem structure and worse than that, having to deal with
moving files and directories to reflect a single change on a layer hierarchy?

I imagine a single OWC document as the WMS config storage format and using
(will need extension) the ows/wms/wfs geotools packages as the in-memory
representation.

I also know it could be more work at first, but if it makes sense for you I
think it worths the effort.

gabriel

4

Quoting Gabriel Roldán <groldan@anonymised.com>:

> I think I'd be for it, but it'd take some good thinking to get it
right.
> My worry with the ingesting is that all the metadata about the
layers
> would just be crap, a bunch of generated junk that wouldn't help
users
> at all. Perhaps we could look into ways of deriving more of it,
like
> from companion xml file to shapefile, and perhaps there's an
equivalent
> to arcgrid? But I suppose if the nesting structure is done right
that
> could help - you'd establish a parent directory with a set of
keywords,
> and all children would inherit appropriately - even though the WFS
> doesn't support the layer concept the children could still grab all
> their keywords from the parent, for example.
>
I admit I have to study it deeper, but have a pretty strong feeling
that
using a OWC (OGC Web Context) document as the back format for
configuring WMS
layers would make a lot more sense.
If we go 'ingesting' the changes on the filesystem, don't are we
going to get
sticked to a filesystem structure and worse than that, having to deal
with
moving files and directories to reflect a single change on a layer
hierarchy?

I imagine a single OWC document as the WMS config storage format and
using
(will need extension) the ows/wms/wfs geotools packages as the
in-memory
representation.

I also know it could be more work at first, but if it makes sense for
you I
think it worths the effort.

It makes sense for sure. I suppose I don't necessarily see the two as
mutually exclusive, and don't really want to rewrite the config right
now. I see the ingestor engine as quickly getting your data in, but
then you'd still do a 'save' on it, which would take the file system
and persist the structure into the OWC document. From there you could
fine tune the structure that you wanted for your layers, but the
ingestor would serve as a starting point. You would never _have_ to
use the ingestor, it would not determine the structure the config
documents would. Though I'm not sure I still understand exactly what
people want out of it, but I think it could just be an additional
option to how you get data in GeoServer. There's editing xml files
directly, there's using the web admin. I'd like to see udig as a
configuration tool, you just drag and drop and use the wizards, then
export a geoserver structure. And then this is just an option to use
the file ingestor - but I think it would still produce DTO's (at least
in version 1.x), and persist with our xml config files.

Chris

gabriel

----------------------------------------------------------
This mail sent through IMP: https://webmail.limegroup.com/