[Geoserver-devel] CSW Module Structure

GeoServer GeoServer Developer
1

Hi Everyone,

I am currently wrapping up the CSW work. The ISO MetaData has been fully implemented and I'm now doing the final touches.
One question that arose was about the module structure.

So far we had chosen for the following structure. There are four modules:

* csw-api : interfaces and base classes
* csw-core : services, encoders, Dublin Core profile
                   -> depends on csw-api
                   -> tests depend on csw-simple-store
* csw-simple-store: implementation of a simple store that uses xml files as data, used for testing in csw-core (only supports DC)
                     -> depends on csw-api
                     -> depends on csw-core
* csw-internal-store: implementation of store linked to geoserver catalog layers through mapping file + implementation if ISO
    MetaData profile with this store.
                     -> depends on csw-api
                     -> depends on csw-core

The question posed was whether csw-internal-store should be merged with csw-core.
csw-core would then test both using the simple-store (which is still in a separate module) and the internal store.
Or are there any other module structures that should be considered?

Another question posed was whether 'internal store' should be renamed to 'default store'.

Kind Regards
Niels Charlier

2

I was thinking that once the “internal” store was completed it would be basically included with the core module, and serve as the default so that without more or less zero configuration you could fire up geoserver and have a working csw. If others agree with that approach the name “default” probably makes more sense than “internal”.

To facilitate the plugging in of other stores like the simple one we could get fancy but something similar to how the ResourceAccessManager for security would work. Basically we do a look up for CatalogStore interface in the spring context, and if one is not found we fall back on the internal one.

So with that the module structure would be:

csw/
api/
core/
simple/

Also there is a test dependency from the core module on the simple module for testing purposes. I think it probably makes sense to refactor all the test cases that currently use the simple store into a set of abstract classes that remain in core. And then simple and default would both implement all the test cases to get the same coverage. More or less how we do jdbc datastore testing in GeoTools.

$0.02

···

On Wed, Dec 5, 2012 at 10:56 AM, Niels Charlier <niels@anonymised.com> wrote:

Hi Everyone,

I am currently wrapping up the CSW work. The ISO MetaData has been fully
implemented and I’m now doing the final touches.
One question that arose was about the module structure.

So far we had chosen for the following structure. There are four modules:

  • csw-api : interfaces and base classes
  • csw-core : services, encoders, Dublin Core profile
    → depends on csw-api
    → tests depend on csw-simple-store
  • csw-simple-store: implementation of a simple store that uses xml files
    as data, used for testing in csw-core (only supports DC)
    → depends on csw-api
    → depends on csw-core
  • csw-internal-store: implementation of store linked to geoserver
    catalog layers through mapping file + implementation if ISO
    MetaData profile with this store.
    → depends on csw-api
    → depends on csw-core

The question posed was whether csw-internal-store should be merged with
csw-core.
csw-core would then test both using the simple-store (which is still in
a separate module) and the internal store.
Or are there any other module structures that should be considered?

Another question posed was whether ‘internal store’ should be renamed to
‘default store’.

Kind Regards
Niels Charlier

LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

On Wed, Dec 5, 2012 at 10:38 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

I was thinking that once the “internal” store was completed it would be basically included with the core module, and serve as the default so that without more or less zero configuration you could fire up geoserver and have a working csw. If others agree with that approach the name “default” probably makes more sense than “internal”.

To facilitate the plugging in of other stores like the simple one we could get fancy but something similar to how the ResourceAccessManager for security would work. Basically we do a look up for CatalogStore interface in the spring context, and if one is not found we fall back on the internal one.

So with that the module structure would be:

csw/
api/
core/
simple/

I am twisted about this one, there are a few ways to skin this cat, not sure this one is the best.

First option, keep the modules as separate as they are today, but include in the release zip only the internal store, leaving the other available
as an example for developers (and for running cite tests).
You drop in the store you want, you go with it, pretty simple.
Advantage:

  • if you are building a custom GeoServer with a custom catalog, you don’t have to carry around needless
    baggage (btw, the ISO bindings should be in api though)
    Disadvantage:
  • a bit rigid, what if one wants multiple catalog sources?

Second option, the one you propose. Similar to the above, wit the extra baggage in the classpath

Third option, do as one (all stores out by default), but put in the extension package all implementations
and create a “composite” store and make thing configurable by the admin:
a GUI allows to list the implementations and choose which to enable and which leave dormant,
this one might have both the internal store and some external ones too.

Also there is a test dependency from the core module on the simple module for testing purposes. I think it probably makes sense to refactor all the test cases that currently use the simple store into a set of abstract classes that remain in core. And then simple and default would both implement all the test cases to get the same coverage. More or less how we do jdbc datastore testing in GeoTools.

Hmm… don’t think this is going to fly, the CSW tests are assuming the CITE test data is available, but
one can have it only if you have around a store allowing to choose the input data (the internal one
is locked onto the layers instead).

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

4

On Thu, Dec 6, 2012 at 6:24 AM, Andrea Aime <andrea.aime@anonymised.com>wrote:

On Wed, Dec 5, 2012 at 10:38 PM, Justin Deoliveira <jdeolive@anonymised.com>wrote:

I was thinking that once the "internal" store was completed it would be
basically included with the core module, and serve as the default so that
without more or less zero configuration you could fire up geoserver and
have a working csw. If others agree with that approach the name "default"
probably makes more sense than "internal".

To facilitate the plugging in of other stores like the simple one we
could get fancy but something similar to how the ResourceAccessManager for
security would work. Basically we do a look up for CatalogStore interface
in the spring context, and if one is not found we fall back on the internal
one.

So with that the module structure would be:

csw/
  api/
  core/
  simple/

I am twisted about this one, there are a few ways to skin this cat, not
sure this one is the best.

First option, keep the modules as separate as they are today, but include
in the release zip only the internal store, leaving the other available
as an example for developers (and for running cite tests).
You drop in the store you want, you go with it, pretty simple.
Advantage:
- if you are building a custom GeoServer with a custom catalog, you don't
have to carry around needless
  baggage (btw, the ISO bindings should be in api though)
Disadvantage:
- a bit rigid, what if one wants multiple catalog sources?

Right... i guess it would help to know what the classpath overhead is in
terms of additional dependencies, etc... of the internal store is. As far
as i can see it looks to be pretty minimal, no additional libs and just a
handful of classes.

@Niels: do I have that right?

Second option, the one you propose. Similar to the above, wit the extra
baggage in the classpath

Third option, do as one (all stores out by default), but put in the
extension package all implementations
and create a "composite" store and make thing configurable by the admin:
a GUI allows to list the implementations and choose which to enable and
which leave dormant,
this one might have both the internal store and some external ones too.

So a single module containing all backend implementations? Not sure how i

feel about that once other implementations start pilling up that could
potentially add overhead in terms of dependencies, etc... The composite
idea makes sense though but should be orthogonal to 1 module vs many
modules. If i understand it correctly.

Also there is a test dependency from the core module on the simple module
for testing purposes. I think it probably makes sense to refactor all the
test cases that currently use the simple store into a set
of abstract classes that remain in core. And then simple and default would
both implement all the test cases to get the same coverage. More or less
how we do jdbc datastore testing in GeoTools.

Hmm... don't think this is going to fly, the CSW tests are assuming the
CITE test data is available, but
one can have it only if you have around a store allowing to choose the
input data (the internal one
is locked onto the layers instead).

I see. Well what if we built the catalog purely around the CSW cite data,
similar to how we setup the cite data for testing the other services. Or
does the cite test data contain more than just "layers" that need to be
returned?

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

No indeed in this approach I would leave each store in its own jar, but all the store jars would end up in the
extension zip file.
Best of both worlds, no extras for custom implementations and everything (but configurable) for the regular user.
Comes at a price though, the composite store needs to be written.

I might have some time to work on it next year, but not sure exactly when

They do contain several types of information and not sure everything can be mapped to them using
only the internal store.
Have a look here:
https://github.com/geoserver/geoserver/tree/master/src/community/csw/csw-simple-store/src/test/resources/org/geoserver/csw/store/simple

These records have to be produced verbatim for the CITE tests, and the tests in csw-core have
several expectations on them too (thought not as strict as the CITE tests themselves)

Cheers
Andrea

···

Second option, the one you propose. Similar to the above, wit the extra baggage in the classpath

Third option, do as one (all stores out by default), but put in the extension package all implementations
and create a “composite” store and make thing configurable by the admin:
a GUI allows to list the implementations and choose which to enable and which leave dormant,
this one might have both the internal store and some external ones too.

So a single module containing all backend implementations? Not sure how i feel about that once other implementations start pilling up that could potentially add overhead in terms of dependencies, etc… The composite idea makes sense though but should be orthogonal to 1 module vs many modules. If i understand it correctly.

Also there is a test dependency from the core module on the simple module for testing purposes. I think it probably makes sense to refactor all the test cases that currently use the simple store into a set of abstract classes that remain in core. And then simple and default would both implement all the test cases to get the same coverage. More or less how we do jdbc datastore testing in GeoTools.

Hmm… don’t think this is going to fly, the CSW tests are assuming the CITE test data is available, but
one can have it only if you have around a store allowing to choose the input data (the internal one
is locked onto the layers instead).

I see. Well what if we built the catalog purely around the CSW cite data, similar to how we setup the cite data for testing the other services. Or does the cite test data contain more than just “layers” that need to be returned?

6

Yes, i only depends on what core and api depends.

···

Right… i guess it would help to know what the classpath overhead is in terms of additional dependencies, etc… of the internal store is. As far as i can see it looks to be pretty minimal, no additional libs and just a handful of classes.

@Niels: do I have that right?

Second option, the one you propose. Similar to the above, wit the extra baggage in the classpath

Third option, do as one (all stores out by default), but put in the extension package all implementations
and create a “composite” store and make thing configurable by the admin:
a GUI allows to list the implementations and choose which to enable and which leave dormant,
this one might have both the internal store and some external ones too.

So a single module containing all backend implementations? Not sure how i feel about that once other implementations start pilling up that could potentially add overhead in terms of dependencies, etc… The composite idea makes sense though but should be orthogonal to 1 module vs many modules. If i understand it correctly.

Also there is a test dependency from the core module on the simple module for testing purposes. I think it probably makes sense to refactor all the test cases that currently use the simple store into a set of abstract classes that remain in core. And then simple and default would both implement all the test cases to get the same coverage. More or less how we do jdbc datastore testing in GeoTools.

Hmm… don’t think this is going to fly, the CSW tests are assuming the CITE test data is available, but
one can have it only if you have around a store allowing to choose the input data (the internal one
is locked onto the layers instead).

I see. Well what if we built the catalog purely around the CSW cite data, similar to how we setup the cite data for testing the other services. Or does the cite test data contain more than just “layers” that need to be returned?

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

7

Could the current work be committed, so that we can have a look at it?

Cheers
Andrea

···

Right… i guess it would help to know what the classpath overhead is in terms of additional dependencies, etc… of the internal store is. As far as i can see it looks to be pretty minimal, no additional libs and just a handful of classes.

@Niels: do I have that right?

8

Could the current work be committed, so that we can have a look at it?

https://github.com/NielsCharlier/geoserver.git
branch: cswpublic

relies on
https://github.com/NielsCharlier/geotools.git
branch: csw

9

On Fri, Dec 7, 2012 at 2:30 PM, Niels Charlier <niels@anonymised.com> wrote:

Could the current work be committed, so that we can have a look at it?

https://github.com/NielsCharlier/geoserver.git
branch: cswpublic

So yeah, looking at the sources I believe all the support for ISO records should be moved into
csw-api anyways, allowing others to build stores that generate ISO records.
Basically this package:
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/records/iso
This one too, should probably be part of csw-core (since it deals with the procotol, instead of the record concept per se):
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/response/iso

This would leave the pure catalog in this module, which is indeed not big:
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/store/internal

I still think that having the store in its own module would look better (do we have any default
datastore or coveragestore in GeoServer? even the ones used by core can be plugged out),
but I’m not opposed to seeing this get merged in as the default one like we did for the security subsystem.

I guess the difference is on whether is it normal to have a store 100% of the time in, with little
likeliness of being replaced by another, or not.
Right now it certainly seems like it will be most of the time in, but if we grow an implementation
that can actually handle efficiently large number of records detached from the GS own
catalog things might change.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

10

Okay so I am going to conclude that we go with Justin's proposal to put the internal store in to core. The other thing is, should I keep its name or rename it to DefaultCatalogStore?

Regards
Niels

On 12/07/2012 05:50 PM, Andrea Aime wrote:

On Fri, Dec 7, 2012 at 2:30 PM, Niels Charlier <niels@anonymised.com <mailto:niels@anonymised.com>> wrote:

        Could the current work be committed, so that we can have a
        look at it?

    https://github.com/NielsCharlier/geoserver.git
    branch: cswpublic

So yeah, looking at the sources I believe all the support for ISO records should be moved into
csw-api anyways, allowing others to build stores that generate ISO records.
Basically this package:
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/records/iso
This one too, should probably be part of csw-core (since it deals with the procotol, instead of the record concept per se):
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/response/iso

This would leave the pure catalog in this module, which is indeed not big:
https://github.com/NielsCharlier/geoserver/tree/cswpublic/src/community/csw/csw-internal-store/src/main/java/org/geoserver/csw/store/internal

I still think that having the store in its own module would look better (do we have any default
datastore or coveragestore in GeoServer? even the ones used by core can be plugged out),
but I'm not opposed to seeing this get merged in as the default one like we did for the security subsystem.

I guess the difference is on whether is it normal to have a store 100% of the time in, with little
likeliness of being replaced by another, or not.
Right now it certainly seems like it will be most of the time in, but if we grow an implementation
that can actually handle efficiently large number of records detached from the GS own
catalog things might change.

Cheers
Andrea

--

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

11

On Mon, Dec 10, 2012 at 12:28 PM, Niels Charlier <niels@anonymised.com> wrote:

Okay so I am going to conclude that we go with Justin’s proposal to put the internal store in to core. The other thing is, should I keep its name or rename it to DefaultCatalogStore?

I’d use a name making it evident the info comes from GeoServer configuration itself, what is used
as the “default” could be changing over time.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it