[Geoserver-devel] Some considerations on cascading WFS

[Andrea_Aime3]

Hi,
so I was looking into integrating cascading WFS for
good. The first issue with cascading WFS using the
WFS datastore was the inability to cascade Geoserver
itself.

The issue is that we end up building invalid GML,
because we do name the layers as prefix:layerName,
and cascading another Geoserver the layer name would
be prefix:otherPrefix:originalLayerName, which is invalid.

The first temptation would be to strip away the original prefix,
but that would not work, we could end up with multiple
layers with the same name (if the original GeoServer had
the same name in different namespaces).

A working alternative would be to strip the : or replace
it with an _. Since this transform is not invertible (the
original name could have an _ already in other parts of
the name) a map from the new to old name would be needed.
This map could live in the WFS datastore itself, and would
have to be pervasive because, like most other datastores,
there is a strong assumption the original name and the
external feature type name are equal.

Yet, thinking about it, I'm not sure it would be the best
place. I mean, there is nothing in the Geotools API stating
that feature type names cannot contain a colon, it's just that
we are using the original file/table name and both the filesystem
and table identifiers ":" is forbidden.

An alternative solution could be to create a wrapper datastore
that alter the feature type names and keeps the map by itself.
This would be cleaner, but not trivial, since one would have
to tap the DataStore, FeatureSource/Store/Locking, and the FeatureCollections (and I may be probably forgetting something).

Any brilliant idea out there?
Cheers
Andrea

[David_Zwiers]

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string … this would certainly fix GS’s issues as you should only ever publish one dataset per namespace.

While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

David

On 8/3/07, Andrea Aime <aaime@anonymised.com> wrote:

Hi,
so I was looking into integrating cascading WFS for
good. The first issue with cascading WFS using the
WFS datastore was the inability to cascade Geoserver
itself.

The issue is that we end up building invalid GML,
because we do name the layers as prefix:layerName,
and cascading another Geoserver the layer name would
be prefix:otherPrefix:originalLayerName, which is invalid.

The first temptation would be to strip away the original prefix,
but that would not work, we could end up with multiple
layers with the same name (if the original GeoServer had
the same name in different namespaces).

A working alternative would be to strip the : or replace
it with an _. Since this transform is not invertible (the
original name could have an _ already in other parts of
the name) a map from the new to old name would be needed.
This map could live in the WFS datastore itself, and would
have to be pervasive because, like most other datastores,
there is a strong assumption the original name and the
external feature type name are equal.

Yet, thinking about it, I’m not sure it would be the best
place. I mean, there is nothing in the Geotools API stating
that feature type names cannot contain a colon, it’s just that
we are using the original file/table name and both the filesystem
and table identifiers “:” is forbidden.

An alternative solution could be to create a wrapper datastore
that alter the feature type names and keeps the map by itself.
This would be cleaner, but not trivial, since one would have
to tap the DataStore, FeatureSource/Store/Locking, and the
FeatureCollections (and I may be probably forgetting something).

Any brilliant idea out there?
Cheers
Andrea

---

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

---

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

[Justin_Deoliveira]

David Zwiers wrote:

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.

-Justin

David

On 8/3/07, *Andrea Aime* <aaime@anonymised.com <mailto:aaime@anonymised.com>> wrote:

    Hi,
    so I was looking into integrating cascading WFS for
    good. The first issue with cascading WFS using the
    WFS datastore was the inability to cascade Geoserver
    itself.

    The issue is that we end up building invalid GML,
    because we do name the layers as prefix:layerName,
    and cascading another Geoserver the layer name would
    be prefix:otherPrefix:originalLayerName, which is invalid.

    The first temptation would be to strip away the original prefix,
    but that would not work, we could end up with multiple
    layers with the same name (if the original GeoServer had
    the same name in different namespaces).

    A working alternative would be to strip the : or replace
    it with an _. Since this transform is not invertible (the
    original name could have an _ already in other parts of
    the name) a map from the new to old name would be needed.
    This map could live in the WFS datastore itself, and would
    have to be pervasive because, like most other datastores,
    there is a strong assumption the original name and the
    external feature type name are equal.

    Yet, thinking about it, I'm not sure it would be the best
    place. I mean, there is nothing in the Geotools API stating
    that feature type names cannot contain a colon, it's just that
    we are using the original file/table name and both the filesystem
    and table identifiers ":" is forbidden.

    An alternative solution could be to create a wrapper datastore
    that alter the feature type names and keeps the map by itself.
    This would be cleaner, but not trivial, since one would have
    to tap the DataStore, FeatureSource/Store/Locking, and the
    FeatureCollections (and I may be probably forgetting something).

    Any brilliant idea out there?
    Cheers
    Andrea

    -------------------------------------------------------------------------
    This SF.net email is sponsored by: Splunk Inc.
    Still grepping through log files to find problems? Stop.
    Now Search log events and configuration files using AJAX and a browser.
    Download your FREE copy of Splunk now >> http://get.splunk.com/
    _______________________________________________
    Geoserver-devel mailing list
    Geoserver-devel@lists.sourceforge.net
    <mailto:Geoserver-devel@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,46b3620a93634901796417!

------------------------------------------------------------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

!DSPAM:4007,46b3620a93634901796417!

------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,46b3620a93634901796417!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org

[Andrea_Aime3]

Justin Deoliveira ha scritto:

David Zwiers wrote:

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.

Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

[Stefan_Hansen]

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

cheers,
stefan

Andrea Aime wrote:

Justin Deoliveira ha scritto:
  

David Zwiers wrote:
    

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.
    
Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

[Rob_Atkinson]

Having had some experience with aggregating WFS instances I have a few perspectives it would be interesting to hear your plans for...

There are three bases for aggregation:

1) aggregate multiple identical services - same query, same response
2) aggregate multiple "similar" services - think of Interfaces and polymorphism - same Filter, query wrapped in different FeatureType
3) aggregate disparate services, with all the attendant schema translation and ontology mapping required.

I know several people looking at the third problem, which is characterised by geoserver's current capability (Feature Type is bound to the persistence layer - different database, different query). The problem is a massive one, which involves complex governance rules for publishing the ad-hoc schemas from WFS and the ontologies and schema translation rules required to support it. If you venture this way, good luck! I think we want to do it one day, but not to aggregate indivual instances but to bridge common WFS from one community to multiple clients in an another community (i.e. the publicaton of the content standards is already solved, communities just need to agree on a mechanism to describe the mappings and ontologies.)

The much simpler approach, I can hear the groans already, is to make geoserver capable of delivering standard data against standard queries. This capability is in progress. Spike solutions exist as unsupported community modules. (community-schema-ds)

The best approach is #2, which extends #1 but requires an additional mechanism for polymorphism. This is called a Feature Type Catalogue - talk to Jim Groffen at Lisasoft for more info.

The next issue is one of error propagation:

how do you intend to make the process robust in the case of permanent or intermittent failure, performance issues, change in compliance levels etc of a single source? My thoughts were running to some concept similar to connection pooling, possibly with failover handling. In any event robnust error reporting would be critical.

Rob Atkinson

Stefan Hansen wrote:

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

cheers,
stefan

Andrea Aime wrote:
  

Justin Deoliveira ha scritto:
  

David Zwiers wrote:
    

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.
    

Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  

[Andrea_Aime3]

Stefan Hansen ha scritto:

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

Happy to be of help

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

Rob has already covered the only thing that came to my mind (and
covered it better than I could anyways).
Oh, maybe I can add one performance consideration and an open problem too. If you're hitting multiple servers, chances are that you can spawn
multiple threads hitting each separate server and have a synchronized
queue to serve the aggregated answer... this should work provided you
are not dealing with a query with a sort clause (WFS 1.1 has sorting,
but I guess you're using WFS 1.0, right?). If you have to deal with
sorting, then you can do some a mergesort by reading one feature from
each server and compare to the others, and so on until you exhaust
all responses.

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

Well, if I remember properly your case is aggregating backend servers with identical schemas, right? I don't know how common this problem is,
but for sure you could try and include your datastore as a gt2 unsupported module, just go to the gt2-devel list and ask for the module
to be created. I would be happy to have a look at the source code anyways.

Cheers
Andrea

[Stefan_Hansen]

Hey!

Andrea Aime wrote:

Stefan Hansen ha scritto:

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

Happy to be of help

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

Rob has already covered the only thing that came to my mind (and
covered it better than I could anyways).

Hmmm... maybe I missed an email? What did Rob say?

Oh, maybe I can add one performance consideration and an open problem too. If you're hitting multiple servers, chances are that you can spawn
multiple threads hitting each separate server and have a synchronized
queue to serve the aggregated answer... this should work provided you
are not dealing with a query with a sort clause (WFS 1.1 has sorting,
but I guess you're using WFS 1.0, right?). If you have to deal with
sorting, then you can do some a mergesort by reading one feature from
each server and compare to the others, and so on until you exhaust
all responses.

So far I only access all data stores sequentially, but using threads is already on my todo-list.
At the moment we are using only WFS 1.0, but I think we will switch to 1.1 as soon as there is WFS 1.1 data store available. So I will keep your comments in mind.

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

Well, if I remember properly your case is aggregating backend servers with identical schemas, right? I don't know how common this problem is,
but for sure you could try and include your datastore as a gt2 unsupported module, just go to the gt2-devel list and ask for the module
to be created. I would be happy to have a look at the source code anyways.

I only have to deal with identical schemas. So my implementation of an aggregating data store will be limited to this special case and therefore not useful for everybody. But what I was actually thinking of was, that if someone is really desperate to use the WFS-1.0 data store with Geoserver 1.6, then this person could use my AggregationDataStore, configure it for only one WFS-data store and it should work.

However, I will try to get an unsupported module for my data store and it would be great to get some feedback on my code.

Thanks for your help!
stefan

[Andrea_Aime3]

Stefan Hansen ha scritto:
...

Rob has already covered the only thing that came to my mind (and
covered it better than I could anyways).

Hmmm... maybe I missed an email? What did Rob say?

The mail about the issues in the aggregation of WFS with different schemas. Not your problem as you stated

I only have to deal with identical schemas. So my implementation of an aggregating data store will be limited to this special case and therefore not useful for everybody. But what I was actually thinking of was, that if someone is really desperate to use the WFS-1.0 data store with Geoserver 1.6, then this person could use my AggregationDataStore, configure it for only one WFS-data store and it should work.

Well, I may be desperate then. Seriously, I have to create the wrapper
that changes the feature type name, and I need it to work both read
and write side, so having your code to look at as a reference could at least speed up my work
Can I have a copy by private mail?

Cheers
Andrea

[Rob_Atkinson]

Andrea,

please consider the CommunitySchema datastore - its heavyweight complete solution, but it does exactly that, and allows correct GML as well (features can correctly have the critical gml:name identifier property.

Creating yet another interim hack solution simply creates more and more complexity in the code base. If Justin has successfully integrated the correct feature model a man of your skills would be able to get this working very quickly, and we would have a platform to build from.

i.e, changing the name is simply a trivial mapping, you just have to add a strategy to map each attribute automatically if a flag is set, instead of reading the target schema to create the feature type.

Logically, an aggregate datastore sits on top of the MappingFeatureIterator from the community-schema-ds, it would be a tragedy to recreate this functionality.

if you need to map, you need to use the mapping datastroe, not a new code base that partially implements it. Even if you hack one up, plug-and-play with the full implementation is required, so we should avoid creating it around the old API - it will need to use an ISOFeature capable API from the start, to support namespace translation.

Rob

Andrea Aime wrote:

Stefan Hansen ha scritto:
...
  

Rob has already covered the only thing that came to my mind (and
covered it better than I could anyways).
      

Hmmm... maybe I missed an email? What did Rob say?
    
The mail about the issues in the aggregation of WFS with different schemas. Not your problem as you stated

I only have to deal with identical schemas. So my implementation of an aggregating data store will be limited to this special case and therefore not useful for everybody. But what I was actually thinking of was, that if someone is really desperate to use the WFS-1.0 data store with Geoserver 1.6, then this person could use my AggregationDataStore, configure it for only one WFS-data store and it should work.
    
Well, I may be desperate then. Seriously, I have to create the wrapper
that changes the feature type name, and I need it to work both read
and write side, so having your code to look at as a reference could at least speed up my work
Can I have a copy by private mail?

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  

[Justin_Deoliveira]

While I agree with you Rob about short terms hacks are undesirable I
don't see much of an alternative. The new feature model is still very
far off from being usable in a production GeoServer release.

Rob Atkinson wrote:

Andrea,

please consider the CommunitySchema datastore - its heavyweight complete
solution, but it does exactly that, and allows correct GML as well
(features can correctly have the critical gml:name identifier property.

Creating yet another interim hack solution simply creates more and more
complexity in the code base. If Justin has successfully integrated the
correct feature model a man of your skills would be able to get this
working very quickly, and we would have a platform to build from.

i.e, changing the name is simply a trivial mapping, you just have to
add a strategy to map each attribute automatically if a flag is set,
instead of reading the target schema to create the feature type.

Logically, an aggregate datastore sits on top of the
MappingFeatureIterator from the community-schema-ds, it would be a
tragedy to recreate this functionality.

if you need to map, you need to use the mapping datastroe, not a new
code base that partially implements it. Even if you hack one up,
plug-and-play with the full implementation is required, so we should
avoid creating it around the old API - it will need to use an ISOFeature
capable API from the start, to support namespace translation.

Rob

Andrea Aime wrote:

Stefan Hansen ha scritto:
...

Rob has already covered the only thing that came to my mind (and
covered it better than I could anyways).
      

Hmmm... maybe I missed an email? What did Rob say?
    
The mail about the issues in the aggregation of WFS with different
schemas. Not your problem as you stated

I only have to deal with identical schemas. So my implementation of
an aggregating data store will be limited to this special case and
therefore not useful for everybody. But what I was actually thinking
of was, that if someone is really desperate to use the WFS-1.0 data
store with Geoserver 1.6, then this person could use my
AggregationDataStore, configure it for only one WFS-data store and it
should work.
    
Well, I may be desperate then. Seriously, I have to create the wrapper
that changes the feature type name, and I need it to work both read
and write side, so having your code to look at as a reference could at
least speed up my work
Can I have a copy by private mail?

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  
!DSPAM:4007,46bad8b2307633327367457!

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

!DSPAM:4007,46bad8b2307633327367457!

------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,46bad8b2307633327367457!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org

[Stefan_Hansen]

Thanks for your comments!

It took our mail server a few hours to pass it on to me, so I got a bit confused with Andrea referring to your response...

Anyway, I think theoretically I have to deal with the first approach. The services are supposed to be identical. However, I have to deal with a whole bunch of different servers providing WFS (Mapserver, Geoserver, Cubwerx). Geoserver adds a prefix, Mapserver doesn't and already the FeatureTypeNames are not really identical anymore, so I have deal with them slightly differently... and that's probably not the only problem I will have and not the last time I bother you with my problems...

Thanks,
stefan

Rob Atkinson wrote:

Having had some experience with aggregating WFS instances I have a few perspectives it would be interesting to hear your plans for...

There are three bases for aggregation:

1) aggregate multiple identical services - same query, same response
2) aggregate multiple "similar" services - think of Interfaces and polymorphism - same Filter, query wrapped in different FeatureType
3) aggregate disparate services, with all the attendant schema translation and ontology mapping required.

I know several people looking at the third problem, which is characterised by geoserver's current capability (Feature Type is bound to the persistence layer - different database, different query). The problem is a massive one, which involves complex governance rules for publishing the ad-hoc schemas from WFS and the ontologies and schema translation rules required to support it. If you venture this way, good luck! I think we want to do it one day, but not to aggregate indivual instances but to bridge common WFS from one community to multiple clients in an another community (i.e. the publicaton of the content standards is already solved, communities just need to agree on a mechanism to describe the mappings and ontologies.)

The much simpler approach, I can hear the groans already, is to make geoserver capable of delivering standard data against standard queries. This capability is in progress. Spike solutions exist as unsupported community modules. (community-schema-ds)

The best approach is #2, which extends #1 but requires an additional mechanism for polymorphism. This is called a Feature Type Catalogue - talk to Jim Groffen at Lisasoft for more info.

The next issue is one of error propagation:

how do you intend to make the process robust in the case of permanent or intermittent failure, performance issues, change in compliance levels etc of a single source? My thoughts were running to some concept similar to connection pooling, possibly with failover handling. In any event robnust error reporting would be critical.

Rob Atkinson

Stefan Hansen wrote:

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

cheers,
stefan

Andrea Aime wrote:

Justin Deoliveira ha scritto:
     

David Zwiers wrote:
         

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.
          

Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

-------------------------------------------------------------------------

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  

[Andrea_Aime3]

Stefan Hansen ha scritto:

Thanks for your comments!

It took our mail server a few hours to pass it on to me, so I got a bit confused with Andrea referring to your response...

Anyway, I think theoretically I have to deal with the first approach. The services are supposed to be identical. However, I have to deal with a whole bunch of different servers providing WFS (Mapserver, Geoserver, Cubwerx).

Hum, interesting. Are there public WFS? Can I have some links?
Cheers
Andrea

[Rob_Atkinson]

Stefan Hansen wrote:

Thanks for your comments!

It took our mail server a few hours to pass it on to me, so I got a bit confused with Andrea referring to your response...

Anyway, I think theoretically I have to deal with the first approach. The services are supposed to be identical.

i dont think they can be, except by a very fragile manipulation of broken behaviour from the technologies.

Geoserver and Mapserver expose the underlying persistence schema in an arbitrary fashion. This is not a basis for creating identical services.

The killer will be if you cannot handle the most important property of any feature - gml:name correctly. gml:name is the identifier of the object (gid is only a local identifier and cannot be used to reference the object). If you cant maintain gml:name in the gml namespace correctly then you do not have a valid GML feature type.

Rob

However, I have to deal with a whole bunch of different servers providing WFS (Mapserver, Geoserver, Cubwerx). Geoserver adds a prefix, Mapserver doesn't and already the FeatureTypeNames are not really identical anymore, so I have deal with them slightly differently... and that's probably not the only problem I will have and not the last time I bother you with my problems...

Thanks,
stefan

Rob Atkinson wrote:
  

Having had some experience with aggregating WFS instances I have a few perspectives it would be interesting to hear your plans for...

There are three bases for aggregation:

1) aggregate multiple identical services - same query, same response
2) aggregate multiple "similar" services - think of Interfaces and polymorphism - same Filter, query wrapped in different FeatureType
3) aggregate disparate services, with all the attendant schema translation and ontology mapping required.

I know several people looking at the third problem, which is characterised by geoserver's current capability (Feature Type is bound to the persistence layer - different database, different query). The problem is a massive one, which involves complex governance rules for publishing the ad-hoc schemas from WFS and the ontologies and schema translation rules required to support it. If you venture this way, good luck! I think we want to do it one day, but not to aggregate indivual instances but to bridge common WFS from one community to multiple clients in an another community (i.e. the publicaton of the content standards is already solved, communities just need to agree on a mechanism to describe the mappings and ontologies.)

The much simpler approach, I can hear the groans already, is to make geoserver capable of delivering standard data against standard queries. This capability is in progress. Spike solutions exist as unsupported community modules. (community-schema-ds)

The best approach is #2, which extends #1 but requires an additional mechanism for polymorphism. This is called a Feature Type Catalogue - talk to Jim Groffen at Lisasoft for more info.

The next issue is one of error propagation:

how do you intend to make the process robust in the case of permanent or intermittent failure, performance issues, change in compliance levels etc of a single source? My thoughts were running to some concept similar to connection pooling, possibly with failover handling. In any event robnust error reporting would be critical.

Rob Atkinson

Stefan Hansen wrote:
    

Hi!

We are developing at the moment a data store that forwards a request to multiple other data stores and aggregates their responses. Since this data store is supposed to aggregate mainly WFS-servers (not only, but also Geoservers), I was thinking of how to solve the problems the of the WFS data store cascading a Geoserver as well, when I read your mails discussing issue. Since I hadn't come up with my own solution, so I simply took one of yours (Thanks! ).

For our purposes the easiest option was to implement a wrapper data store. I simply had to some changes in my aggregation data store...

However, we have now a data store running that solves the issues with the WFS-data store (by replacing the prefix). It's rather simple, it only supports read-only access (which is all we need in our project) and it has not really been tested yet, but so far it seems to work. I'm not too familiar yet with the whole WFS-thing, so I was wondering, if you can think any special cases which might be problematic for our solution and which I should consider?

I'm not sure of how much interest this is for you, since our implementation is very limited so far (read-only) and we're not planning at the moment to add writing-access. But it is working for Geoserver 1.6 and if someone doesn't want to wait for version 1.7 to cascade a WFS-server, we are happy to commit our data store.

cheers,
stefan

Andrea Aime wrote:

Justin Deoliveira ha scritto:
     

David Zwiers wrote:
         

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.
          

Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

-------------------------------------------------------------------------

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

-------------------------------------------------------------------------

This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  

[jive]

The class Justin refers to (called "Name") is available in GeoTools now, if you produced a Name from the information DataStore has (ie URI and typeName) would that accomplish the same effect.
Jody

Justin Deoliveira ha scritto:
  

David Zwiers wrote:
    

I remember thinking about this issue a fair bit. At the time I was in favour of altering the FeatureType.Name to be a QName rather than a string ... this would certainly fix GS's issues as you should only ever publish one dataset per namespace.
While there are a number of arguments against QName (for example how to handle database instances), but in each case the issue could be handled by a default datastore namespace, or a hint passed in at creation.

Yeah... this would probably the cleanest and best solution to the problem... and also in the geotools feature model there is now a class which is identical to the xml QName.. its used in the new feature model.
    

Yeah, but that would delay the usability of the WFS datastore up
to Geoserver 1.7.something.... I think the plan was to have it running
in 1.6.0.

Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel