[Geoserver-devel] Feeding metadata to DataStores

GeoServer GeoServer Developer
1

Hi,

The XStreamPersister Andrea designed is working great for me after I made the small change I mentioned a few hours ago. But this still leaves part of the problem of feeding featuretype.xml configuration to DataStores / FeatureSources.

The parsed metadata is part of the GS catalog FeatureTypeInfo and currently, I’m not aware of any mechanism to pass the metadata to the actual GT feature source.

Here’s a few approches I’ve thought of so far:

  1. add a MetadataAware interface into GT:

public interface MetadataAware {
public void setMetadata(Map<String, ?> metadata);
pblic Map<String, ?> getMetadata();
}

The setter would be called for FeatureSources which implement this interface when the FeatureSource is created, in FeatureTypeInfoImpl.getFeatureSource(). This is however clumsy as the FeatureSources are more often than not wrapped in other FeatureSources. And as there is no common abstract class, it’s a lot of work implementing this for every wrapping FeatureSource, not to mention error-prone when new wrappers are written.

  1. Add similar methods to SimpleFeatureSource. Most classes implement this anyway, so it would negate the issue of forgetting to implement an extra interface for FeatureSource implementations and wrappers.

  2. Add the featuretype metadata in the GT FeatureType.userData. This is a crufty as this information is not really to do with the feature type, but the current layer based on this feature type. There’s a strong possibility of things getting mixed up in situations where multiple layers are created from the same feature type.

Thoughts? Any better ideas that come to mind?

Sampo

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

2

One additional strategy comes to mind. The metadata could be put in the Query hints. A reference to the complete metadata map could be injected as a single entry in the query hints. This would require little or no refactoring. All that is required is a common entry point for queries where this hint should be injected at.

Sampo

···

On Fri, Apr 25, 2014 at 12:33 PM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:

Hi,

The XStreamPersister Andrea designed is working great for me after I made the small change I mentioned a few hours ago. But this still leaves part of the problem of feeding featuretype.xml configuration to DataStores / FeatureSources.

The parsed metadata is part of the GS catalog FeatureTypeInfo and currently, I’m not aware of any mechanism to pass the metadata to the actual GT feature source.

Here’s a few approches I’ve thought of so far:

  1. add a MetadataAware interface into GT:

public interface MetadataAware {
public void setMetadata(Map<String, ?> metadata);
pblic Map<String, ?> getMetadata();
}

The setter would be called for FeatureSources which implement this interface when the FeatureSource is created, in FeatureTypeInfoImpl.getFeatureSource(). This is however clumsy as the FeatureSources are more often than not wrapped in other FeatureSources. And as there is no common abstract class, it’s a lot of work implementing this for every wrapping FeatureSource, not to mention error-prone when new wrappers are written.

  1. Add similar methods to SimpleFeatureSource. Most classes implement this anyway, so it would negate the issue of forgetting to implement an extra interface for FeatureSource implementations and wrappers.

  2. Add the featuretype metadata in the GT FeatureType.userData. This is a crufty as this information is not really to do with the feature type, but the current layer based on this feature type. There’s a strong possibility of things getting mixed up in situations where multiple layers are created from the same feature type.

Thoughts? Any better ideas that come to mind?

Sampo

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

3

On Fri, Apr 25, 2014 at 11:33 AM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Hi,

The XStreamPersister Andrea designed is working great for me after I made
the small change I mentioned a few hours ago. But this still leaves part of
the problem of feeding featuretype.xml configuration to DataStores /
FeatureSources.

Good to hear. Btw, the XStreamPersister is Justin's work, I just did the
initializers plugins :slight_smile:

The parsed metadata is part of the GS catalog FeatureTypeInfo and
currently, I'm not aware of any mechanism to pass the metadata to the
actual GT feature source.

Here's a few approches I've thought of so far:

1. add a MetadataAware interface into GT:

public interface MetadataAware {
  public void setMetadata(Map<String, ?> metadata);
  pblic Map<String, ?> getMetadata();
}

The setter would be called for FeatureSources which implement this
interface when the FeatureSource is created, in
FeatureTypeInfoImpl.getFeatureSource(). This is however clumsy as the
FeatureSources are more often than not wrapped in other FeatureSources. And
as there is no common abstract class, it's a lot of work implementing this
for every wrapping FeatureSource, not to mention error-prone when new
wrappers are written.

Yes, that's a common issue as we wrap feature sources a lot.

2. Add similar methods to SimpleFeatureSource. Most classes implement this
anyway, so it would negate the issue of forgetting to implement an extra
interface for FeatureSource implementations and wrappers.

Hmm... still fragile, not all wrappers might be aware of it, and it's
limited to simple features (like it or not, complex features are going to
grow).

3. Add the featuretype metadata in the GT FeatureType.userData. This is a
crufty as this information is not really to do with the feature type, but
the current layer based on this feature type. There's a strong possibility
of things getting mixed up in situations where multiple layers are created
from the same feature type.

Except you need a feature type to start with.

Thoughts? Any better ideas that come to mind?

I would generalize and externalize what we are currently doing for
JDBCDataStores in ResourcePool:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L872
https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L844
https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L1112

That is, create a FeatureTypeInitializer interface that we can use to
delegate all types of special initializations, the
interface woudl act as a broker between the programmatic interfaces
provided by GeoTools and the configuration
stored in the feature type metadata maps

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

4

On Fri, Apr 25, 2014 at 3:18 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Fri, Apr 25, 2014 at 11:33 AM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Hi,

The XStreamPersister Andrea designed is working great for me after I made
the small change I mentioned a few hours ago. But this still leaves part of
the problem of feeding featuretype.xml configuration to DataStores /
FeatureSources.

Good to hear. Btw, the XStreamPersister is Justin's work, I just did the
initializers plugins :slight_smile:

The parsed metadata is part of the GS catalog FeatureTypeInfo and
currently, I'm not aware of any mechanism to pass the metadata to the
actual GT feature source.

Here's a few approches I've thought of so far:

1. add a MetadataAware interface into GT:

public interface MetadataAware {
  public void setMetadata(Map<String, ?> metadata);
  pblic Map<String, ?> getMetadata();
}

The setter would be called for FeatureSources which implement this
interface when the FeatureSource is created, in
FeatureTypeInfoImpl.getFeatureSource(). This is however clumsy as the
FeatureSources are more often than not wrapped in other FeatureSources. And
as there is no common abstract class, it's a lot of work implementing this
for every wrapping FeatureSource, not to mention error-prone when new
wrappers are written.

Yes, that's a common issue as we wrap feature sources a lot.

2. Add similar methods to SimpleFeatureSource. Most classes implement
this anyway, so it would negate the issue of forgetting to implement an
extra interface for FeatureSource implementations and wrappers.

Hmm... still fragile, not all wrappers might be aware of it, and it's
limited to simple features (like it or not, complex features are going to
grow).

3. Add the featuretype metadata in the GT FeatureType.userData. This is a
crufty as this information is not really to do with the feature type, but
the current layer based on this feature type. There's a strong possibility
of things getting mixed up in situations where multiple layers are created
from the same feature type.

Except you need a feature type to start with.

Thoughts? Any better ideas that come to mind?

I would generalize and externalize what we are currently doing for
JDBCDataStores in ResourcePool:

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L872

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L844

https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L1112

That is, create a FeatureTypeInitializer interface that we can use to
delegate all types of special initializations, the
interface woudl act as a broker between the programmatic interfaces
provided by GeoTools and the configuration
stored in the feature type metadata maps

Generalizing this is definitely one way to go. I'm a bit wary of starting
doing that, especially since from the looks of it, the SQL view
implementation looks a bit iffy. See the third line you quoted:
  if(dataStore instanceof JDBCDataStore && info.getMetadata() != null &&

info.getMetadata().containsKey(FeatureTypeInfo.JDBC_VIRTUAL_TABLE)) {
    ...

The dataStore object might already be wrapped! This means this code does
not work in every case.

What do you think about my later suggestion that we could add the metadata
as a Query hint? I'm now prototyping it so that GeoServerFeatureSource is
given the metadata map, which it then injects into Queries in
getFeatures(). It looks promising from my point of view.

Sampo

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

5

On Fri, Apr 25, 2014 at 2:47 PM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Generalizing this is definitely one way to go. I'm a bit wary of starting

doing that, especially since from the looks of it, the SQL view
implementation looks a bit iffy. See the third line you quoted:

  if(dataStore instanceof JDBCDataStore && info.getMetadata() != null &&

info.getMetadata().containsKey(FeatureTypeInfo.JDBC_VIRTUAL_TABLE)) {
    ...

The dataStore object might already be wrapped! This means this code does
not work in every case.

We don't wrap stores at the ResourcePool level, if we do, it's happening
higher level in the security
wrappers. ResourcePool gets the stores from the factories directly, there
are no intermediaries that
I'm aware of, just check the getDataStore code paths.

What do you think about my later suggestion that we could add the metadata
as a Query hint? I'm now prototyping it so that GeoServerFeatureSource is
given the metadata map, which it then injects into Queries in
getFeatures(). It looks promising from my point of view.

For it to work, the feature type must to exist already, and the wrapping
must not change it
(since query hints are provided only when accessing data, not when asking
for metadata).
I'd rather use a mechanism that is not making these assumptions, they make
the use
case pretty narrow.

I could go with it, if you can show the modification to support this case
are minimal.
I don't like it because no other store works that way, so we'd be left with
a special
case for virtual stores, plus one that is likely going to be used by wfs-ng
only.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

6

On Fri, Apr 25, 2014 at 4:21 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Fri, Apr 25, 2014 at 2:47 PM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Generalizing this is definitely one way to go. I'm a bit wary of starting

doing that, especially since from the looks of it, the SQL view
implementation looks a bit iffy. See the third line you quoted:

  if(dataStore instanceof JDBCDataStore && info.getMetadata() != null &&

info.getMetadata().containsKey(FeatureTypeInfo.JDBC_VIRTUAL_TABLE)) {
    ...

The dataStore object might already be wrapped! This means this code does
not work in every case.

We don't wrap stores at the ResourcePool level, if we do, it's happening
higher level in the security
wrappers. ResourcePool gets the stores from the factories directly, there
are no intermediaries that
I'm aware of, just check the getDataStore code paths.

When I debugged this, my dataStore was wrapped as a RetypingDataStore.
Check DataStoreUtils.getDataAccess()

What do you think about my later suggestion that we could add the metadata

as a Query hint? I'm now prototyping it so that GeoServerFeatureSource is
given the metadata map, which it then injects into Queries in
getFeatures(). It looks promising from my point of view.

For it to work, the feature type must to exist already, and the wrapping
must not change it
(since query hints are provided only when accessing data, not when asking
for metadata).
I'd rather use a mechanism that is not making these assumptions, they make
the use
case pretty narrow.

As this would be a mechanism to provide extension specific metadata to
configured features, it would make sense that the FeatureTypeInfo should
exist when this mechanism is used. Also, GeoServerFeatureSource.create() is
used only in ResourcePool.getFeatureSource() - which is definitely a case
where the feature type exists.

I could go with it, if you can show the modification to support this case
are minimal.
I don't like it because no other store works that way, so we'd be left
with a special
case for virtual stores, plus one that is likely going to be used by
wfs-ng only.

Yeah. We don't want to add something that will only be used for wfs-ng. I
would however argue, that the reason nobody else has yet wondered about
this problem is because before pluggable XStreamParsersInitializers, the
extensions were themselves in charge of configuration files and they didn't
have to to pipe their config from pure generic GS to GT. Now that the
pluggable mechanism makes this a possibility, I see a strong case that
other extensions might want to opt for something like this.

Here are the relevant lines of code that I've changed when trying this
model out.

Add metadata to GeoServerFeatureSources:
https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L104
https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L116
https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L159
https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureLocking.java#L54

Inject the metadata to Query objects:
https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L396

Grab the configuration in wfs-ng:
https://github.com/sampov2/geotools/blob/258b099a5b9174d1b3eeca69a6dd98754cc4815d/modules/unsupported/wfs-ng/src/main/java/org/geotools/data/wfs/internal/v2_0/StrictWFS_2_0_Strategy.java#L367
https://github.com/sampov2/geotools/blob/258b099a5b9174d1b3eeca69a6dd98754cc4815d/modules/unsupported/wfs-ng/src/main/java/org/geotools/data/wfs/internal/v2_0/StrictWFS_2_0_Strategy.java#L454

I'd say the changes are quite contained.

However, like you pointed out, it doesn't cover other cases (than queries)
where interaction between GS and GT requires the GT side of things to have
the configuration.

Sampo

--
Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

7

On Fri, Apr 25, 2014 at 4:16 PM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

We don't wrap stores at the ResourcePool level, if we do, it's happening

higher level in the security
wrappers. ResourcePool gets the stores from the factories directly, there
are no intermediaries that
I'm aware of, just check the getDataStore code paths.

When I debugged this, my dataStore was wrapped as a RetypingDataStore.
Check DataStoreUtils.getDataAccess()

Ah, you're right, forgot about it, your store must be emitting invalid
names (names with : inside), something that indeed
with JDBC store is possible, but very unlikely.

I believe this could be worked by removing that wrapping and making GUI and
REST config more strict about
the names they are accepting, but I haven't checked how much work that will
be.

What do you think about my later suggestion that we could add the

metadata as a Query hint? I'm now prototyping it so that
GeoServerFeatureSource is given the metadata map, which it then injects
into Queries in getFeatures(). It looks promising from my point of view.

For it to work, the feature type must to exist already, and the wrapping
must not change it
(since query hints are provided only when accessing data, not when asking
for metadata).
I'd rather use a mechanism that is not making these assumptions, they
make the use
case pretty narrow.

As this would be a mechanism to provide extension specific metadata to
configured features, it would make sense that the FeatureTypeInfo should
exist when this mechanism is used. Also, GeoServerFeatureSource.create() is
used only in ResourcePool.getFeatureSource() - which is definitely a case
where the feature type exists.

Nope, ResourcePool.getFeatureSource creates the feature type for virtual
tables, the feature type does not exist in the store before
we call addVirtualTable.

I could go with it, if you can show the modification to support this case
are minimal.
I don't like it because no other store works that way, so we'd be left
with a special
case for virtual stores, plus one that is likely going to be used by
wfs-ng only.

Yeah. We don't want to add something that will only be used for wfs-ng. I
would however argue, that the reason nobody else has yet wondered about
this problem is because before pluggable XStreamParsersInitializers, the
extensions were themselves in charge of configuration files and they didn't
have to to pipe their config from pure generic GS to GT. Now that the
pluggable mechanism makes this a possibility, I see a strong case that
other extensions might want to opt for something like this.

I don't buy this, personally I don't remember anybody ever asking or
experssing interest on the subject. The way I see it, stored queries are
pretty unique to the WFS 2.0 specification. Do you know of other data
sources in the wild that limit access to the store the same way? Make me
just a single example and I'll be with you about supporting this use case.

Anyways, the mechanism I talked about will be created the day I find time
to introduce feature type restructuring via TransformFeatureSource it will
be needed (but it will be my problem, along with fixing DataStoreUtils).

Here are the relevant lines of code that I've changed when trying this
model out.

Add metadata to GeoServerFeatureSources:

https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L104

https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L116

https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L159

https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureLocking.java#L54

Inject the metadata to Query objects:

https://github.com/sampov2/geoserver/blob/22e01d28f13d536c1fcc1657a6a8fa0716489bb8/src/main/src/main/java/org/vfny/geoserver/global/GeoServerFeatureSource.java#L396

Grab the configuration in wfs-ng:

https://github.com/sampov2/geotools/blob/258b099a5b9174d1b3eeca69a6dd98754cc4815d/modules/unsupported/wfs-ng/src/main/java/org/geotools/data/wfs/internal/v2_0/StrictWFS_2_0_Strategy.java#L367

https://github.com/sampov2/geotools/blob/258b099a5b9174d1b3eeca69a6dd98754cc4815d/modules/unsupported/wfs-ng/src/main/java/org/geotools/data/wfs/internal/v2_0/StrictWFS_2_0_Strategy.java#L454

I'd say the changes are quite contained.

Can we see a diff? Looking for the changes inspecting the source code is
pretty inefficient.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

8

On Fri, Apr 25, 2014 at 5:34 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Fri, Apr 25, 2014 at 4:16 PM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

We don't wrap stores at the ResourcePool level, if we do, it's

happening higher level in the security
wrappers. ResourcePool gets the stores from the factories directly,
there are no intermediaries that
I'm aware of, just check the getDataStore code paths.

When I debugged this, my dataStore was wrapped as a RetypingDataStore.
Check DataStoreUtils.getDataAccess()

Ah, you're right, forgot about it, your store must be emitting invalid
names (names with : inside), something that indeed
with JDBC store is possible, but very unlikely.

Ah yes. As I'm using the stored query ids as type names, which are uris, I
am emitting invalid names. Do you think this is will become a problem?
Should I remap them in the extension or is it fine to let GS create the
RetypingDataSource?

As this would be a mechanism to provide extension specific metadata to

configured features, it would make sense that the FeatureTypeInfo should
exist when this mechanism is used. Also, GeoServerFeatureSource.create() is
used only in ResourcePool.getFeatureSource() - which is definitely a case
where the feature type exists.

Nope, ResourcePool.getFeatureSource creates the feature type for virtual
tables, the feature type does not exist in the store before
we call addVirtualTable.

Ok. Fair enough. The counter-argument here is though, that in such cases
the user has not had a chance (yet) to create such configuration, so the
empty metadata is fine. It's more of a problem though that in the case of
JDBC Views, the data here would be the metadata of the "parent" feature
type, if I understood the design correctly.

Yeah. We don't want to add something that will only be used for wfs-ng. I

would however argue, that the reason nobody else has yet wondered about
this problem is because before pluggable XStreamParsersInitializers, the
extensions were themselves in charge of configuration files and they didn't
have to to pipe their config from pure generic GS to GT. Now that the
pluggable mechanism makes this a possibility, I see a strong case that
other extensions might want to opt for something like this.

I don't buy this, personally I don't remember anybody ever asking or
experssing interest on the subject. The way I see it, stored queries are
pretty unique to the WFS 2.0 specification. Do you know of other data
sources in the wild that limit access to the store the same way? Make me
just a single example and I'll be with you about supporting this use case.

Passing configuration from GS to an GT extension is not specific to wfs-ng
or other "limited access" sources. It should be easy to argue that many
data stores require more configuration than is provided in vanilla
featuretype.xml.

When I was asking about plugin configuration two weeks back, Jody said: "Other
data stores, such as pregeneralization, or app-schema, make use of their
own config files which are passed in as a datastore parameter.". You then
came up the idea of the pluggable XML parsers, which I read as a sign that
you were not 100% comfortable with the idea that data stores must take care
of the configuration files themselves. I then applied your model to store
configuration. What I'm now asking is how the configuration data the
parsers produce passed to the actual extension.

I'm not entirely sure what you're not buying here? Maybe we've accidentally
started talking past each other?

Anyways, the mechanism I talked about will be created the day I find time

to introduce feature type restructuring via TransformFeatureSource it will
be needed (but it will be my problem, along with fixing DataStoreUtils).

Great. I'll look forward to this refactor!

Can we see a diff? Looking for the changes inspecting the source code is
pretty inefficient.

https://github.com/sampov2/geoserver/commit/22e01d28f13d536c1fcc1657a6a8fa0716489bb8

https://github.com/sampov2/geotools/commit/258b099a5b9174d1b3eeca69a6dd98754cc4815d

(The GT commit is unfortunately a bit muddled up by a refactor of the
configuration beans. Just look at how the StoredQueryConfiguration object
is looked up)

S.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

9

Hi Andrea,

Do you have any further thoughts on this or should I come up with another strategy for passing the configuration objects?

Sampo

···

On Mon, Apr 28, 2014 at 9:03 AM, Sampo Savolainen <sampo.savolainen@anonymised.com.> wrote:

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

On Fri, Apr 25, 2014 at 5:34 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah yes. As I’m using the stored query ids as type names, which are uris, I am emitting invalid names. Do you think this is will become a problem? Should I remap them in the extension or is it fine to let GS create the RetypingDataSource?

Ok. Fair enough. The counter-argument here is though, that in such cases the user has not had a chance (yet) to create such configuration, so the empty metadata is fine. It’s more of a problem though that in the case of JDBC Views, the data here would be the metadata of the “parent” feature type, if I understood the design correctly.

Passing configuration from GS to an GT extension is not specific to wfs-ng or other “limited access” sources. It should be easy to argue that many data stores require more configuration than is provided in vanilla featuretype.xml.

When I was asking about plugin configuration two weeks back, Jody said: “Other data stores, such as pregeneralization, or app-schema, make use of their own config files which are passed in as a datastore parameter.”. You then came up the idea of the pluggable XML parsers, which I read as a sign that you were not 100% comfortable with the idea that data stores must take care of the configuration files themselves. I then applied your model to store configuration. What I’m now asking is how the configuration data the parsers produce passed to the actual extension.

I’m not entirely sure what you’re not buying here? Maybe we’ve accidentally started talking past each other?

Great. I’ll look forward to this refactor!

https://github.com/sampov2/geoserver/commit/22e01d28f13d536c1fcc1657a6a8fa0716489bb8

https://github.com/sampov2/geotools/commit/258b099a5b9174d1b3eeca69a6dd98754cc4815d

(The GT commit is unfortunately a bit muddled up by a refactor of the configuration beans. Just look at how the StoredQueryConfiguration object is looked up)

S.

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

On Fri, Apr 25, 2014 at 4:16 PM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:

Ah, you’re right, forgot about it, your store must be emitting invalid names (names with : inside), something that indeed
with JDBC store is possible, but very unlikely.

When I debugged this, my dataStore was wrapped as a RetypingDataStore. Check DataStoreUtils.getDataAccess()

We don’t wrap stores at the ResourcePool level, if we do, it’s happening higher level in the security

wrappers. ResourcePool gets the stores from the factories directly, there are no intermediaries that
I’m aware of, just check the getDataStore code paths.

Nope, ResourcePool.getFeatureSource creates the feature type for virtual tables, the feature type does not exist in the store before
we call addVirtualTable.

As this would be a mechanism to provide extension specific metadata to configured features, it would make sense that the FeatureTypeInfo should exist when this mechanism is used. Also, GeoServerFeatureSource.create() is used only in ResourcePool.getFeatureSource() - which is definitely a case where the feature type exists.

I don’t buy this, personally I don’t remember anybody ever asking or experssing interest on the subject. The way I see it, stored queries are pretty unique to the WFS 2.0 specification. Do you know of other data sources in the wild that limit access to the store the same way? Make me just a single example and I’ll be with you about supporting this use case.

Yeah. We don’t want to add something that will only be used for wfs-ng. I would however argue, that the reason nobody else has yet wondered about this problem is because before pluggable XStreamParsersInitializers, the extensions were themselves in charge of configuration files and they didn’t have to to pipe their config from pure generic GS to GT. Now that the pluggable mechanism makes this a possibility, I see a strong case that other extensions might want to opt for something like this.

Anyways, the mechanism I talked about will be created the day I find time
to introduce feature type restructuring via TransformFeatureSource it will be needed (but it will be my problem, along with fixing DataStoreUtils).

Can we see a diff? Looking for the changes inspecting the source code is pretty inefficient.

Cheers

Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

10

On Wed, Apr 30, 2014 at 8:44 AM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Hi Andrea,

Do you have any further thoughts on this or should I come up with another
strategy for passing the configuration objects?

Thanks for the diff. I believe it's ok... I have some generic concerns:
* passing the whole metadata map seems a bit excessive... but rolling a way
to fliter the good keys is probably
  overdoing it... unless anybody else has ideas, I'm good with it
* from the point of view of GeoTools, there is no reason to have the map
point to Serializable, why not use
  Object instead (the hint has to make sense in GeoTools without factorying
GeoServer in the picture)
* for the same reason above, I would use a key name other than
FEATURETYPEINFO_METADATA

Just thinking out loud, another option could be to just merge the metadata
values into the query hints
map directly, not as a submap, every time a key in the metadata map matches
one of the Hints
well known values... that said, this would make it a bit restrictive, which
would bring us back to
some pluggable mechanism to list which hints should be merged... I'm
probably overthinking it.
Anyone else has opinions?

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

11

On Wed, Apr 30, 2014 at 10:57 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Wed, Apr 30, 2014 at 8:44 AM, Sampo Savolainen <
sampo.savolainen@anonymised.com> wrote:

Hi Andrea,

Do you have any further thoughts on this or should I come up with another
strategy for passing the configuration objects?

Thanks for the diff. I believe it's ok... I have some generic concerns:
* passing the whole metadata map seems a bit excessive... but rolling a
way to fliter the good keys is probably
  overdoing it... unless anybody else has ideas, I'm good with it

* from the point of view of GeoTools, there is no reason to have the map

point to Serializable, why not use
  Object instead (the hint has to make sense in GeoTools without
factorying GeoServer in the picture)

I'll convert it to Object. I was just using the type of object that was
available for a proof of concept.

* for the same reason above, I would use a key name other than
FEATURETYPEINFO_METADATA

Just thinking out loud, another option could be to just merge the metadata
values into the query hints
map directly, not as a submap, every time a key in the metadata map
matches one of the Hints
well known values... that said, this would make it a bit restrictive,
which would bring us back to
some pluggable mechanism to list which hints should be merged... I'm
probably overthinking it.
Anyone else has opinions?

Merging the two maps but I can see at least two problems with that:
- we lose the distinction between query hints and feature configuration
- the keys are incompatible (RenderingHints$Key vs String)

How about I create a new RenderingHints$Key implementation, say
"ConfigurationMetadataKey" which wraps the featuretypeinfo metadata String
keys. Then we could just merge the two maps into one?

Sampo

--
Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

12

Hi,

Here’s how it would look like:

GT: https://github.com/sampov2/geotools/commit/5b838eea027014c0149bea9015b5323118d389db
GS: https://github.com/sampov2/geoserver/commit/a3013d12d9f72a3f95f4db2f7bf5f78a0a94b1b9

Sampo

···

On Mon, May 5, 2014 at 9:43 AM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

On Wed, Apr 30, 2014 at 10:57 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I’ll convert it to Object. I was just using the type of object that was available for a proof of concept.

Merging the two maps but I can see at least two problems with that:

  • we lose the distinction between query hints and feature configuration
  • the keys are incompatible (RenderingHints$Key vs String)

How about I create a new RenderingHints$Key implementation, say “ConfigurationMetadataKey” which wraps the featuretypeinfo metadata String keys. Then we could just merge the two maps into one?

Sampo

–

Sampo Savolainen
R&D Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc

This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.

On Wed, Apr 30, 2014 at 8:44 AM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:

Hi Andrea,

Do you have any further thoughts on this or should I come up with another strategy for passing the configuration objects?

Thanks for the diff. I believe it’s ok… I have some generic concerns:

  • passing the whole metadata map seems a bit excessive… but rolling a way to fliter the good keys is probably
    overdoing it… unless anybody else has ideas, I’m good with it

  • from the point of view of GeoTools, there is no reason to have the map point to Serializable, why not use
    Object instead (the hint has to make sense in GeoTools without factorying GeoServer in the picture)

  • for the same reason above, I would use a key name other than FEATURETYPEINFO_METADATA

Just thinking out loud, another option could be to just merge the metadata values into the query hints
map directly, not as a submap, every time a key in the metadata map matches one of the Hints
well known values… that said, this would make it a bit restrictive, which would bring us back to
some pluggable mechanism to list which hints should be merged… I’m probably overthinking it.
Anyone else has opinions?