[Geoserver-devel] Propose a new extension point in ResourcePool for retyping feature types and features

Dear all,
long story short, I would like to propose adding a new extension point in ResourcePool that would allow extensions to retype a feature type, and consequently change the associated features.
In practice it would allow us to create extensions that would be able to add\hide attributes, generate new attributes or even just change the values of existing attributes.

For those wondering if the current FeatureTypeCallbacl [2] doesn’t allow us already to do that, the answer is no :slight_smile:
FeatureTypeCallBack allow extensions to initiate, and manage, feature types that are generated based on user provided information, for example, SQL views or Solr feature types, but not to change existing types or features.

The extensions code would be executed here [3] and here [4], when a feature type is build (buildFeatureType) and when the source for a feature type is returned (getFeatureSource).
The later would allow us to wrap a feature source, setting the feature type we want and\or apply the wrappers we want to the future stream of features.

I’m proposing the following interface for the extension point:

/**

  • Extension point for {@link ResourcePool} allowing us to retype an existing {@link FeatureType},
  • and to rearrange the features produced by the correspondent {@link FeatureSource} by wrapping it.
  • This extension point can be, for example, used to add a new attribute or remove an existing one.
  • If the intend is instead to create a new feature type, the existing {@link FeatureTypeCallback}
  • extension point should be used.
    */
    public interface RetypeFeatureTypeCallback {

/**

  • Gives a chance to this callback to retype the provided feature type, if this callback has no
  • interest in retyping the provided feature type, then the unchanged provided feature type
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureType non NULL GeoTools data source feature type
  • @return retyped feature type or the unchanged provided feature type
    */
    FeatureType retypeFeatureType(FeatureTypeInfo featureTypeInfo, FeatureType featureType);

/**

  • Gives a chance to this callback to wrap the provided feature source, if this callback has no
  • interest in wrapping the provided feature source, then the unchanged provided feature source
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureSource non NULL GeoTools feature source
  • @return wrapped feature source or the unchanged provided feature source
    */
    <T extends FeatureType, U extends Feature> FeatureSource<T, U> wrapFeatureSource(
    FeatureTypeInfo featureTypeInfo, FeatureSource<T, U> featureSource);
    }

If this extension point is accepted, we would also like to backport it to 2.16.x, opinions are welcome :slight_smile:

Kind regards,
Nuno Oliveira

[1] https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java
[2] https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/FeatureTypeCallback.java
[3] https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L1114
[4] https://github.com/geoserver/geoserver/blob/master/src/main/src/main/java/org/geoserver/catalog/ResourcePool.java#L1393

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

Nuno:

  1. This is very similar to GeoServer 1.0 functionality that allowed attribute to be named, described, or hidden, …
  2. Do you need more information to handle the query efficiently? I had for some time wished to see a “java view” created with a UI similar to “sql view” making use of the transform process. Sadly this is a little complex due to the need to reverse any transformation, mapping a query back to the original data source. Does your " RetypeFeatureTypeCallback" also need to process queries?
···


Jody Garnett

Hi Jody,
thank you for the feedback, please seem my answers bellow :-)

Nuno:

  1. This is very similar to GeoServer 1.0 functionality that allowed attribute to be named, described, or hidden, …

When GeoServer 1.0 was releases (~2003) I was still in high-school :D, so I didn’t use it and was not aware of that.

  1. Do you need more information to handle the query efficiently? I had for some time wished to see a “java view” created with a UI similar to “sql view” making use of the transform process. Sadly this is a little complex due to the need to reverse any transformation, mapping a query back to the original data source. Does your " RetypeFeatureTypeCallback" also need to process queries?

Yes, with this extension point you will be able to define a transformation, in the FeatureSource wrapper, that can handle queries and map them has needed to the real attributes.

One of the use case I have for this is the following:

Often we deal with databases were for a reason or another simple latitudes\longitudes are used and no spatial indexing is available, one option here would be to create an SQL view but there is several cons, e.g. no good indexing, negative impacts on heavily written tables, …
I will later propose a community module that will make use of this extension point,and will give us the option when configuring a layer to generate a new geometry attribute on the fly from the latitudes and longitudes:

  • Clients will see a valid geometry attribute;
  • GeoServer will translate the queries, transforming the geometry to latitudes and longitudes ranged queries;
  • This will give us the best of two worlds, GIS clients will see geometries, the database will see latitudes \ longitudes (with the proper indexes)

Best regards,
Nuno Oliveira

···


Jody Garnett

Dear all,
long story short, I would like to propose adding a new extension point in ResourcePool that would allow extensions to retype a feature type, and consequently change the associated features.
In practice it would allow us to create extensions that would be able to add\hide attributes, generate new attributes or even just change the values of existing attributes.

Works for me.

For those wondering if the current FeatureTypeCallbacl [2] doesn’t allow us already to do that, the answer is no :slight_smile:
FeatureTypeCallBack allow extensions to initiate, and manage, feature types that are generated based on user provided information, for example, SQL views or Solr feature types, but not to change existing types or features.

Indeed it does something different now… however… what about adding new methods to it, with default implementations?
On one side, two focused interfaces are normally better, on the other side FeatureTypeCallback name is generic, could host more
behavior related to feature types… just considering options out loud, not pushing in a particular direction.

The extensions code would be executed here [3] and here [4], when a feature type is build (buildFeatureType) and when the source for a feature type is returned (getFeatureSource).
The later would allow us to wrap a feature source, setting the feature type we want and\or apply the wrappers we want to the future stream of features.

I’m proposing the following interface for the extension point:

/**

  • Extension point for {@link ResourcePool} allowing us to retype an existing {@link FeatureType},
  • and to rearrange the features produced by the correspondent {@link FeatureSource} by wrapping it.
  • This extension point can be, for example, used to add a new attribute or remove an existing one.
  • If the intend is instead to create a new feature type, the existing {@link FeatureTypeCallback}
  • extension point should be used.
    */
    public interface RetypeFeatureTypeCallback {

/**

  • Gives a chance to this callback to retype the provided feature type, if this callback has no
  • interest in retyping the provided feature type, then the unchanged provided feature type
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureType non NULL GeoTools data source feature type
  • @return retyped feature type or the unchanged provided feature type
    */
    FeatureType retypeFeatureType(FeatureTypeInfo featureTypeInfo, FeatureType featureType);

/**

  • Gives a chance to this callback to wrap the provided feature source, if this callback has no
  • interest in wrapping the provided feature source, then the unchanged provided feature source
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureSource non NULL GeoTools feature source
  • @return wrapped feature source or the unchanged provided feature source
    */
    <T extends FeatureType, U extends Feature> FeatureSource<T, U> wrapFeatureSource(
    FeatureTypeInfo featureTypeInfo, FeatureSource<T, U> featureSource);
    }

If this extension point is accepted, we would also like to backport it to 2.16.x, opinions are welcome :slight_smile:

Works for me.

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

One way to implement this would be to use the TransformFeatureSource, already used in ResourcePool, along with
a definition function that builds a point, something like “ToPointFunction” present in app-schema already.
Along maybe with a generic way to go back from the function result to its parameters, I’m thinking out loud here:

public interface InvertibleFunction {
List getParameters(Object result);
}

which would allow for a generic support of function transformations in TransformFeatureSource.

That said, just an idea.

Cheers
Andrea

(attachments)

face-smile.png

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Do you need to add a method then? To “handle queries and map them as needed to the real attributes” ?

/** Transform query to operate on original (pretransformed) data source. */
public Query dataQuery( Query query);

(attachments)

face-smile.png

···


Jody Garnett

That would be the internal job of the wrapping feature source returned by:

T extends FeatureType, U extends Feature> FeatureSource<T, U> wrapFeatureSource(
FeatureTypeInfo featureTypeInfo, FeatureSource<T, U> featureSource);

It exposes a different feature type, and back maps the queries, filters and the like.
Each implementation of it can do it the way it sees fit.

Cheers
Andrea

(attachments)

face-smile.png

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Thanks for the clarification.

(attachments)

face-smile.png

···


Jody Garnett

So with the clarifications (thanks Andrea) I understand, would you like to write up a GSIP page?

···


Jody Garnett

Hi Andrea,
thx for the feedback, please see my answers bellow:

Dear all,
long story short, I would like to propose adding a new extension point in ResourcePool that would allow extensions to retype a feature type, and consequently change the associated features.
In practice it would allow us to create extensions that would be able to add\hide attributes, generate new attributes or even just change the values of existing attributes.

Works for me.

For those wondering if the current FeatureTypeCallbacl [2] doesn’t allow us already to do that, the answer is no :slight_smile:
FeatureTypeCallBack allow extensions to initiate, and manage, feature types that are generated based on user provided information, for example, SQL views or Solr feature types, but not to change existing types or features.

Indeed it does something different now… however… what about adding new methods to it, with default implementations?
On one side, two focused interfaces are normally better, on the other side FeatureTypeCallback name is generic, could host more
behavior related to feature types… just considering options out loud, not pushing in a particular direction.

I had the same thoughts, what make me go in the direction of having a new interface is that FeatureTypeCallback is quite clear now, the current methods adapt well to the feature type life-cycle:

  • canHandle
  • initialize
  • flush
  • dispose

… adding the new retype methods makes it look weird to me, but I’m like you no strong preference for one or another :frowning:

The extensions code would be executed here [3] and here [4], when a feature type is build (buildFeatureType) and when the source for a feature type is returned (getFeatureSource).
The later would allow us to wrap a feature source, setting the feature type we want and\or apply the wrappers we want to the future stream of features.

I’m proposing the following interface for the extension point:

/**

  • Extension point for {@link ResourcePool} allowing us to retype an existing {@link FeatureType},
  • and to rearrange the features produced by the correspondent {@link FeatureSource} by wrapping it.
  • This extension point can be, for example, used to add a new attribute or remove an existing one.
  • If the intend is instead to create a new feature type, the existing {@link FeatureTypeCallback}
  • extension point should be used.
    */
    public interface RetypeFeatureTypeCallback {

/**

  • Gives a chance to this callback to retype the provided feature type, if this callback has no
  • interest in retyping the provided feature type, then the unchanged provided feature type
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureType non NULL GeoTools data source feature type
  • @return retyped feature type or the unchanged provided feature type
    */
    FeatureType retypeFeatureType(FeatureTypeInfo featureTypeInfo, FeatureType featureType);

/**

  • Gives a chance to this callback to wrap the provided feature source, if this callback has no
  • interest in wrapping the provided feature source, then the unchanged provided feature source
  • should be returned. NULL should never be returned.
  • @param featureTypeInfo non NULL GeoServer feature type info
  • @param featureSource non NULL GeoTools feature source
  • @return wrapped feature source or the unchanged provided feature source
    */
    <T extends FeatureType, U extends Feature> FeatureSource<T, U> wrapFeatureSource(
    FeatureTypeInfo featureTypeInfo, FeatureSource<T, U> featureSource);
    }

If this extension point is accepted, we would also like to backport it to 2.16.x, opinions are welcome :slight_smile:

Works for me.

Cheers
Andrea

==


<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

GeoServer Professional Services from the experts! Visit [http://goo.gl/it488V](http://goo.gl/it488V) for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 [http://www.geo-solutions.it](http://www.geo-solutions.it) [http://twitter.com/geosolutions_it](http://twitter.com/geosolutions_it) ------------------------------------------------------- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.*

</details>

Nice, thx fr the tip Andrea!

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Jody,
just for the sake of clarity I will write one and then ask for votes :-).

Thank you both @Jody and @Andrea for the feedback!

···


Jody Garnett