[Geoserver-devel] Fun with nested/derived/delegated properties and JDBCConfig

GeoServer GeoServer Developer
1

Hi,
another issue that came up with JDBCConfig is that we found it breaks when dealing with derived properties.

There are two types of derived properties:

  • computed ones, that lack a getter, such as “prefixedName()”
  • delegated ones, that lack a field, and point directly to the value in another object, for example LayerInfo enabled/advertised

For the derived ones, I wanted to use PublishedInfo/LayerInfo/LayerGroupInfo/.prefixedName… but JDBCConfig does not
see it, because it does not have a getter, whilst it sees the ResoruceInfo one because there is a deprecated getPrefixedName()
method.
For the time being, to move forward, I’ve added getPrefixedName() to PublishedInfo too (and subclasses), but I guess it would
be best if we had a way to declare these derived properties that only have a getter, to allow filtering/sorting on them.
Maybe we could have a derived_properties.properties, in the same spirit as nested_properties.properties?

The delegated ones are nasty… JDBCConfig saves LayerInfo enabled/advertised in the db at the time of import,
and then the GUI only touches ResourceInfo advertised/enabled, so the code saving them only saves the change
in ResourceInfo, but not in LayerInfo.
For these it would be nice to just have nested_properties.propertise handle the case such as:
LayerInfo.enabled=ResourceInfo.enabled
LayerInfo.advertised=ResourceInfo.advertised

since in the end we are accessing a nested property.

One final nasty case is title, which is exposed as a property in PublishedInfo, and it is direct in LayerGroupInfo,
but when it comes to layers, there is a direct property that is never used, and we actually live off the title of the nested
resource.
I guess a delegation declaration such as:
LayerInfo.title=ResourceInfo.title

would do the trick, and we can kill LayerInfo own title? It does not seem anything is using it anyways

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

On Fri, Nov 14, 2014 at 11:31 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Hi,
another issue that came up with JDBCConfig is that we found it breaks when
dealing with derived properties.

There are two types of derived properties:
* computed ones, that lack a getter, such as "prefixedName()"
* delegated ones, that lack a field, and point directly to the value in
another object, for example LayerInfo enabled/advertised

DbMappings.properties() already have a couple hacks for some derived
properties. I guess the easiest path forward would be to build on top of
that and add a couple more conditionals?

For the derived ones, I wanted to use
PublishedInfo/LayerInfo/LayerGroupInfo/.prefixedName... but JDBCConfig does
not
see it, because it does not have a getter, whilst it sees the ResoruceInfo
one because there is a deprecated getPrefixedName()
method.
For the time being, to move forward, I've added getPrefixedName() to
PublishedInfo too (and subclasses), but I guess it would
be best if we had a way to declare these derived properties that only have
a getter, to allow filtering/sorting on them.
Maybe we could have a derived_properties.properties, in the same spirit as
nested_properties.properties?

The delegated ones are nasty... JDBCConfig saves LayerInfo
enabled/advertised in the db at the time of import,
and then the GUI only touches ResourceInfo advertised/enabled, so the code
saving them only saves the change
in ResourceInfo, but not in LayerInfo.
For these it would be nice to just have nested_properties.propertise
handle the case such as:
LayerInfo.enabled=ResourceInfo.enabled
LayerInfo.advertised=ResourceInfo.advertised

why not just build a filter like enabled=true AND resource.enabled = true
AND resource.store.enabled = true
Otherwise how will you differentiate from enabled and enabled (that is,
isEnabled() and enabled())?

since in the end we are accessing a nested property.

One final nasty case is title, which is exposed as a property in
PublishedInfo, and it is direct in LayerGroupInfo,
but when it comes to layers, there is a direct property that is never
used, and we actually live off the title of the nested
resource.
I guess a delegation declaration such as:
LayerInfo.title=ResourceInfo.title
would do the trick, and we can kill LayerInfo own title? It does not seem
anything is using it anyways

that sounds about right to me. Although multiple LayerInfo's may exist for
the same ResourceInfo (Catalog's List<LayerInfo> getLayers( ResourceInfo
resource ) method?).

So why not let LayerInfo.title live instead? It looks to me like it'd be
better that way, and client code would be free to make a null check on
LayerInfo.getTitle() and fallback to its resource title if deemed
appropriate?

my 2c/
Gabriel

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.

http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--

Gabriel Roldán
Software Developer | Boundless <http://boundlessgeo.com/&gt;
groldan@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;

3

On Fri, Nov 14, 2014 at 11:15 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

On Fri, Nov 14, 2014 at 11:31 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

Hi,
another issue that came up with JDBCConfig is that we found it breaks
when dealing with derived properties.

There are two types of derived properties:
* computed ones, that lack a getter, such as "prefixedName()"
* delegated ones, that lack a field, and point directly to the value in
another object, for example LayerInfo enabled/advertised

DbMappings.properties() already have a couple hacks for some derived
properties. I guess the easiest path forward would be to build on top of
that and add a couple more conditionals?

This works for "title", allows me to pick the resource one, but does not
work for "prefixedName()" because it's not a property,
the code only recognizes "getPrefixedName()" as a property.
Our current convention in the code is that getters are there only for true
properties, whist we avoid using the get/is prefix
for properties that are derived from others, but have no real storage
(e.g., like size() in a List).
The UI shows often the prefixed name of a layer (and the flat name of a
layer group, remember I'm filtering
against a PublishedInfo), but JDBCConfig does not have a way to deal with
it.

That's why in my previous mail I've said:

For the derived ones, I wanted to use
PublishedInfo/LayerInfo/LayerGroupInfo/.prefixedName... but JDBCConfig does
not
see it, because it does not have a getter, whilst it sees the
ResoruceInfo one because there is a deprecated getPrefixedName()
method.
For the time being, to move forward, I've added getPrefixedName() to
PublishedInfo too (and subclasses), but I guess it would
be best if we had a way to declare these derived properties that only
have a getter, to allow filtering/sorting on them.
Maybe we could have a derived_properties.properties, in the same spirit
as nested_properties.properties?

Shall we go down this road, or shall we stop using the "derived property"
convention for the sake of keeping things
JDBCConfig compatible?

The delegated ones are nasty... JDBCConfig saves LayerInfo
enabled/advertised in the db at the time of import,
and then the GUI only touches ResourceInfo advertised/enabled, so the
code saving them only saves the change
in ResourceInfo, but not in LayerInfo.
For these it would be nice to just have nested_properties.propertise
handle the case such as:
LayerInfo.enabled=ResourceInfo.enabled
LayerInfo.advertised=ResourceInfo.advertised

why not just build a filter like enabled=true AND resource.enabled = true
AND resource.store.enabled = true
Otherwise how will you differentiate from enabled and enabled (that is,
isEnabled() and enabled())?

Right, I can also do that. By the way, to make this work against
PublishedInfo, and thus in particular against LayerGroupInfo,
that does not have a resource, I also had to make a change in the way the
queries are encoded, you had the right comment,
but the wrong implementation.
In FilterToCatalogSQL you had:

        builder.addType(PropertyIsNull.class);// whether a property exists
at all
        builder.addType(PropertyIsNil.class);// whether the property exists
AND it's value is null

But then the implementation of PropertyIsNull was not actually checking if
the property was not even there (which is
the case of resource and layer group), I've changed it in my fork to read:

@Override
    public Object visit(PropertyIsNull filter, Object extraData) {
        final PropertyName propertyName = (PropertyName)
filter.getExpression();
        final String propertyTypesParam = propertyTypesParam(propertyName);

        StringBuilder builder = append(extraData,
                "oid IN (select oid from object_property where
property_type in (:",
                propertyTypesParam,
                ") and value IS NULL) OR oid IN (select oid from object
where oid not in (select oid from object_property where property_type in (:"
                        + propertyTypesParam + "))) /* ",
filter.toString(), " */ \n");
        return builder;
    }

So that it also selects missing properties.
Oh, PropertyIsNil is declared to be supported, but then it's translation to
sql is not coded:

/**
     * @param filter
     * @param extraData
     * @return
     * @see
org.opengis.filter.FilterVisitor#visit(org.opengis.filter.PropertyIsNil,
     * java.lang.Object)
     */
    @Override
    public Object visit(PropertyIsNil filter, Object extraData) {

        return extraData;
    }

I guess not a problem, nothing is using PropertyIsNil right now but...
either the caps or the encoding should be fixed

One thing that is still left out are the checks for LayerGroup being
enabled, or not. In the current code that is done
programmatically (not that he layers() method unfolds all nested layers,
either directly in the group, or contained
by another nested group):

https://github.com/geoserver/geoserver/blob/master/src/web/demo/src/main/java/org/geoserver/web/demo/PreviewLayerProvider.java#L70

I guess one could add a "enabled()" method to LayerGroup to centralize that
check, but still, JDBCConfig would not
be able to use it because it does not start with a "get"
Moreover, that derived property depends on all layers nested in the group,
so for example if layer group
that contains another layer group that contains a layer that just got
disabled, the enabled status of the group
should be modified... but JDBCConfig is not going to be notified of it.

That's why I was tempted to put a flag in the group metadata section, and
have that flag be maintained by
a catalog callback that ends up updating it any time a layer contained in a
group enabled flag is modified.
Any other idea?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

4

On Mon, Nov 17, 2014 at 8:09 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Fri, Nov 14, 2014 at 11:15 PM, Gabriel Roldan <groldan@anonymised.com
> wrote:

On Fri, Nov 14, 2014 at 11:31 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

Hi,
another issue that came up with JDBCConfig is that we found it breaks
when dealing with derived properties.

There are two types of derived properties:
* computed ones, that lack a getter, such as "prefixedName()"
* delegated ones, that lack a field, and point directly to the value in
another object, for example LayerInfo enabled/advertised

DbMappings.properties() already have a couple hacks for some derived
properties. I guess the easiest path forward would be to build on top of
that and add a couple more conditionals?

This works for "title", allows me to pick the resource one, but does not
work for "prefixedName()" because it's not a property,
the code only recognizes "getPrefixedName()" as a property.
Our current convention in the code is that getters are there only for true
properties, whist we avoid using the get/is prefix
for properties that are derived from others, but have no real storage
(e.g., like size() in a List).
The UI shows often the prefixed name of a layer (and the flat name of a
layer group, remember I'm filtering
against a PublishedInfo), but JDBCConfig does not have a way to deal with
it.

That's why in my previous mail I've said:

For the derived ones, I wanted to use
PublishedInfo/LayerInfo/LayerGroupInfo/.prefixedName... but JDBCConfig does
not
see it, because it does not have a getter, whilst it sees the
ResoruceInfo one because there is a deprecated getPrefixedName()
method.
For the time being, to move forward, I've added getPrefixedName() to
PublishedInfo too (and subclasses), but I guess it would
be best if we had a way to declare these derived properties that only
have a getter, to allow filtering/sorting on them.
Maybe we could have a derived_properties.properties, in the same spirit
as nested_properties.properties?

Shall we go down this road, or shall we stop using the "derived property"
convention for the sake of keeping things
JDBCConfig compatible?

Definitely go with derived_properties.properties. It doesn't make sense to
break a sensible API convention to accommodate to JDBC's convenience.

The delegated ones are nasty... JDBCConfig saves LayerInfo
enabled/advertised in the db at the time of import,
and then the GUI only touches ResourceInfo advertised/enabled, so the
code saving them only saves the change
in ResourceInfo, but not in LayerInfo.
For these it would be nice to just have nested_properties.propertise
handle the case such as:
LayerInfo.enabled=ResourceInfo.enabled
LayerInfo.advertised=ResourceInfo.advertised

why not just build a filter like enabled=true AND resource.enabled = true
AND resource.store.enabled = true
Otherwise how will you differentiate from enabled and enabled (that is,
isEnabled() and enabled())?

Right, I can also do that. By the way, to make this work against
PublishedInfo, and thus in particular against LayerGroupInfo,
that does not have a resource, I also had to make a change in the way the
queries are encoded, you had the right comment,
but the wrong implementation.
In FilterToCatalogSQL you had:

        builder.addType(PropertyIsNull.class);// whether a property exists
at all
        builder.addType(PropertyIsNil.class);// whether the property
exists AND it's value is null

But then the implementation of PropertyIsNull was not actually checking if
the property was not even there (which is
the case of resource and layer group), I've changed it in my fork to read:

oh, my bad.

@Override
    public Object visit(PropertyIsNull filter, Object extraData) {
        final PropertyName propertyName = (PropertyName)
filter.getExpression();
        final String propertyTypesParam = propertyTypesParam(propertyName);

        StringBuilder builder = append(extraData,
                "oid IN (select oid from object_property where
property_type in (:",
                propertyTypesParam,
                ") and value IS NULL) OR oid IN (select oid from object
where oid not in (select oid from object_property where property_type in (:"
                        + propertyTypesParam + "))) /* ",
filter.toString(), " */ \n");
        return builder;
    }

So that it also selects missing properties.
Oh, PropertyIsNil is declared to be supported, but then it's translation
to sql is not coded:

/**
     * @param filter
     * @param extraData
     * @return
     * @see
org.opengis.filter.FilterVisitor#visit(org.opengis.filter.PropertyIsNil,
     * java.lang.Object)
     */
    @Override
    public Object visit(PropertyIsNil filter, Object extraData) {

        return extraData;
    }

I guess not a problem, nothing is using PropertyIsNil right now but...

either the caps or the encoding should be fixed

cool, thanks for catching it up.

One thing that is still left out are the checks for LayerGroup being
enabled, or not. In the current code that is done
programmatically (not that he layers() method unfolds all nested layers,
either directly in the group, or contained
by another nested group):

https://github.com/geoserver/geoserver/blob/master/src/web/demo/src/main/java/org/geoserver/web/demo/PreviewLayerProvider.java#L70

I guess one could add a "enabled()" method to LayerGroup to centralize
that check, but still, JDBCConfig would not
be able to use it because it does not start with a "get"
Moreover, that derived property depends on all layers nested in the group,
so for example if layer group
that contains another layer group that contains a layer that just got
disabled, the enabled status of the group
should be modified... but JDBCConfig is not going to be notified of it.

yeah, tough one. I guess that kind of nesting wasn't even possible when
jdbcconfig was created.

That's why I was tempted to put a flag in the group metadata section, and
have that flag be maintained by
a catalog callback that ends up updating it any time a layer contained in
a group enabled flag is modified.

well that looks like a much saner solution than crafting custom SQL for
that property to join with all its children.

Any other idea?

Nope, sounds good so far. Have fun.

Gabriel.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--

Gabriel Roldán
Software Developer | Boundless <http://boundlessgeo.com/&gt;
groldan@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;