[Geoserver-devel] Helping JDBCConfig with new derived properties

GeoServer GeoServer Developer
1

Hi,
working on the scalability of the UI along with JDBCConfig with stumbled into
a few issues in the preview tables, which require some changes in JDBCConfig
and GeoServer alike to work efficiently.

The cached layers preview basically does a scan along all layers to find the cached
ones, matching them against the GWC own tiled cached layers (which are a parallel
set of layers, not kept in the DB).
To avoid this, we’d like to propose adding in each layer metadata map a “tileCached”
property, and then expose this new nested property as one that JDBCConfig can
filter against, I guess it would look like “metadata.tileCached” syntax wise.

Managing this property might be tricky, as far as I can see it would have to be done
in the UI itself.
And we’d need to find a way to populate it for all layers that are missing it, on startup,
have some class that looks for all layers not having this property, and would populate it.

Just to be sure, with the integrated GWC there is no way to manipulate the caching properties
of a layer via REST?

The global layer preview requires to treat layers and layer groups and a single entity,
that can be filtered, sorted and paged upon.
At the catalog level we have PublishedInfo, which is not managed by JDBCConfig today,
but I guess nothing prevents adding it, right?

Here again we have to deal with a notion of a layer being “visible”, that is enabled
and advertised, and a layer group being “deeply enabled”, that is, make sure that
every nested layer in the group is enabled (eventually deeply nested, layer groups
can contain other layer groups).
In this case I’d suggest to add in the metadata map of the layer groups a derived
property called visible, that specific CatalogListener will maintain over time,
and have this be one of the properties that JDBCConfig can filter against.

The above works assuming JDBCConfig manages inheritance hierarchies of sorts,
which I thought it did since I see is has property types for both resourceinfo,
featuretypeinfo and coverageinfo, but looking at imported data, it seems that
nothing was filled in the resourceinfo property set??

Could anybody point at the code that manages this, if it does at all?

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Hi,
figured out how inheritance works (it uses the ClassMappings.class), still don’t see a way out of derived
properties in metadata for the other issues.
I’ll move on with this approach unless I hear otherwise

Cheers
Andrea

···

On Thu, Nov 13, 2014 at 7:58 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
working on the scalability of the UI along with JDBCConfig with stumbled into
a few issues in the preview tables, which require some changes in JDBCConfig
and GeoServer alike to work efficiently.

The cached layers preview basically does a scan along all layers to find the cached
ones, matching them against the GWC own tiled cached layers (which are a parallel
set of layers, not kept in the DB).
To avoid this, we’d like to propose adding in each layer metadata map a “tileCached”
property, and then expose this new nested property as one that JDBCConfig can
filter against, I guess it would look like “metadata.tileCached” syntax wise.

Managing this property might be tricky, as far as I can see it would have to be done
in the UI itself.
And we’d need to find a way to populate it for all layers that are missing it, on startup,
have some class that looks for all layers not having this property, and would populate it.

Just to be sure, with the integrated GWC there is no way to manipulate the caching properties
of a layer via REST?

The global layer preview requires to treat layers and layer groups and a single entity,
that can be filtered, sorted and paged upon.
At the catalog level we have PublishedInfo, which is not managed by JDBCConfig today,
but I guess nothing prevents adding it, right?

Here again we have to deal with a notion of a layer being “visible”, that is enabled
and advertised, and a layer group being “deeply enabled”, that is, make sure that
every nested layer in the group is enabled (eventually deeply nested, layer groups
can contain other layer groups).
In this case I’d suggest to add in the metadata map of the layer groups a derived
property called visible, that specific CatalogListener will maintain over time,
and have this be one of the properties that JDBCConfig can filter against.

The above works assuming JDBCConfig manages inheritance hierarchies of sorts,
which I thought it did since I see is has property types for both resourceinfo,
featuretypeinfo and coverageinfo, but looking at imported data, it seems that
nothing was filled in the resourceinfo property set??

Could anybody point at the code that manages this, if it does at all?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

On Fri, Nov 14, 2014 at 10:45 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Hi,
figured out how inheritance works (it uses the ClassMappings.class), still
don't see a way out of derived
properties in metadata for the other issues.
I'll move on with this approach unless I hear otherwise

IIRC you should be able to add the PublishedInfo superinterface to
ClassMappings and override concreteInterfaces() to return both LayerInfo
and LayerGroupInfo. I _think_ that should be it, as queries against
PublishedInfo should then query both LayerInfos and LayerGroupInfos, but
will never query for PublishedInfo as it would be an "abstract" class.

Cheers
Andrea

On Thu, Nov 13, 2014 at 7:58 PM, Andrea Aime <andrea.aime@anonymised.com
> wrote:

Hi,
working on the scalability of the UI along with JDBCConfig with stumbled
into
a few issues in the preview tables, which require some changes in
JDBCConfig
and GeoServer alike to work efficiently.

The cached layers preview basically does a scan along all layers to find
the cached
ones, matching them against the GWC own tiled cached layers (which are a
parallel
set of layers, not kept in the DB).

hmmm that actually sounds odd. I can't think of where that'd be happening,
querying all LayerInfo's that's it. (you mean CachedLayersPage?)

What's probably happening is that "create tile layer automatically" is set
on, and hence there are thousands of tile layers, so when
CachedLayersPage's provider accesses the tile layers they do get queried
once for each tile layer.
If that's the case, then a better solution would be to add paging to
CachedLayersPage itself.

It also strikes me as bad practice not to have tile layers automatically
created for such large catalogs. Although that'd be a workaround and
someone could legitimately want that enabled.

To avoid this, we'd like to propose adding in each layer metadata map a

"tileCached"
property, and then expose this new nested property as one that JDBCConfig
can
filter against, I guess it would look like "metadata.tileCached" syntax
wise.

I'm not against it, but I'm not sure that's the way to go either.
Can you point out where the layer/groupinfo full scan is being triggered in
the gwc module? maybe there's a better way to solve the issue.

Managing this property might be tricky, as far as I can see it would have
to be done
in the UI itself.
And we'd need to find a way to populate it for all layers that are
missing it, on startup,
have some class that looks for all layers not having this property, and
would populate it.

Just to be sure, with the integrated GWC there is no way to manipulate
the caching properties
of a layer via REST?

Yes there is <
http://docs.geoserver.org/stable/en/user/geowebcache/rest/layers.html#adding-a-geoserver-layer

The root element is GeoServerLayer instead of wmsLayer.

hope that helps.

Cheers,
Gabriel

The global layer preview requires to treat layers and layer groups and a
single entity,
that can be filtered, sorted and paged upon.
At the catalog level we have PublishedInfo, which is not managed by
JDBCConfig today,
but I guess nothing prevents adding it, right?

Here again we have to deal with a notion of a layer being "visible", that
is enabled
and advertised, and a layer group being "deeply enabled", that is, make
sure that
every nested layer in the group is enabled (eventually deeply nested,
layer groups
can contain other layer groups).
In this case I'd suggest to add in the metadata map of the layer groups a
derived
property called visible, that specific CatalogListener will maintain over
time,
and have this be one of the properties that JDBCConfig can filter against.

The above works assuming JDBCConfig manages inheritance hierarchies of
sorts,
which I thought it did since I see is has property types for both
resourceinfo,
featuretypeinfo and coverageinfo, but looking at imported data, it seems
that
nothing was filled in the resourceinfo property set??

Could anybody point at the code that manages this, if it does at all?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be confidential
or proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.

http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--

Gabriel Roldán
Software Developer | Boundless <http://boundlessgeo.com/&gt;
groldan@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;

4

On Fri, Nov 14, 2014 at 10:55 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

On Fri, Nov 14, 2014 at 10:45 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

Hi,
figured out how inheritance works (it uses the ClassMappings.class),
still don't see a way out of derived
properties in metadata for the other issues.
I'll move on with this approach unless I hear otherwise

IIRC you should be able to add the PublishedInfo superinterface to
ClassMappings and override concreteInterfaces() to return both LayerInfo
and LayerGroupInfo. I _think_ that should be it, as queries against
PublishedInfo should then query both LayerInfos and LayerGroupInfos, but
will never query for PublishedInfo as it would be an "abstract" class.

Yep, that's what I ended up doing :slight_smile:

The cached layers preview basically does a scan along all layers to find

the cached
ones, matching them against the GWC own tiled cached layers (which are a
parallel
set of layers, not kept in the DB).

hmmm that actually sounds odd. I can't think of where that'd be happening,
querying all LayerInfo's that's it. (you mean CachedLayersPage?)

What's probably happening is that "create tile layer automatically" is set
on, and hence there are thousands of tile layers, so when
CachedLayersPage's provider accesses the tile layers they do get queried
once for each tile layer.
If that's the case, then a better solution would be to add paging to
CachedLayersPage itself.

It also strikes me as bad practice not to have tile layers automatically
created for such large catalogs. Although that'd be a workaround and
someone could legitimately want that enabled.

I'll dig deeper into this one and get back to you.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------