[Geoserver-devel] Upgrading JDBCConfig

GeoServer GeoServer Developer
1

Hi,
as a related concern to the work we’re doing these days, I’ve noticed that for
JDBCConfig to notice the new delegated/nested properties we are adding,
we basically have to re-import the configuration from the data directory.

Which is not a problem for us, but what will happen to those that already
have a real large catalog inside JDBCConfig?

I guess there should be some way to updated the various meta tables, empty
object_property, and then have it re-populated from the object table somehow.

As said, no big deal for us, and maybe no big deal in general since the
module is unsupported anyways but… I guess someone out there might be
actually using it, and having to loose the entire configuration because of
structure updates might be rather nasty.

Ideas?

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Yeah, I have thought about this before. What i was thinking was that we would have a metadata table stored by jdbc config to store various types of stuff. One of the things could be a version of the data directory. However, I am not sure if we have anything that indicates that. I was thinking we could add a simple number and increment it everytime the configuration changes. Anyways, might be some issues with that but that was my initial thinking on the issue.

···

On Tue, Nov 18, 2014 at 3:33 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
as a related concern to the work we’re doing these days, I’ve noticed that for
JDBCConfig to notice the new delegated/nested properties we are adding,
we basically have to re-import the configuration from the data directory.

Which is not a problem for us, but what will happen to those that already
have a real large catalog inside JDBCConfig?

I guess there should be some way to updated the various meta tables, empty
object_property, and then have it re-populated from the object table somehow.

As said, no big deal for us, and maybe no big deal in general since the
module is unsupported anyways but… I guess someone out there might be
actually using it, and having to loose the entire configuration because of
structure updates might be rather nasty.

Ideas?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
VP Engineering | Boundless
jdeolive@anonymised.com
@boundlessgeo

3

I was thinking along the same line. And also for the GWC config that Andrea mentioned was making a full scan at start up.

Basically having a “version” that’s checked at startup, and calls an upgrade function only if the current version is lower than it should be.

···

On Tue, Nov 18, 2014 at 1:18 PM, Justin Deoliveira <jdeolive@anonymised.com9…> wrote:

Yeah, I have thought about this before. What i was thinking was that we would have a metadata table stored by jdbc config to store various types of stuff. One of the things could be a version of the data directory. However, I am not sure if we have anything that indicates that. I was thinking we could add a simple number and increment it everytime the configuration changes. Anyways, might be some issues with that but that was my initial thinking on the issue.

Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

On Tue, Nov 18, 2014 at 3:33 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
as a related concern to the work we’re doing these days, I’ve noticed that for
JDBCConfig to notice the new delegated/nested properties we are adding,
we basically have to re-import the configuration from the data directory.

Which is not a problem for us, but what will happen to those that already
have a real large catalog inside JDBCConfig?

I guess there should be some way to updated the various meta tables, empty
object_property, and then have it re-populated from the object table somehow.

As said, no big deal for us, and maybe no big deal in general since the
module is unsupported anyways but… I guess someone out there might be
actually using it, and having to loose the entire configuration because of
structure updates might be rather nasty.

Ideas?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
VP Engineering | Boundless
jdeolive@anonymised.com9…
@boundlessgeo

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

4

On Tue, Nov 18, 2014 at 5:18 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yeah, I have thought about this before. What i was thinking was that we
would have a metadata table stored by jdbc config to store various types of
stuff. One of the things could be a version of the data directory. However,
I am not sure if we have anything that indicates that. I was thinking we
could add a simple number and increment it everytime the configuration
changes. Anyways, might be some issues with that but that was my initial
thinking on the issue.

Right, that is one concern, changes in the data model should be reflected
in the persisted vision of it (objects and direct properties) that
JDBCConfig has of the data model.

And that's one thing... the other thing is that JDBCConfig itself could
have some new nested property, or a new handling for a
derived property, and so on.

So I guess if we go by the version numbers, we'd need two, one for the
GeoServer object model, and one for the JDBCConfig
interpretation of it

I'm just worried about the GeoServer object model part, will people
remember to update the number when new direct
fields, or new objects are added into the model?
Maybe JDBCConfig could do a hash of the objects and direct fields, and
compare with one stored in the database.
For nested fields we could also do the same... the issue is with the
derived properties, which are handled programmatically...
so yeah, there we'd probably need a a version number of sorts, hoping that
whoever works on JDBCconfig remembers
to update it?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

On Tue, Nov 18, 2014 at 9:53 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Tue, Nov 18, 2014 at 5:18 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yeah, I have thought about this before. What i was thinking was that we
would have a metadata table stored by jdbc config to store various types of
stuff. One of the things could be a version of the data directory. However,
I am not sure if we have anything that indicates that. I was thinking we
could add a simple number and increment it everytime the configuration
changes. Anyways, might be some issues with that but that was my initial
thinking on the issue.

Right, that is one concern, changes in the data model should be reflected
in the persisted vision of it (objects and direct properties) that
JDBCConfig has of the data model.

And that's one thing... the other thing is that JDBCConfig itself could
have some new nested property, or a new handling for a
derived property, and so on.

So I guess if we go by the version numbers, we'd need two, one for the
GeoServer object model, and one for the JDBCConfig
interpretation of it

I'm just worried about the GeoServer object model part, will people
remember to update the number when new direct
fields, or new objects are added into the model?
Maybe JDBCConfig could do a hash of the objects and direct fields, and
compare with one stored in the database.
For nested fields we could also do the same... the issue is with the
derived properties, which are handled programmatically...
so yeah, there we'd probably need a a version number of sorts, hoping that
whoever works on JDBCconfig remembers
to update it?

Well I think it kind of happens infrequently enough to be manageable, but

yeah something we would have to remember. But yeah, some sort of hash to
automatically detect would be ideal if it worked.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--
Justin Deoliveira
VP Engineering | Boundless <http://boundlessgeo.com/&gt;
jdeolive@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;

6

On Wed, Nov 19, 2014 at 4:25 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Well I think it kind of happens infrequently enough to be manageable, but

yeah something we would have to remember. But yeah, some sort of hash to
automatically detect would be ideal if it worked.

JDBCConfig takes the classes in org.geosever.catalog.impl.ClassMappings,
and uses reflection to find
all bean properties, that's how the "automatic" property indexing is
created (then there is a nested_properties.properties
file for nested properties JDCBConfig wants to index, and some code that
does magic for certain properties, picking
them in different places).

So... limited to what it automatically extracted from the object model, if
we picked the list of class and property names
and made a SHA1 digest of them, we should have a reliable digest that
changes any time any of CatalogInfo subclasses
changes (property wise, that is)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

7

On Wed, Nov 19, 2014 at 8:29 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Wed, Nov 19, 2014 at 4:25 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Well I think it kind of happens infrequently enough to be manageable,

but yeah something we would have to remember. But yeah, some sort of hash
to automatically detect would be ideal if it worked.

JDBCConfig takes the classes in org.geosever.catalog.impl.ClassMappings,
and uses reflection to find
all bean properties, that's how the "automatic" property indexing is
created (then there is a nested_properties.properties
file for nested properties JDCBConfig wants to index, and some code that
does magic for certain properties, picking
them in different places).

So... limited to what it automatically extracted from the object model, if
we picked the list of class and property names
and made a SHA1 digest of them, we should have a reliable digest that
changes any time any of CatalogInfo subclasses
changes (property wise, that is)

Makes sense. Definitely +1 on that approach.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--
Justin Deoliveira
VP Engineering | Boundless <http://boundlessgeo.com/&gt;
jdeolive@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;