HarvestHistory already stores all params etc and details about current harvesters and all previous harvest sessions - could be that a simpler solution would be to retrieve the harvester definition from there rather than the settings table?
It has harvesteruuid, harvestername, harvestertype exposed as sql columns (for indexing as keys) with the params as an XML string stored in another column.
Cheers,
Simon
________________________________________
From: heikki [tropicano@anonymised.com]
Sent: Tuesday, 7 May 2013 8:11 PM
To: Jesse Eichar
Cc: Devel geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] Storing harvesters out of Settings
SELECT key, value FROM harvester WHERE uuid=?
gives a list of all params and their values for a specific harvester
On Tue, May 7, 2013 at 12:07 PM, Jesse Eichar <jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>> wrote:
I don't have a strong opinion other than we need to get them out of settings. I favor 1 & 2 over 3 myself.
Once requirement that would be nice would be to have any easy way to get all the parameters of a harvester easily via SQL.
so a harvester table with a link to a params table or something might be nice.
Jesse
On Tue, May 7, 2013 at 11:57 AM, heikki <tropicano@anonymised.com<mailto:tropicano@anonymised.com>> wrote:
.. with CASCADE DELETE you would delete harvesters associated to a category, when you delete a category. I'm not very sure that is what we want either 
The trouble with table columns is is that there are many, many harvester properties that are only used in one or a few harvesters. Or that have optional values. The table would become very sparse, I think. This is probably the reason that is was stuffed into Settings in the first place.
For the moment I'm leaning toward the KVP table option. It can always be migrated to a more complex structure if that's deemed useful.
On Tue, May 7, 2013 at 11:50 AM, Jose Garcia <jose.garcia@anonymised.com<mailto:jose.garcia@anonymised.com>> wrote:
Hi Heikki
Depends on the options of foreign keys, at least some databases allow CASCADE DELETE, but not sure if this is standard.
But I was meaning mostly related to proper model of information in table columns instead of KVP (where categories for example can be stored as a comma separated list, without additional table if you prefer). If in any moment we move database code to use ORM probably this is a better option.
But also not sure if this is going to happen at least in near future, so my vote for any of the options to change actual harvesters storage.
Regards,
Jose García
On Tue, May 7, 2013 at 11:38 AM, heikki <tropicano@anonymised.com<mailto:tropicano@anonymised.com>> wrote:
hi Jose,
why do you think option 2 is better ? Relational integrity to mean e.g. you can't delete a category, if it is associated to a harvester ? We don't have that now, and I am not at all sure that's what we want.
On Tue, May 7, 2013 at 11:17 AM, Jose Garcia <jose.garcia@anonymised.com<mailto:jose.garcia@anonymised.com>> wrote:
Hi Heikki
+1 for this. Depending on time available for this, option 2 would be better I think. For sure the refactor also the client/server code, but not sure if this is on the plan.
I think Mathieu did some time ago some initial work on refactoring the harvesters, so maybe would be a good option to verify with him about status and availability of this code.
Regards,
Jose García
On Tue, May 7, 2013 at 10:57 AM, Francois Prunayre <fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>> wrote:
Hello Heikki,
2013/5/7 heikki <tropicano@anonymised.com<mailto:tropicano@anonymised.com>>:
hello list,
many of us have complained for some time that the harvester's configuration
is stored in Settings. This makes it a little harder to find back things
when you look at the DB and it pollutes the Settings table with this
harvester info.
+1 for the initiative whatever the option.
Some options to do this could be:
(1) a simple KVP-like table, with columns for harvester uuid, key name, and
key value
Sounds good. And we could also apply KVP structure to settings later on.
(2) a more elaborate table structure with separate facilities for privileges
and categories, so relational integrity can be enforced on those. Not sure
if that's even desired though
This option will probably require more work and at some point part of
the configuration will probably be a KVP table due to harvester
differences.
Relational integrity is good, but I think it would be interesting to
go that direction only if the client/server code of the harvester is
also refactored.
(3) dump a harvester config in XML-form in a simple table. We read it only
at startup and write only when saving, which is not so often, and don't do
complex queries on it. Still it may be a bit less readable than a simple KVP
table.
This is simple.
Francois
Do you have an opinion on which of these solutions (or something else) is
preferred ?
Kind regards
Heikki Doeleman
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and
their applications. This 200-page book is written by three acclaimed
leaders in the field. The early access version is available now.
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and
their applications. This 200-page book is written by three acclaimed
leaders in the field. The early access version is available now.
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
--
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net/> for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/>
--
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net/> for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/>
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and
their applications. This 200-page book is written by three acclaimed
leaders in the field. The early access version is available now.
Download your free book today! Best Open Source Mac Front-Ends 2024
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
