[GeoNetwork-devel] Drafts for published metadata

GeoNetwork GeoNetwork Developer
1

Hi, as Maria indicated I remember from last years Bolsena discussion that Simon suggested to use svn (and/or git) for metadata workflow.

As I remember the idea was as soon as a user starts an editing session, the update is stored as a branch in svn/github. When the metadata is published the change-branch is merged with master.

Recent developments around workflow added an aspect of adding the changed metadata to the index, so users that created metadata, which is not published yet, are also able to find their metadata using q services. But this feature is not directly related to the type of storage we select. I guess the index can also index all latest versions in all git/svn branches.

Certainly using git/svn will be a bigger development as the one Maria suggested, and the Maria approach may be a first step to introducing proper workflow. Because it includes quite a lot of UI challenges also:
- new metadata statuses are introduced that need to be visualised in the UI; draft-created, draft-modified, draft-removed
- new screen elements for request for publication (from/to/comment), list of pending requests/review tasks, review screen (view changes?) with option to deny (from/to/comment)
- notification system to send review requests

2

Hi Paul

Apologies as I don’t remember about the git/svn discussion in Bolsena for the workflow. The developments for the draft metadata were done for 2.8 version for AGIV in Belgium and migrated/improved to 2.10 for Environment Canada. The scope for this work is to migrate all these developments to GN 3.0 and refactor it properly. The actual feature works as described Maria in previous mails, using Lucene and database.

I think is good to discuss any improvements or alternative storage if we think this can be better. But related to your comment:

As I remember the idea was as soon as a user starts an editing session, the update is stored as a branch in svn/github. When the metadata is published the change-branch is merged with master.

I think that can solve the storage, but at least actually searches are done against the lucene index. In any case I guess Simon can provide some feedback about this stuff.

Regards,
Jose García

···

On Wed, Dec 2, 2015 at 9:10 AM, Paul van Genuchten <paul.vangenuchten@anonymised.com> wrote:

Hi, as Maria indicated I remember from last years Bolsena discussion
that Simon suggested to use svn (and/or git) for metadata workflow.

As I remember the idea was as soon as a user starts an editing session,
the update is stored as a branch in svn/github. When the metadata is
published the change-branch is merged with master.

Recent developments around workflow added an aspect of adding the
changed metadata to the index, so users that created metadata, which is
not published yet, are also able to find their metadata using q
services. But this feature is not directly related to the type of
storage we select. I guess the index can also index all latest versions
in all git/svn branches.

Certainly using git/svn will be a bigger development as the one Maria
suggested, and the Maria approach may be a first step to introducing
proper workflow. Because it includes quite a lot of UI challenges also:

  • new metadata statuses are introduced that need to be visualised in the
    UI; draft-created, draft-modified, draft-removed
  • new screen elements for request for publication (from/to/comment),
    list of pending requests/review tasks, review screen (view changes?)
    with option to deny (from/to/comment)
  • notification system to send review requests

Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

–

Vriendelijke groeten / Kind regards,

Jose García


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: +31 (0)318 416664

Please consider the environment before printing this email.

3

Hi Paul,

The svn stuff wasn't just a suggestion - certainly in 2.10.x it is actually implemented. I haven't checked 3.x.x to see whether it works as originally implemented, but the core (at least) appears to have been ported. The original proposal is doco'd at metadatachanges – GeoNetwork opensource Developer website

At the time the proposal was written and after the work was done, the linkage between database commits and subversion repo commits became more difficult to manage - this would need to be reviewed in the light of very large changes that have been made to the way in which persistence is handled in 3.x.x. However, if we went this way it would provide a much more powerful facility than the other proposal (especially as it captures changes to metadata attributes eg. categories, status, permissions etc). No need to implement the whole history mechanism in the U/I to begin with, it would probably be suitable to just implement the functions required to support editing versions.

Cheers,
Simon
________________________________________
From: Paul van Genuchten [paul.vangenuchten@anonymised.com]
Sent: Wednesday, 2 December 2015 7:10 PM
To: geonetwork-devel@lists.sourceforge.net; Pigot, Simon (O&A, Hobart)
Subject: Re: Drafts for published metadata

Hi, as Maria indicated I remember from last years Bolsena discussion
that Simon suggested to use svn (and/or git) for metadata workflow.

As I remember the idea was as soon as a user starts an editing session,
the update is stored as a branch in svn/github. When the metadata is
published the change-branch is merged with master.

Recent developments around workflow added an aspect of adding the
changed metadata to the index, so users that created metadata, which is
not published yet, are also able to find their metadata using q
services. But this feature is not directly related to the type of
storage we select. I guess the index can also index all latest versions
in all git/svn branches.

Certainly using git/svn will be a bigger development as the one Maria
suggested, and the Maria approach may be a first step to introducing
proper workflow. Because it includes quite a lot of UI challenges also:
- new metadata statuses are introduced that need to be visualised in the
UI; draft-created, draft-modified, draft-removed
- new screen elements for request for publication (from/to/comment),
list of pending requests/review tasks, review screen (view changes?)
with option to deny (from/to/comment)
- notification system to send review requests

4

Hi,

Maybe we can implement both (or add it later the repository thing)?

The main issue right now is that it cannot be handled only in the
repository. The lucene index and the database have to know about this
branch so it can be edited and shown on the metadata view.

I think it is a good idea to have branches (and move from svn to git
on the metadata data), but I see it as an addon, specially if we are
not sure if it still works. Did we ever had some kind of gui tool to
view the changes?

Regards,
María.

On Wed, Dec 2, 2015 at 10:00 AM, <Simon.Pigot@anonymised.com> wrote:

Hi Paul,

The svn stuff wasn't just a suggestion - certainly in 2.10.x it is actually implemented. I haven't checked 3.x.x to see whether it works as originally implemented, but the core (at least) appears to have been ported. The original proposal is doco'd at http://trac.osgeo.org/geonetwork/wiki/metadatachanges

At the time the proposal was written and after the work was done, the linkage between database commits and subversion repo commits became more difficult to manage - this would need to be reviewed in the light of very large changes that have been made to the way in which persistence is handled in 3.x.x. However, if we went this way it would provide a much more powerful facility than the other proposal (especially as it captures changes to metadata attributes eg. categories, status, permissions etc). No need to implement the whole history mechanism in the U/I to begin with, it would probably be suitable to just implement the functions required to support editing versions.

Cheers,
Simon

5

Hi Simon, yes I am aware of the current SVN implementation. However I remembered you suggested at some point to extend this mechanism to also include a mechanism for metadata workflow, or did you mean the current implementation does facilitate workflow, in a sense that, because you have a history, you can always revert to previous versions, which can be considered workflow.

Our suggestion is to introduce workflow in a way that metadata is
create/updated in a non-public location, not overwriting anything that is published yet. And as soon as a reviewer 'allows' the create/modify the published vesion is replaced.

In SVN this could be implemented using branches which are committed to
master/trunk as soon as a reviewer has "published" the record. The database only has the metadata that is in master/trunk. The index may have both trunk records and branch records, because editors should be able to find records that they are currently working on.

Op 2-12-2015 om 10:00 schreef Simon.Pigot@anonymised.com:

Hi Paul,

The svn stuff wasn't just a suggestion - certainly in 2.10.x it is actually implemented. I haven't checked 3.x.x to see whether it works as originally implemented, but the core (at least) appears to have been ported. The original proposal is doco'd at http://trac.osgeo.org/geonetwork/wiki/metadatachanges

At the time the proposal was written and after the work was done, the linkage between database commits and subversion repo commits became more difficult to manage - this would need to be reviewed in the light of very large changes that have been made to the way in which persistence is handled in 3.x.x. However, if we went this way it would provide a much more powerful facility than the other proposal (especially as it captures changes to metadata attributes eg. categories, status, permissions etc). No need to implement the whole history mechanism in the U/I to begin with, it would probably be suitable to just implement the functions required to support editing versions.

Cheers,
Simon
________________________________________
From: Paul van Genuchten [paul.vangenuchten@anonymised.com]
Sent: Wednesday, 2 December 2015 7:10 PM
To: geonetwork-devel@lists.sourceforge.net; Pigot, Simon (O&A, Hobart)
Subject: Re: Drafts for published metadata

Hi, as Maria indicated I remember from last years Bolsena discussion
that Simon suggested to use svn (and/or git) for metadata workflow.

As I remember the idea was as soon as a user starts an editing session,
the update is stored as a branch in svn/github. When the metadata is
published the change-branch is merged with master.

Recent developments around workflow added an aspect of adding the
changed metadata to the index, so users that created metadata, which is
not published yet, are also able to find their metadata using q
services. But this feature is not directly related to the type of
storage we select. I guess the index can also index all latest versions
in all git/svn branches.

Certainly using git/svn will be a bigger development as the one Maria
suggested, and the Maria approach may be a first step to introducing
proper workflow. Because it includes quite a lot of UI challenges also:
- new metadata statuses are introduced that need to be visualised in the
UI; draft-created, draft-modified, draft-removed
- new screen elements for request for publication (from/to/comment),
list of pending requests/review tasks, review screen (view changes?)
with option to deny (from/to/comment)
- notification system to send review requests

6

Maria, Heikki developed a view-changes UI for Agiv, based on some xml-div library. But these developments were never brought to master, because the workflow-pull request that was a dependancy for these developments was unfortunately never integrated.

Op 2-12-2015 om 10:08 schreef María Arias de Reyna:

Hi,

Maybe we can implement both (or add it later the repository thing)?

The main issue right now is that it cannot be handled only in the
repository. The lucene index and the database have to know about this
branch so it can be edited and shown on the metadata view.

I think it is a good idea to have branches (and move from svn to git
on the metadata data), but I see it as an addon, specially if we are
not sure if it still works. Did we ever had some kind of gui tool to
view the changes?

Regards,
María.

On Wed, Dec 2, 2015 at 10:00 AM, <Simon.Pigot@anonymised.com> wrote:

Hi Paul,

The svn stuff wasn't just a suggestion - certainly in 2.10.x it is actually implemented. I haven't checked 3.x.x to see whether it works as originally implemented, but the core (at least) appears to have been ported. The original proposal is doco'd at http://trac.osgeo.org/geonetwork/wiki/metadatachanges

At the time the proposal was written and after the work was done, the linkage between database commits and subversion repo commits became more difficult to manage - this would need to be reviewed in the light of very large changes that have been made to the way in which persistence is handled in 3.x.x. However, if we went this way it would provide a much more powerful facility than the other proposal (especially as it captures changes to metadata attributes eg. categories, status, permissions etc). No need to implement the whole history mechanism in the U/I to begin with, it would probably be suitable to just implement the functions required to support editing versions.

Cheers,
Simon

7

Hi Maria,

If it doesn't work as implemented then we should fix it! :slight_smile: There is the facility to turn on version control in 3.x.x - its in the editor under settings. It doesn't seem to work in 3.x.x (at least for the H2 db) - throws an exception from somewhere within the spring transaction innards.

No gui tool in GN - we were relying on external repository viewers like viewvc.

On the surface at least, it seems the svn part could be replaced with jgit fairly easily.

Cheers,
Simon

________________________________________
From: María Arias de Reyna [delawen@anonymised.com]
Sent: Wednesday, 2 December 2015 8:08 PM
To: Pigot, Simon (O&A, Hobart)
Cc: Paul van Genuchten; Devel geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] Drafts for published metadata

Hi,

Maybe we can implement both (or add it later the repository thing)?

The main issue right now is that it cannot be handled only in the
repository. The lucene index and the database have to know about this
branch so it can be edited and shown on the metadata view.

I think it is a good idea to have branches (and move from svn to git
on the metadata data), but I see it as an addon, specially if we are
not sure if it still works. Did we ever had some kind of gui tool to
view the changes?

Regards,
María.

On Wed, Dec 2, 2015 at 10:00 AM, <Simon.Pigot@anonymised.com> wrote:

Hi Paul,

The svn stuff wasn't just a suggestion - certainly in 2.10.x it is actually implemented. I haven't checked 3.x.x to see whether it works as originally implemented, but the core (at least) appears to have been ported. The original proposal is doco'd at metadatachanges – GeoNetwork opensource Developer website

At the time the proposal was written and after the work was done, the linkage between database commits and subversion repo commits became more difficult to manage - this would need to be reviewed in the light of very large changes that have been made to the way in which persistence is handled in 3.x.x. However, if we went this way it would provide a much more powerful facility than the other proposal (especially as it captures changes to metadata attributes eg. categories, status, permissions etc). No need to implement the whole history mechanism in the U/I to begin with, it would probably be suitable to just implement the functions required to support editing versions.

Cheers,
Simon