[GeoNetwork-devel] Open data ecosystem compatibility (and DCAT)

Jo_Cook2 · August 3, 2021, 8:58am

Hi All,

We’d be really interested in this.

Coincidentally, last week I managed to get the harvester from https://github.com/metadata101/dcat-ap1.1 working in GeoNetwork 3.10- harvesting the dcat endpoint from a CKAN catalog- bringing in both geospatial and non-geospatial datasets. I only learnt about https://github.com/GIM-be/dcat2 very recently- I think my changes to the harvester code should move across pretty easily, and I’m happy to try to get it working in 3.12. I haven’t yet contributed the code back due to a lack of time though!

All the best

Jo

(attachments)

···

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

Cosyn_Bart · August 4, 2021, 3:33pm

Hi,

In Flanders/Belgium (in collaboration with GIM) we made quite some effort in developing the DCAT schema plugin (https://github.com/metadata101/dcat2. ) together with some devs from the community.

We contributed also to https://github.com/metadata101/dcat-ap1.1. This plugin is only compatible with GeoNetwork 3.6 and 3.8. But currently we focus on https://github.com/metadata101/dcat2.

This is the public version which has been part of the GeoNetwork DCAT2 codesprint April 2021 · geonetwork/core-geonetwork Wiki (github.com) codesprint mentioned before.

The newer https://github.com/GIM-be/dcat2 initiative is a custom implementation of the DCAT2 which contains specific needs for Flanders. We promote to incorporate as much as possible functionality in the community versions.

So I would like to invite you all to contribute to https://github.com/metadata101/dcat2

Regards

Bart Cosyn

DIGITAAL VLAANDEREN

M: +32(0)494564900

Bart.Cosyn@anonymised.com

Havenlaan 88, 1000 Brussel | Koningin Maria Hendrikaplein 70, 9000 Gent

Meer weten? Surf naar www.vlaanderen.be/digitaalvlaanderen

Stel je vraag aan de overheid!

Bezoek www.vlaanderen.be of bel gratis 1700

//////////////////////////////////////////////////////////////////////////////////

Afbeelding met tekst Automatisch gegenereerde beschrijving

(attachments)

···

Van: Jo Cook via GeoNetwork-devel <geonetwork-devel@lists.sourceforge.net>
Verzonden: Tuesday, 3 August 2021 10:58
Aan: Devel geonetwork-devel@lists.sourceforge.net <geonetwork-devel@lists.sourceforge.net>; Florent Gravin <florent.gravin@anonymised.com>
Onderwerp: Re: [GeoNetwork-devel] Open data ecosystem compatibility (and DCAT)

Hi All,

We’d be really interested in this.

Coincidentally, last week I managed to get the harvester from https://github.com/metadata101/dcat-ap1.1 working in GeoNetwork 3.10- harvesting the dcat endpoint from a CKAN catalog- bringing in both geospatial and non-geospatial datasets. I only learnt about https://github.com/GIM-be/dcat2 very recently- I think my changes to the harvester code should move across pretty easily, and I’m happy to try to get it working in 3.12. I haven’t yet contributed the code back due to a lack of time though!

All the best

Jo

---------- Forwarded message ----------
From: Florent Gravin <florent.gravin@anonymised.com>
To: “Devel geonetwork-devel@lists.sourceforge.net” <geonetwork-devel@anonymised.coms.sourceforge.net>
Cc:
Bcc:
Date: Mon, 2 Aug 2021 15:26:47 +0200
Subject: [GeoNetwork-devel] Open data ecosystem compatibility (and DCAT)

Hi everyone,

We are willing to improve the DCAT compatibility in Geonetwork to make Geonetwork able to interact better in the open data ecosystem. Here a list of what we would like to have:

harvest a CKAN node

harvest an Opendatasoft node

harvest a uData node (dat.gouv.fr)

harvest ESRI catalog

be harvested by uData (hopefully by other opendata catalogs)

friendly user DCAT editor

All of these requirements are meant to make GeoNetwork the datahub of a french region (opendata portal for geo and non geo datasets).

At this point, we don’t even know if we are going to use native API of CKAN/ ODS or the DCAT output for the hravesting.

Concerning DCAT, I know there has been a great codesprint sooner in the year to move forward with the DCAT compatibility in the DCAT2 plugin.

I also know about the https://github.com/GIM-be/dcat2 initiative.

I would like to know if anyone in this list, users, contributors or customers would like to work on those aspects with us, to see if it’s opportune to organize some meetings or code sprints.

Also anyone who feels concerned by this topic can show up to help defining what are the correct actions to lead to achieve our goal.

Note that this work is part of a bigger goal to help Geonetwork providing a better experience with open dataset (UI with dataviz tools, a data API and so on…).

Thanks a lot, looking forward to hearing from you

Have a nice day

Best,

–

camptocamp

INNOVATIVE SOLUTIONS

BY OPEN SOURCE EXPERTS

Florent Gravin

Technical Leader - Architect

+33 4 58 48 20 36

---------- Forwarded message ----------
From: Byron Cochrane <byron@anonymised.com>
To: Florent Gravin <florent.gravin@anonymised.com>
Cc: “Devel geonetwork-devel@lists.sourceforge.net” <geonetwork-devel@anonymised.coms.sourceforge.net>
Bcc:
Date: Tue, 3 Aug 2021 13:26:45 +1200
Subject: Re: [GeoNetwork-devel] Open data ecosystem compatibility (and DCAT)

Hi Florent,

We would be very keen to be involved in these DCAT activities you describe.

There are a few initiatives here in the Antipodes that could use these type of improvements.

Are you thinking of a schema plugin to store DCAT natively in GeoNetwork or something similar?

We would be particularly keen to contribute towards the creation of a user friendly DCAT editor. Recently we created user friendly a GeoNetwork 19115-3 editor for ANZLIC. So many of the editing aids would be so much easier if we could better leverage RDF.

Cheers,

Byron Cochrane

OpenWork Ltd
Nelson, New Zealand
byron@anonymised.com
+64 21 794 501
https://www.openwork.nz/

“The whole problem with the world is that fools and fanatics are so certain of themselves, yet wiser people so full of doubt” - Bertrand Russell
"Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it’s the only thing that ever has.” - Margaret Mead

On 3/08/2021, at 1:26 AM, Florent Gravin via GeoNetwork-devel <geonetwork-devel@lists.sourceforge.net> wrote:

Hi everyone,

We are willing to improve the DCAT compatibility in Geonetwork to make Geonetwork able to interact better in the open data ecosystem. Here a list of what we would like to have:

harvest a CKAN node

harvest an Opendatasoft node

harvest a uData node (dat.gouv.fr)

harvest ESRI catalog

be harvested by uData (hopefully by other opendata catalogs)

friendly user DCAT editor

All of these requirements are meant to make GeoNetwork the datahub of a french region (opendata portal for geo and non geo datasets).

At this point, we don’t even know if we are going to use native API of CKAN/ ODS or the DCAT output for the hravesting.

Concerning DCAT, I know there has been a great codesprint sooner in the year to move forward with the DCAT compatibility in the DCAT2 plugin.

I also know about the https://github.com/GIM-be/dcat2 initiative.

I would like to know if anyone in this list, users, contributors or customers would like to work on those aspects with us, to see if it’s opportune to organize some meetings or code sprints.

Also anyone who feels concerned by this topic can show up to help defining what are the correct actions to lead to achieve our goal.

Note that this work is part of a bigger goal to help Geonetwork providing a better experience with open dataset (UI with dataviz tools, a data API and so on…).

Thanks a lot, looking forward to hearing from you

Have a nice day

Best,

–

camptocamp

INNOVATIVE SOLUTIONS

BY OPEN SOURCE EXPERTS

Florent Gravin

Technical Leader - Architect

+33 4 58 48 20 36

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek

Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, 120 Manor Green Road, Epsom, Surrey, KT19 8LN, UK

t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform

GeoServer, PostGIS and QGIS training

Helpdesk and customer portal

^{Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.}

fgravin · September 3, 2021, 7:29am

Hi all,

Thanks for responding and giving inputs.

Coincidentally, last week I managed to get the harvester from https://github.com/metadata101/dcat-ap1.1 working in GeoNetwork 3.10- harvesting the dcat endpoint from a CKAN catalog- bringing in both geospatial and non-geospatial datasets

@Jo Cook sorry it’s confusing (i am not english), did you pull the metadata from the CKAN into GeoNetwork or the opposite ?
What do you think about the CKAN DCAT output ? format ?

Has anyone analysed the DCAT implementation of CKAN, is there a big difference with the Geonetwork implementation, or the 2 DCATS are mostly similar ?

Would you have CKAN DCAT entrypoints to figure it out ?

Thanks

(attachments)

···

camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS

Florent Gravin
Technical Leader - Architect
+33 4 58 48 20 36

Jo_Cook2 · September 3, 2021, 8:14am

Hi Florent,

I actually have two projects- one which is pulling data from CKAN into GeoNetwork and one which is pulling data from GeoNetwork into CKAN.

This is the CKAN endpoint I am using for pulling data into GeoNetwork: https://test.data.spatialhub.scot/catalog.xml.

For pulling data from GeoNetwork into CKAN- it’s data.gov.uk that I’m trying to pull data into, but I’m using a local copy based on https://github.com/alphagov/docker-ckan. Then I’m using the information here https://guidance.data.gov.uk/publish_and_manage_data/harvest_or_add_data/harvest_data/dcat/#accepted-dcat-and-data-json-fields to create the right output. I’ve had to make some changes to the GeoNetwork code to get valid results, mostly just to get rdflib to validate the output.

I’m not quite at the stage where I’d be happy to formally commit my code changes back to GeoNetwork for either project but I’m happy to share what I’ve done.

All the best

Jo

(attachments)

···

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

fgravin · September 3, 2021, 10:06am

Thanks Jo for the informations.
I am a bit lost with the state of this in GN, for instance i don’t find any DCAT harvest in the admin console/harvester page.
Do we have a DCAT harvester somewhere, how to use it ?

Thanks

(attachments)

···

camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS

Florent Gravin
Technical Leader - Architect
+33 4 58 48 20 36

Jo_Cook2 · September 6, 2021, 8:39am

Hi Florent,

I managed to get the DCAT harvester from https://github.com/metadata101/dcat-ap1.1 working in GeoNetwork 3.10, with a few code changes. There might be a few more to get it working in GeoNetwork 3.12 and later though. Let me see if I can get a copy of my code onto GitHub today.

All the best

Jo

(attachments)

···

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

fgravin · September 22, 2021, 10:54am

Hi all,

I have merged the JSON harvester PR, with 2 XSL that transforms ESRI JSON-LD DCAT and OPENDATASOFT V1 to DCAT2 format.
https://github.com/geonetwork/core-geonetwork/pull/5942

As the dcat2 is not completed, the harvested records are not very well integrated into the catalog (mostly indexing issues, also editing issues).

I have a concern about the distributions, that are not really mapped to the ISO online resources.
looking at this distribution from ESRI
{
“@type”: “dcat:Distribution”,
“title”: “Shapefile”,
“format”: “ZIP”,
“mediaType”: “application/zip”,
“accessURL”: “https://data-atmo-hdf.opendata.arcgis.com/datasets/e5cb0148704b492f83725fe08c03b116_1.zip?outSR=%7B%22latestWkid%22%3A2154%2C%22wkid%22%3A102110%7D”
}

I wonder how we can guess that it is a shapefile, and how we should put it in the index (also how would it be represented in the ISO) ?
the index link object has

url
name
description
protocol
function
applicationProfile

Shall we add a new property for the format ?
What would be the protocol in my usecase ? WWW:DOWNLOAD ?
Can we have a WWW:DOWNLOAD:SHP for instance ?

The issue i have is that I want to list all available exports from the dataset, and from what I have in the index (which mostly comes from ISO), i have no clear implementation to list the available dataset exports. It is hard to guess the mimeType from the url for instance.

Do you have any experience with that, or any suggestion ?
ping @Francois Prunayre as well for the technical part.

Thanks

(attachments)

···

camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS

Florent Gravin
Technical Leader - Architect
+33 4 58 48 20 36