[GeoNetwork-users] CKAN csw Metadata Harvesting

Hi All,

Has anyone had any experience in harvesting metadata from CKAN's csw server?

I recently installed a CKAN open source data management system and set up its spatial and harvester extensions. CKAN csw harvester can harvest metadata from GeoNetwork's csw endpoint. However, harvesting metadata from CKAN csw server into my GeoNetwork's instance (see two endpoints listed below) doesn't yield expected results.

The harvesting process successfully run and created only one ISO19139 metadata record (see the attached XML file) in my GN's instance. In that CKAN instance, there're about 53 datasets (see CKAN csw endpoint #2 below). So, I was expecting to see 53 metadata records to be harvested into my GN.

CKAN csw endpoint #1
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW

CKAN csw endpoint #2
http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full

I'm not sure at this point in time if I've mis-configured GN's OGC Catalogue Service (CSW) version 2.0.2 harvester or CKAN's csw service is not supported by GN.

Last but not least, if you've used CKAN as a catalogue service and have successfully harvested CKAN's datasets into GN, mind to share your "how-to" tips and tricks. Thanks.

Kind regards,
Richard Goh

(attachments)

CKAN-CSW-iso19139.xml (13.3 KB)

Hi Richard

Which version of GeoNetwork are you using? I have tried with 2.10 and it's
harvesting 53 metadata.

Regards,
Jose García

On Mon, Aug 19, 2013 at 11:03 AM, <Richard.Goh@anonymised.com> wrote:

Hi All,

Has anyone had any experience in harvesting metadata from CKAN's csw
server?

I recently installed a CKAN open source data management system and set up
its spatial and harvester extensions. CKAN csw harvester can harvest
metadata from GeoNetwork's csw endpoint. However, harvesting metadata from
CKAN csw server into my GeoNetwork's instance (see two endpoints listed
below) doesn't yield expected results.

The harvesting process successfully run and created only one ISO19139
metadata record (see the attached XML file) in my GN's instance. In that
CKAN instance, there're about 53 datasets (see CKAN csw endpoint #2 below).
So, I was expecting to see 53 metadata records to be harvested into my GN.

CKAN csw endpoint #1
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW

CKAN csw endpoint #2

http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full

I'm not sure at this point in time if I've mis-configured GN's OGC
Catalogue Service (CSW) version 2.0.2 harvester or CKAN's csw service is
not supported by GN.

Last but not least, if you've used CKAN as a catalogue service and have
successfully harvested CKAN's datasets into GN, mind to share your "how-to"
tips and tricks. Thanks.

Kind regards,
Richard Goh

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--
*
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on
GeoServer and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/&gt;

*

Hi Jose,

Many thanks for your reply. The version of GeoNetwork I used is 2.7.0.

I just tried the latest one which is 2.10.1-0 and I still harvested only one metadata.

There must be something wrong with my harvesting configuration.

The following shows the configuration I used:

1. Log in as admin and go to Administration page then to Harvesting Management page.

2. Add a new harvesting rule which is of type OGC Web services (i.e. WMS, WFS, WCS, WPS, CSW, SOS)

3. On the harvesting data entry page, I

Name: ckan-metadata
Type of OGC web service: OGC Catalogue Service (CSW) Version 2.0.2
Service: Tried the following URLs: http://118.138.240.163:8080/csw, http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW and http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full
User account: unchecked
Metadata language: eng
ISO topic category: blank
Type of import: Tried no options and also with all options
Target schema: iso19139
Run options (I run it manually)
Privileges: Add All
Category for Service & datasets: Default values

Do you mind sharing your workflow or configuration on how you harvested 53 metadata records? Thanks in advance.

Kind regards,
Richard Goh

From: Jose Garcia [mailto:jose.garcia@…444…]
Sent: Monday, 19 August 2013 6:07 PM
To: Goh, Richard (CESRE, Kensington)
Cc: geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] CKAN csw Metadata Harvesting

Hi Richard

Which version of GeoNetwork are you using? I have tried with 2.10 and it's harvesting 53 metadata.

Regards,
Jose García

On Mon, Aug 19, 2013 at 11:03 AM, <Richard.Goh@...448...<mailto:Richard.Goh@…448…>> wrote:
Hi All,

Has anyone had any experience in harvesting metadata from CKAN's csw server?

I recently installed a CKAN open source data management system and set up its spatial and harvester extensions. CKAN csw harvester can harvest metadata from GeoNetwork's csw endpoint. However, harvesting metadata from CKAN csw server into my GeoNetwork's instance (see two endpoints listed below) doesn't yield expected results.

The harvesting process successfully run and created only one ISO19139 metadata record (see the attached XML file) in my GN's instance. In that CKAN instance, there're about 53 datasets (see CKAN csw endpoint #2 below). So, I was expecting to see 53 metadata records to be harvested into my GN.

CKAN csw endpoint #1
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW

CKAN csw endpoint #2
http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full

I'm not sure at this point in time if I've mis-configured GN's OGC Catalogue Service (CSW) version 2.0.2 harvester or CKAN's csw service is not supported by GN.

Last but not least, if you've used CKAN as a catalogue service and have successfully harvested CKAN's datasets into GN, mind to share your "how-to" tips and tricks. Thanks.

Kind regards,
Richard Goh

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net<mailto:GeoNetwork-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net/&gt; for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/&gt;

Hi Richard

There's a specific CSW harvester (Catalogue Services for the Web ISO
profile 2.0), please use it instead. I setup this url in the harvester:
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW and
works fine.

The OGC Web Services harvester, harvest only the capabilities document of
the OGC service configured, that explains why you get only 1 record.

Regards,
Jose García

On Tue, Aug 20, 2013 at 2:16 AM, <Richard.Goh@anonymised.com> wrote:

Hi Jose,****

** **

Many thanks for your reply. The version of GeoNetwork I used is 2.7.0.****

** **

I just tried the latest one which is 2.10.1-0 and I still harvested only
one metadata.****

** **

There must be something wrong with my harvesting configuration. ****

** **

The following shows the configuration I used:****

** **

**1. **Log in as admin and go to Administration page then to
Harvesting Management page.****

**2. **Add a new harvesting rule which is of type OGC Web services
(i.e. WMS, WFS, WCS, WPS, CSW, SOS)****

**3. **On the harvesting data entry page, I ****

** **

Name: ckan-metadata****

Type of OGC web service: OGC Catalogue Service (CSW) Version 2.0.2****

Service: Tried the following URLs: http://118.138.240.163:8080/csw,
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW and
http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full
****

User account: unchecked****

Metadata language: eng****

ISO topic category: blank****

Type of import: Tried no options and also with all options****

Target schema: iso19139****

Run options (I run it manually)****

Privileges: Add All****

Category for Service & datasets: Default values****

** **

Do you mind sharing your workflow or configuration on how you harvested 53
metadata records? Thanks in advance.****

** **

Kind regards,****

Richard Goh****

** **

*From:* Jose Garcia [mailto:jose.garcia@anonymised.com]
*Sent:* Monday, 19 August 2013 6:07 PM
*To:* Goh, Richard (CESRE, Kensington)
*Cc:* geonetwork-users@lists.sourceforge.net
*Subject:* Re: [GeoNetwork-users] CKAN csw Metadata Harvesting****

** **

Hi Richard****

** **

Which version of GeoNetwork are you using? I have tried with 2.10 and it's
harvesting 53 metadata.****

Regards,****

Jose García****

****

** **

On Mon, Aug 19, 2013 at 11:03 AM, <Richard.Goh@anonymised.com> wrote:****

Hi All,

Has anyone had any experience in harvesting metadata from CKAN's csw
server?

I recently installed a CKAN open source data management system and set up
its spatial and harvester extensions. CKAN csw harvester can harvest
metadata from GeoNetwork's csw endpoint. However, harvesting metadata from
CKAN csw server into my GeoNetwork's instance (see two endpoints listed
below) doesn't yield expected results.

The harvesting process successfully run and created only one ISO19139
metadata record (see the attached XML file) in my GN's instance. In that
CKAN instance, there're about 53 datasets (see CKAN csw endpoint #2 below).
So, I was expecting to see 53 metadata records to be harvested into my GN.

CKAN csw endpoint #1
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW

CKAN csw endpoint #2

http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full

I'm not sure at this point in time if I've mis-configured GN's OGC
Catalogue Service (CSW) version 2.0.2 harvester or CKAN's csw service is
not supported by GN.

Last but not least, if you've used CKAN as a catalogue service and have
successfully harvested CKAN's datasets into GN, mind to share your "how-to"
tips and tricks. Thanks.

Kind regards,
Richard Goh

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork****

****

** **

-- **

GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on
GeoServer and GeoNetwork. Visit http://geocat.net for details. ****

_________________________****

Jose García****

GeoCat bv****

Veenderweg 13****

6721 WD Bennekom****

The Netherlands****

http://GeoCat.net/&gt;\*\*\*\*

** **

--
*
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on
GeoServer and GeoNetwork. Visit http://geocat.net for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/&gt;

*

Hi Jose,

Thanks again for your help. I tried the one you suggested and I managed to get it working.

I can see 53 metadata records harvested into my GeoNetwork server instance.

Cheers
Richard Goh

From: Jose Garcia [mailto:jose.garcia@…444…]
Sent: Tuesday, 20 August 2013 2:08 PM
To: Goh, Richard (CESRE, Kensington)
Cc: geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] CKAN csw Metadata Harvesting

Hi Richard

There's a specific CSW harvester (Catalogue Services for the Web ISO profile 2.0), please use it instead. I setup this url in the harvester: http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW and works fine.

The OGC Web Services harvester, harvest only the capabilities document of the OGC service configured, that explains why you get only 1 record.

Regards,
Jose García

On Tue, Aug 20, 2013 at 2:16 AM, <Richard.Goh@...448...<mailto:Richard.Goh@…448…>> wrote:
Hi Jose,

Many thanks for your reply. The version of GeoNetwork I used is 2.7.0.

I just tried the latest one which is 2.10.1-0 and I still harvested only one metadata.

There must be something wrong with my harvesting configuration.

The following shows the configuration I used:

1. Log in as admin and go to Administration page then to Harvesting Management page.

2. Add a new harvesting rule which is of type OGC Web services (i.e. WMS, WFS, WCS, WPS, CSW, SOS)

3. On the harvesting data entry page, I

Name: ckan-metadata
Type of OGC web service: OGC Catalogue Service (CSW) Version 2.0.2
Service: Tried the following URLs: http://118.138.240.163<tel:118.138.240.163>:8080/csw, http://118.138.240.163:8080/csw?request=GetCapabilities&amp;service=CSW and http://118.138.240.163:8080/csw?request=GetRecords&amp;service=CSW&amp;resultType=results&amp;elementSetName=full
User account: unchecked
Metadata language: eng
ISO topic category: blank
Type of import: Tried no options and also with all options
Target schema: iso19139
Run options (I run it manually)
Privileges: Add All
Category for Service & datasets: Default values

Do you mind sharing your workflow or configuration on how you harvested 53 metadata records? Thanks in advance.

Kind regards,
Richard Goh

From: Jose Garcia [mailto:jose.garcia@…444…]
Sent: Monday, 19 August 2013 6:07 PM
To: Goh, Richard (CESRE, Kensington)
Cc: geonetwork-users@lists.sourceforge.net<mailto:geonetwork-users@lists.sourceforge.net>
Subject: Re: [GeoNetwork-users] CKAN csw Metadata Harvesting

Hi Richard

Which version of GeoNetwork are you using? I have tried with 2.10 and it's harvesting 53 metadata.

Regards,
Jose García

On Mon, Aug 19, 2013 at 11:03 AM, <Richard.Goh@...448...<mailto:Richard.Goh@…448…>> wrote:
Hi All,

Has anyone had any experience in harvesting metadata from CKAN's csw server?

I recently installed a CKAN open source data management system and set up its spatial and harvester extensions. CKAN csw harvester can harvest metadata from GeoNetwork's csw endpoint. However, harvesting metadata from CKAN csw server into my GeoNetwork's instance (see two endpoints listed below) doesn't yield expected results.

The harvesting process successfully run and created only one ISO19139 metadata record (see the attached XML file) in my GN's instance. In that CKAN instance, there're about 53 datasets (see CKAN csw endpoint #2 below). So, I was expecting to see 53 metadata records to be harvested into my GN.

CKAN csw endpoint #1
http://118.138.240.163:8080/csw?request=GetCapabilities&service=CSW

CKAN csw endpoint #2
http://118.138.240.163:8080/csw?request=GetRecords&service=CSW&resultType=results&elementSetName=full

I'm not sure at this point in time if I've mis-configured GN's OGC Catalogue Service (CSW) version 2.0.2 harvester or CKAN's csw service is not supported by GN.

Last but not least, if you've used CKAN as a catalogue service and have successfully harvested CKAN's datasets into GN, mind to share your "how-to" tips and tricks. Thanks.

Kind regards,
Richard Goh

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net<mailto:GeoNetwork-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net/&gt; for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/&gt;

--
GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net/&gt; for details.
_________________________
Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net/&gt;