[GeoNetwork-users] Thredds harvesting ISO metadata (GN2.10)

GeoNetwork GeoNetwork User
1

I was able to set up a test harvest of our thredds catalog using GN 2.10 but I am still having issues because the harvest does not seem to crawl through and pull in all the ISO metadata.

So, assuming I should be using the harvest instructions Simon provided awhile back:

use 'Extract Unidata dataset discovery metadata' (instead of 'Extract DIF .....'),
output schema iso19139, select the 'thredds-metadata' stylesheet for creating
fragments and the 'HARVESTING TEMPLATE - THREDDS - UNIDATA DISCOVERY' template
in the harvester definition,

... but I do not get a dropdown for the harvesting template (it is a blank box - there are no templates offered there, although there are in the other dropdowns).

I know what it should look like because I do get it with my production 2.8 GN version. Is there some setting or parameter I need to set to make this work as it should in 2.10?

Kathy
<../../../../tdekker/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/8B0OUOWO/www.limno.com>

2

Hi Kathy,

You need to add the iso19139 templates (in Administration), before you will see the thredds harvesting templates (netcdf attributes and thredds metadata) in the dropdown.

Regards,
Craig Jones
Integrated Marine Observing System

On 02/10/14 06:04, Kathy Koch wrote:

I was able to set up a test harvest of our thredds catalog using GN 2.10 but I am still having issues because the harvest does not seem to crawl through and pull in all the ISO metadata.

So, assuming I should be using the harvest instructions Simon provided awhile back:

use 'Extract Unidata dataset discovery metadata' (instead of 'Extract DIF .....'),
output schema iso19139, select the 'thredds-metadata' stylesheet for creating
fragments and the 'HARVESTING TEMPLATE - THREDDS - UNIDATA DISCOVERY' template
in the harvester definition,

... but I do not get a dropdown for the harvesting template (it is a blank box - there are no templates offered there, although there are in the other dropdowns).

I know what it should look like because I do get it with my production 2.8 GN version. Is there some setting or parameter I need to set to make this work as it should in 2.10?

Kathy
<../../../../tdekker/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/8B0OUOWO/www.limno.com>
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

3

Thanks. I must have done that without remembering back when I set up the 2.8 version.

This is closer but still not quite populating the GeoNetwork record the way I would expect. I would expect if there is a good metadata tag in thredds, that GN would fill that in exactly. Maybe there is an issue with the template??? Or one of my settings isn't quite right?

1) The thredds record has a title but the GN is using what looks like the filename

Thredds title in metadata is "Lake Erie - MODIS - Lake Surface Temperature (LST) - Aggregation"
GeoNetwork title appears as "SST/LakeErieSST-Agg"

2) The keywords come in but they are all in one long string separated by commas so the search engine doesn't put them in the keywords search box. Also pulls in some of the keywords but not all of them.

3) I set topic category code on the harvest to 'inland waters'. The thredds iso states 'climatologyMeteorologyAtmosphere' so it wouldn't have surprised me if that overrode my choice, but weirdly (IMHO), the GN record has 'oceans' for its category.

4) Abstract reads 'thredds dataset' instead of the abstract provided in the iso metadata record.

5) Puts the organization name for the responsible party in the individual name field. Does not put in the individual name information that is available in the thredds record.

There are more but these are the biggies.

Kathy
_____________________________________
Kathy Koch
Data Curator
Contractor (LimnoTech / GLOS)
501 Avis Drive, Ann Arbor, MI 48108
Office: 734-332-1200 email:kkoch@anonymised.com
LimnoTech www.limno.com

________________________________________
From: Craig Jones [Craig.Jones@anonymised.com]
Sent: Wednesday, October 01, 2014 5:28 PM
To: geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] Thredds harvesting ISO metadata (GN2.10)

Hi Kathy,

You need to add the iso19139 templates (in Administration), before you
will see the thredds harvesting templates (netcdf attributes and thredds
metadata) in the dropdown.

Regards,
Craig Jones
Integrated Marine Observing System

On 02/10/14 06:04, Kathy Koch wrote:

I was able to set up a test harvest of our thredds catalog using GN 2.10 but I am still having issues because the harvest does not seem to crawl through and pull in all the ISO metadata.

So, assuming I should be using the harvest instructions Simon provided awhile back:

use 'Extract Unidata dataset discovery metadata' (instead of 'Extract DIF .....'),
output schema iso19139, select the 'thredds-metadata' stylesheet for creating
fragments and the 'HARVESTING TEMPLATE - THREDDS - UNIDATA DISCOVERY' template
in the harvester definition,

... but I do not get a dropdown for the harvesting template (it is a blank box - there are no templates offered there, although there are in the other dropdowns).

I know what it should look like because I do get it with my production 2.8 GN version. Is there some setting or parameter I need to set to make this work as it should in 2.10?

Kathy
<../../../../tdekker/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/8B0OUOWO/www.limno.com>
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

4

Hi Kathy,

For the Unidata Data Discovery fragments option, the generated metadata is built using the harvesting template and xsl stylesheet selected (in your case it sounds like you selected the thredds-metadata stylesheet).

The harvesting template and xsl stylesheet provided were originally written as an example of how the generated metadata could be generated following the NetCDF Attribute Convention for Dataset Discovery, as defined at that time (refer http://www.unidata.ucar.edu/software/thredds/v4.3/netcdf-java/formats/DataDiscoveryAttConvention.html). It was expected that these templates and stylesheets would be customised as required, allowing additional client specific metadata to be included and the harvest to be otherwise tailored as required.

You can edit the harvesting template in GeoNetwork, noting that some of the elements in the template are replaced during harvesting (based on matching iso metadata fragments generated by the stylesheet to template elements using id attributes on the template elements).

The stylesheets used can be found in the iso19139 thredds harvester stylesheet directory: <iso19139 schema directory>/convert/ThreddsToFragments.

Re your issues below:

1) the conventions above map the title to the dataset/@name attribute in the thredds metadata
2) yes the stylesheet should split up the keywords as per the conventions - but currently isn't
3) the topic category is currently being sourced from the harvesting template - it should be possible to source this from the harvester settings, but currently it isn't
4) the abstract will be set to 'thredds dataset' if no tds:documentation[@type='summary'] elements are found in the thredds metadata (the element mapped to the abstract in the conventions) - is this where your abstract is stored?
5) creator/name is mapped to both the individual name and the organisation name in the conventions which is probably why this ended up like this

Might be worth creating issues for 2 and 3 and I can look at them when I get time.

Regards,
Craig Jones

On 03/10/14 01:31, Kathy Koch wrote:

Thanks. I must have done that without remembering back when I set up the 2.8 version.

This is closer but still not quite populating the GeoNetwork record the way I would expect. I would expect if there is a good metadata tag in thredds, that GN would fill that in exactly. Maybe there is an issue with the template??? Or one of my settings isn't quite right?

1) The thredds record has a title but the GN is using what looks like the filename

Thredds title in metadata is "Lake Erie - MODIS - Lake Surface Temperature (LST) - Aggregation"
GeoNetwork title appears as "SST/LakeErieSST-Agg"

2) The keywords come in but they are all in one long string separated by commas so the search engine doesn't put them in the keywords search box. Also pulls in some of the keywords but not all of them.

3) I set topic category code on the harvest to 'inland waters'. The thredds iso states 'climatologyMeteorologyAtmosphere' so it wouldn't have surprised me if that overrode my choice, but weirdly (IMHO), the GN record has 'oceans' for its category.

4) Abstract reads 'thredds dataset' instead of the abstract provided in the iso metadata record.

5) Puts the organization name for the responsible party in the individual name field. Does not put in the individual name information that is available in the thredds record.

There are more but these are the biggies.

Kathy
_____________________________________
Kathy Koch
Data Curator
Contractor (LimnoTech / GLOS)
501 Avis Drive, Ann Arbor, MI 48108
Office: 734-332-1200email:kkoch@anonymised.com
LimnoTechwww.limno.com

________________________________________
From: Craig Jones [Craig.Jones@anonymised.com]
Sent: Wednesday, October 01, 2014 5:28 PM
To:geonetwork-users@lists.sourceforge.net
Subject: Re: [GeoNetwork-users] Thredds harvesting ISO metadata (GN2.10)

Hi Kathy,

You need to add the iso19139 templates (in Administration), before you
will see the thredds harvesting templates (netcdf attributes and thredds
metadata) in the dropdown.

Regards,
Craig Jones
Integrated Marine Observing System

On 02/10/14 06:04, Kathy Koch wrote:

I was able to set up a test harvest of our thredds catalog using GN 2.10 but I am still having issues because the harvest does not seem to crawl through and pull in all the ISO metadata.

So, assuming I should be using the harvest instructions Simon provided awhile back:

use 'Extract Unidata dataset discovery metadata' (instead of 'Extract DIF .....'),
output schema iso19139, select the 'thredds-metadata' stylesheet for creating
fragments and the 'HARVESTING TEMPLATE - THREDDS - UNIDATA DISCOVERY' template
in the harvester definition,

... but I do not get a dropdown for the harvesting template (it is a blank box - there are no templates offered there, although there are in the other dropdowns).

I know what it should look like because I do get it with my production 2.8 GN version. Is there some setting or parameter I need to set to make this work as it should in 2.10?

Kathy
<../../../../tdekker/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/8B0OUOWO/www.limno.com>
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained athttp://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained athttp://sourceforge.net/projects/geonetwork