[GeoNetwork-users] approaches to hierarchical ISO19115 metadata

Herbert_David_J · March 5, 2010, 10:24am

Hello,

I have been evaluating Geonetwork 2.4.2 to support an geographic image catalogue project we're doing at British Antarctic Survey. I have hit problems due to the hierarchical nature of much of the data I am going to be dealing with. GN clearly works out of the box with simple flat categories containing e.g. images. What I need to model is a hierarchy of e.g. image products, or metadata for aerial photographs where much of the metadata (e.g. camera settings) will be common to a sortie flight, and only a small subset will belong to an actual image.

ISO19115 / 19139 has provision for rich hierarchical data, for example through parentIdentifier, hierarchyLevel and hierarchyLevelName under MD_Metadata. ISO19115-2 also allows for a hierarchy of missions and events.

What I am struggling with is how, using either GN portal services or (preferably) CSW I can achieve this hierarchical model. I have been asked to write a tree-explorer-like front end to drill down into our data holdings, so I am needing CSW GetRecords queries using a filter where parentIdentifier = some id. Despite parentIdentifier being listed as an ISO Core Queryable for GN, I am unable to get such a query to work (always returns no results, doesn't give any error message). OrganisationName (for example) works fine in the same query context. A helpful poster on this list informed me that GN may not have parentIdentifier in its Lucene index. Not being an XSL expert (though very happy with Java) I have been unable to find out how to add this field to the index. Can anyone enlighten me here?

I have also looked into the parallel possibility of using GN's own relations between metadata. The manual helpfully tells me how to use xml.relation.get to find relations (thus solving the querying problem above). However I am unable to find out the parameters to its stablemates xml.relation.insert and xml.relation.delete to create the relations in the first place - looking in config.xml tells me the Java class used but not the parameter names and types that the service accepts. I find this to be true of all the Portal services (e.g. portal.search.present). Am I just missing something obvious, or does one have to resort to looking at the Java source code from svn to figure out what the parameters to these services are? The Javadoc doesn't help as all the methods take Jeeves wrapper objects. Could someone tell me where I can find this information?

I have also read posts on the dev forum that suggest that there might be core support for more hierarchical data in a future GN version, and that there may be a patch I can try with 2.4.2?

Thanks in advance for any insight into these problems,

David Herbert
British Antarctic Survey.

ajs6f · March 5, 2010, 1:43pm

For adding fields to the Lucene index, take a look at:

http://geonetwork-opensource.org/documentation/how-to/geonetwork-v2-2-shema-template-howto

specifically:

http://geonetwork-opensource.org/documentation/how-to/geonetwork-v2-2-shema-template-howto/#S5

If I understand the process rightly, there's a single stylesheet for extracting index info, and a stylesheet to assemble Lucene queries from service requests. Of course, you are looking to alter the ISO schema-related 'sheets, not create new ones, but the process should be similar. I hope that's what you were trying to find. Or is it a question of how to alter the indexing XSLT to add the field in question?

My experience has been that browsing the source code to understand the Jeeves services is pretty much what you have to do. Luckily, it's fairly easy-to-read source and it doesn't take long.

Another member of this list, Christopher Miller of Purdue University, chose a different and clever path around indexing needs. He aimed an instance of Apache Solr:

http://lucene.apache.org/solr/

a fantastic Web front-end for Lucene which offers great faceting action, at the GN index, and is thereby able to query the index directly using Solr's straightforward GET interface. That might be the basis for a different approach. (Other than using the Jeeves services or CSW). I'm sure Mr. Miller would be willing to say a little more about that if it sounds interesting to the list.

---
A. Soroka
Digital Research and Scholarship R & D
the University of Virginia Library

On Mar 5, 2010, at 5:24 AM, Herbert, David J wrote:

Hello,

I have been evaluating Geonetwork 2.4.2 to support an geographic image catalogue project we're doing at British Antarctic Survey. I have hit problems due to the hierarchical nature of much of the data I am going to be dealing with. GN clearly works out of the box with simple flat categories containing e.g. images. What I need to model is a hierarchy of e.g. image products, or metadata for aerial photographs where much of the metadata (e.g. camera settings) will be common to a sortie flight, and only a small subset will belong to an actual image.

ISO19115 / 19139 has provision for rich hierarchical data, for example through parentIdentifier, hierarchyLevel and hierarchyLevelName under MD_Metadata. ISO19115-2 also allows for a hierarchy of missions and events.

What I am struggling with is how, using either GN portal services or (preferably) CSW I can achieve this hierarchical model. I have been asked to write a tree-explorer-like front end to drill down into our data holdings, so I am needing CSW GetRecords queries using a filter where parentIdentifier = some id. Despite parentIdentifier being listed as an ISO Core Queryable for GN, I am unable to get such a query to work (always returns no results, doesn't give any error message). OrganisationName (for example) works fine in the same query context. A helpful poster on this list informed me that GN may not have parentIdentifier in its Lucene index. Not being an XSL expert (though very happy with Java) I have been unable to find out how to add this field to the index. Can anyone enlighten me here?

I have also looked into the parallel possibility of using GN's own relations between metadata. The manual helpfully tells me how to use xml.relation.get to find relations (thus solving the querying problem above). However I am unable to find out the parameters to its stablemates xml.relation.insert and xml.relation.delete to create the relations in the first place - looking in config.xml tells me the Java class used but not the parameter names and types that the service accepts. I find this to be true of all the Portal services (e.g. portal.search.present). Am I just missing something obvious, or does one have to resort to looking at the Java source code from svn to figure out what the parameters to these services are? The Javadoc doesn't help as all the methods take Jeeves wrapper objects. Could someone tell me where I can find this information?

I have also read posts on the dev forum that suggest that there might be core support for more hierarchical data in a future GN version, and that there may be a patch I can try with 2.4.2?

Thanks in advance for any insight into these problems,

David Herbert
British Antarctic Survey.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

C.C_Miller · March 5, 2010, 4:13pm

David,

I am certainly willing to talk about how/what we added to the index, but I think you're right that you shouldn't have to. One way to start troubleshooting this would be to determine if it's a CSW or GN+Lucene problem, and I suppose I would start by running Luke to crack open the Lucene index and see if there is indeed a parentId value in there. If it's there, you can try to query against it (assuming you have metadata with a parentId value, which I presume).

I just checked a gn instance I have from svn and it's in there. Querying in Luke works, as does a query using the native GN UI. This exposed a problem I have with CSW, however, so I haven't been able to check the same query using FILTER in CSW (which would probably be the most helpful bit of information I could have offered, of course).

Chris

On Mar 5, 2010, at 8:43 AM, <ajs6f@anonymised.com<mailto:ajs6f@anonymised.com>> <ajs6f@anonymised.com<mailto:ajs6f@anonymised.com>> wrote:

For adding fields to the Lucene index, take a look at:

http://geonetwork-opensource.org/documentation/how-to/geonetwork-v2-2-shema-template-howto

specifically:

http://geonetwork-opensource.org/documentation/how-to/geonetwork-v2-2-shema-template-howto/#S5

If I understand the process rightly, there's a single stylesheet for extracting index info, and a stylesheet to assemble Lucene queries from service requests. Of course, you are looking to alter the ISO schema-related 'sheets, not create new ones, but the process should be similar. I hope that's what you were trying to find. Or is it a question of how to alter the indexing XSLT to add the field in question?

My experience has been that browsing the source code to understand the Jeeves services is pretty much what you have to do. Luckily, it's fairly easy-to-read source and it doesn't take long.

Another member of this list, Christopher Miller of Purdue University, chose a different and clever path around indexing needs. He aimed an instance of Apache Solr:

http://lucene.apache.org/solr/

a fantastic Web front-end for Lucene which offers great faceting action, at the GN index, and is thereby able to query the index directly using Solr's straightforward GET interface. That might be the basis for a different approach. (Other than using the Jeeves services or CSW). I'm sure Mr. Miller would be willing to say a little more about that if it sounds interesting to the list.

---
A. Soroka
Digital Research and Scholarship R & D
the University of Virginia Library

On Mar 5, 2010, at 5:24 AM, Herbert, David J wrote:

Hello,

I have been evaluating Geonetwork 2.4.2 to support an geographic image catalogue project we're doing at British Antarctic Survey. I have hit problems due to the hierarchical nature of much of the data I am going to be dealing with. GN clearly works out of the box with simple flat categories containing e.g. images. What I need to model is a hierarchy of e.g. image products, or metadata for aerial photographs where much of the metadata (e.g. camera settings) will be common to a sortie flight, and only a small subset will belong to an actual image.

ISO19115 / 19139 has provision for rich hierarchical data, for example through parentIdentifier, hierarchyLevel and hierarchyLevelName under MD_Metadata. ISO19115-2 also allows for a hierarchy of missions and events.

What I am struggling with is how, using either GN portal services or (preferably) CSW I can achieve this hierarchical model. I have been asked to write a tree-explorer-like front end to drill down into our data holdings, so I am needing CSW GetRecords queries using a filter where parentIdentifier = some id. Despite parentIdentifier being listed as an ISO Core Queryable for GN, I am unable to get such a query to work (always returns no results, doesn't give any error message). OrganisationName (for example) works fine in the same query context. A helpful poster on this list informed me that GN may not have parentIdentifier in its Lucene index. Not being an XSL expert (though very happy with Java) I have been unable to find out how to add this field to the index. Can anyone enlighten me here?

I have also looked into the parallel possibility of using GN's own relations between metadata. The manual helpfully tells me how to use xml.relation.get to find relations (thus solving the querying problem above). However I am unable to find out the parameters to its stablemates xml.relation.insert and xml.relation.delete to create the relations in the first place - looking in config.xml tells me the Java class used but not the parameter names and types that the service accepts. I find this to be true of all the Portal services (e.g. portal.search.present). Am I just missing something obvious, or does one have to resort to looking at the Java source code from svn to figure out what the parameters to these services are? The Javadoc doesn't help as all the methods take Jeeves wrapper objects. Could someone tell me where I can find this information?

I have also read posts on the dev forum that suggest that there might be core support for more hierarchical data in a future GN version, and that there may be a patch I can try with 2.4.2?

Thanks in advance for any insight into these problems,

David Herbert
British Antarctic Survey.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C.C. Miller
Assistant Professor of Library Science
Geographic Information Systems Specialist
Purdue University Libraries

http://gis.lib.purdue.edu
feed://www.lib.purdue.edu/gis/rss.php

ccmiller@anonymised.com<mailto:ccmiller@anonymised.com>
765.496.9474
IM=ccmiller@anonymised.com<mailto:IM=ccmiller@anonymised.com>
AIM=cecmcgee
Jabber=ccmiller@anonymised.com<mailto:Jabber=ccmiller@anonymised.com>
Twitter=http://twitter.com/pugolian

2215E EAS (CIVL)
Earth & Atmospheric Sciences Library (EAS)
Civil Engineering Building (CIVL)
550 Stadium Mall Drive
West Lafayette, IN 47907-2051
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Herbert_David_J · March 5, 2010, 5:30pm

Hi Chris,

Thank you (and the other poster) for the help. I was unaware of Luke as a tool for looking inside indices, so this has been interesting. I have looked at my GN index and can't find parentId in there. Nothing else that might be the same field (I know from http://trac.osgeo.org/geonetwork/wiki/CSW202Improvements that ISO field names and their GN equivalents don't always match - please can you confirm the name I'm looking for is "parentId"?) is available. The only name I can find is "relation" but this doesn't seem to be usable via CSW. So I conclude my index hasn't been built including this field.

Could you tell me what URL you used to perform the same query via the native GN UI? I know of portal.search.present, but that only accepts a limited number of possible parameters (I now find, by examining the Java code). I need to understand the Lucene query syntax properly before I can use Luke to perform searches.

Thanks for your help,

David.

-----Original Message-----
From: Miller, Christopher C [mailto:ccmiller@anonymised.com]
Sent: 05 March 2010 16:13
To: geonetwork-users Users
Subject: Re: [GeoNetwork-users] approaches to hierarchical ISO19115 metadata

David,

I am certainly willing to talk about how/what we added to the index, but I think you're right that you shouldn't have to. One way to start troubleshooting this would be to determine if it's a CSW or GN+Lucene problem, and I suppose I would start by running Luke to crack open the Lucene index and see if there is indeed a parentId value in there. If it's there, you can try to query against it (assuming you have metadata with a parentId value, which I presume).

I just checked a gn instance I have from svn and it's in there. Querying in Luke works, as does a query using the native GN UI. This exposed a problem I have with CSW, however, so I haven't been able to check the same query using FILTER in CSW (which would probably be the most helpful bit of information I could have offered, of course).

Chris

C.C_Miller · March 8, 2010, 6:57pm

David,

It is parentId in the index, yeah. See http://www.lib.purdue.edu/gis/support//gis.lib-20100308-135328.png for a screencap (noting that the lone value shown there is indeed a hand-written dummy value).

As for the native UI query, I may have been misleading. I didn't do a search specifically for parentId=<some value>. I just mean that parentId is evidently included in queries against "all", as in...

http://server:port/geonetwork/srv/en/main.search.embedded?any=09090909090909&sortBy=relevance&hitsPerPage=10&output=full

...where '09090909090909' is that dummy parentId value.

On Mar 5, 2010, at 12:30 PM, Herbert, David J wrote:

Hi Chris,

Thank you (and the other poster) for the help. I was unaware of Luke as a tool for looking inside indices, so this has been interesting. I have looked at my GN index and can't find parentId in there. Nothing else that might be the same field (I know from http://trac.osgeo.org/geonetwork/wiki/CSW202Improvements that ISO field names and their GN equivalents don't always match - please can you confirm the name I'm looking for is "parentId"?) is available. The only name I can find is "relation" but this doesn't seem to be usable via CSW. So I conclude my index hasn't been built including this field.

Could you tell me what URL you used to perform the same query via the native GN UI? I know of portal.search.present, but that only accepts a limited number of possible parameters (I now find, by examining the Java code). I need to understand the Lucene query syntax properly before I can use Luke to perform searches.

Thanks for your help,

David.

-----Original Message-----
From: Miller, Christopher C [mailto:ccmiller@anonymised.com]
Sent: 05 March 2010 16:13
To: geonetwork-users Users
Subject: Re: [GeoNetwork-users] approaches to hierarchical ISO19115 metadata

David,

I am certainly willing to talk about how/what we added to the index, but I think you're right that you shouldn't have to. One way to start troubleshooting this would be to determine if it's a CSW or GN+Lucene problem, and I suppose I would start by running Luke to crack open the Lucene index and see if there is indeed a parentId value in there. If it's there, you can try to query against it (assuming you have metadata with a parentId value, which I presume).

I just checked a gn instance I have from svn and it's in there. Querying in Luke works, as does a query using the native GN UI. This exposed a problem I have with CSW, however, so I haven't been able to check the same query using FILTER in CSW (which would probably be the most helpful bit of information I could have offered, of course).

Chris

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C.C. Miller
Assistant Professor of Library Science
Geographic Information Systems Specialist
Purdue University Libraries

http://gis.lib.purdue.edu
feed://www.lib.purdue.edu/gis/rss.php

ccmiller@anonymised.com<mailto:ccmiller@anonymised.com>
765.496.9474
IM=ccmiller@anonymised.com<mailto:IM=ccmiller@anonymised.com>
AIM=cecmcgee
Jabber=ccmiller@anonymised.com<mailto:Jabber=ccmiller@anonymised.com>
Twitter=http://twitter.com/pugolian

2215E EAS (CIVL)
Earth & Atmospheric Sciences Library (EAS)
Civil Engineering Building (CIVL)
550 Stadium Mall Drive
West Lafayette, IN 47907-2051
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^