[GeoNetwork-devel] bug with schemaPlugins and schemaLocation? / questions

[Timo_Proescholdt]

Dear all,

it seems that Gn 2.7 trunk still has some issues with the schemaPlugins. I
get nullpointer exceptions when trying to upload a schemaPlugin I created
from the schemaPlugins directory in trunk. (for example the
iso19139.mcp-1.4 or iso19139.mcp). The reason is that there is no
"schemaLocation" in the schemaIdent, but SchemaManager.java:1070 expects
one to be there. (extractSchemaLocation(..)).

Can someone tell if the bug is in GN or the schemaPlugin? Ergo, is a
schemaLocation in schema-ident.xml mandatory and if yes, which value has
to be in there?

I am hesitant to put a custom schemaLocation there, as I do not know the
repercussions.

Simon's great work on the schemaPlugins has finally made it possible to
create a WMO core profile for GN. I would very much appreciate some
comments on the following points I want to implement.

1) different XML schema
It seems I could just replace the schema.xsd and schema/ . But is this
also possible if I create a profile of iso19139, say iso19139.wmocp?
Simon's slides say that parts of the iso19139 schema can be overwritten in
a profile, so I am not sure if replacing the whole schema will work.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing the
sections that insert the uuid as fileIdentifier and make it impossible to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is the
base for e.G the OAI-PMH id).

3) schematron.
In the MCP I see schematron rules, so I assume that support for them is
now there. Is there any documentation on how they work / can be changed /
are shown in the userinterface?

4) custom templates
I would like to provide custom templates for the different WMO datatypes.
While importing templates works ok when I add a file to the normal
iso19139 templates/ directory, this seems not to work with custom
profiles.
For quick testing I copied the whole iso19139 directory, renamed it to
iso19139-wmo, changed the <name> in schema-ident.xml and
present/metadata-iso19139-wmo.xsl (also the "match" xsl values in it) and
tried to import the templates. While the UI did not show any errors and
the log showed that the files were sucessfully loaded, I was not able to
create a new MD record, because no templates were available.

<response status="true">
  <iso19139-wmo>
    <template name="deegree_fragment_tester.xml" status="loaded" />
<template name="geoserver_fragment_tester.xml" status="loaded" />
[..]
  </iso19139-wmo>
</response>

I investigated, and saw that applying "xsl/portal-present.xsl"
Get.java:88, List list = Xml.transform(..) returns no templates, although
some were found in the search above.
Any ideas where the error might be found?

GN would get quite a lot of new users if we can create a profile for WMO
metadata, any help would thus be very much appreciated.

best regards
Timo

[sppigot]

Hi Timo,

On 06/18/2011 04:08 AM, Timo Proescholdt wrote:

Dear all,

it seems that Gn 2.7 trunk still has some issues with the schemaPlugins. I
get nullpointer exceptions when trying to upload a schemaPlugin I created
from the schemaPlugins directory in trunk. (for example the
iso19139.mcp-1.4 or iso19139.mcp). The reason is that there is no
"schemaLocation" in the schemaIdent, but SchemaManager.java:1070 expects
one to be there. (extractSchemaLocation(..)).

Can someone tell if the bug is in GN or the schemaPlugin? Ergo, is a
schemaLocation in schema-ident.xml mandatory and if yes, which value has
to be in there?

Mandatory - should have the official location of the schemas that describe the metadata record. NPE was caused by the lack of validation on the schema-ident.xml file (fixed in svn rev 7862).

I am hesitant to put a custom schemaLocation there, as I do not know the
repercussions.

Repercussions are that the contents of that element will be used whenever a schemaLocation attribute is required to be added to the record. For now that is in:

    * metadata.show - replaces old code with hard coded schemaLocation
      (and only adds it if the schemaLocation attribute doesn't already
      exist in metadata)
    * oaipmh - ListMetadataFormats verb (replaces the old code that was
      cobbling together a link to the schema.xsd in the GeoNetwork
      instance but you can still get that behaviour if you want it by
      putting that link into the schemaLocation element in the
      schema-ident.xml file for your schema)

The plan is to send the schemaLocation through to the updated-fixed-info.xsl file for those that want to fix this attribute in metadata records created and edited in GeoNetwork (easy but not done yet). Also, at present GeoNetwork requires the XSDs to be physically present for editing, it may be that we could read the schemas from the schemaLocation instead of distributing them with GeoNetwork.

Simon's great work on the schemaPlugins has finally made it possible to
create a WMO core profile for GN. I would very much appreciate some
comments on the following points I want to implement.
1) different XML schema
It seems I could just replace the schema.xsd and schema/ . But is this
also possible if I create a profile of iso19139, say iso19139.wmocp?
Simon's slides say that parts of the iso19139 schema can be overwritten in
a profile, so I am not sure if replacing the whole schema will work.

The major work in defining a schema for use in GeoNetwork is the presentation of the elements in XSLT. With XSLT and profiles in mind (particularly iso19139), it seemed more sensible to keep the work for the iso19139 schema presentation in the iso19139 base schema, separate from the additional elements/behaviour required by a profile (although the profile presentation can be processed in the same mode as iso19139 - see mcp for example). So the idea is that you would separate the iso19139.wmocp profile from iso19139 for ease of maintenance reasons as well as giving it an identity that indicates that it isn't iso19139.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing the
sections that insert the uuid as fileIdentifier and make it impossible to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is the
base for e.G the OAI-PMH id).

We've also had this problem come up down here. The suggestion here was that you could move the GeoNetwork uuid to a uuid attribute attached to the gmd:MD_Metadata element. This would need changes to the relevant templates of set-uuid.xsl, extract-uuid.xsl and update-fixed-info.xsl for the schema/profile - which sounds similar to what you've done but won't automatically provide the ability to refer to the record by the fileIdentifier in OAIPMH or any other method.

(On a related note I thought perhaps we could also define a new class to handle uuid generation - default behaviour would be the existing one ie. generate a type 4 random uuid. This behaviour could then be replaced/overridden with other behaviour (eg. we have a requirement to support type 1 uuids) maybe on a per schema basis as part of a plugin. Presumably this could also be implemented to use whatever process is used to generate a unique identifier for gmd:fileIdentifier in your schema?).

3) schematron.
In the MCP I see schematron rules, so I assume that support for them is
now there. Is there any documentation on how they work / can be changed /
are shown in the userinterface?

Not sure on the documentation front - can't see anything by searching the doco on but the plan is to evolve the skeletal slides on schemaPlugins into a section in the developers documentation describing plugin schemas, how they work and what is required - this would include schematrons. For the moment, you could look at the mcp schematrons which are in the schemaPlugins/trunk/schematrons and follow the same path?

4) custom templates
I would like to provide custom templates for the different WMO datatypes.
While importing templates works ok when I add a file to the normal
iso19139 templates/ directory, this seems not to work with custom
profiles.
For quick testing I copied the whole iso19139 directory, renamed it to
iso19139-wmo, changed the<name> in schema-ident.xml and
present/metadata-iso19139-wmo.xsl (also the "match" xsl values in it) and
tried to import the templates. While the UI did not show any errors and
the log showed that the files were sucessfully loaded, I was not able to
create a new MD record, because no templates were available.

<response status="true">
   <iso19139-wmo>
     <template name="deegree_fragment_tester.xml" status="loaded" />
<template name="geoserver_fragment_tester.xml" status="loaded" />
[..]
   </iso19139-wmo>
</response>

I investigated, and saw that applying "xsl/portal-present.xsl"
Get.java:88, List list = Xml.transform(..) returns no templates, although
some were found in the search above.
Any ideas where the error might be found?

I think the templates have probably been assigned to the iso19139 schema rather than iso19139.wmo - probably because they go through the schema determining code before they get added - you'd need to make sure they identify as iso19139.wmo instead of iso19139. I know why it happens (because it's consistent with the path that all metadata passing into GeoNetwork takes) but I think perhaps this should be reported to the user so that we don't violate the 'principle of least astonishment'?

GN would get quite a lot of new users if we can create a profile for WMO
metadata, any help would thus be very much appreciated.

Let's work some more on this to make sure that we can make it happen for 2.8!

Cheers and thanks,
Simon

[Timo_Proescholdt]

Hi Simon,

good to hear from you and thanks. (more inline)

Hi Timo,

On 06/18/2011 04:08 AM, Timo Proescholdt wrote:

Dear all,

it seems that Gn 2.7 trunk still has some issues with the schemaPlugins.
I

[..]

Simon's great work on the schemaPlugins has finally made it possible to
create a WMO core profile for GN. I would very much appreciate some
comments on the following points I want to implement.
1) different XML schema
It seems I could just replace the schema.xsd and schema/ . But is this
also possible if I create a profile of iso19139, say iso19139.wmocp?
Simon's slides say that parts of the iso19139 schema can be overwritten
in
a profile, so I am not sure if replacing the whole schema will work.

The major work in defining a schema for use in GeoNetwork is the
presentation of the elements in XSLT. With XSLT and profiles in mind
(particularly iso19139), it seemed more sensible to keep the work for
the iso19139 schema presentation in the iso19139 base schema, separate
from the additional elements/behaviour required by a profile (although
the profile presentation can be processed in the same mode as iso19139 -
see mcp for example). So the idea is that you would separate the
iso19139.wmocp profile from iso19139 for ease of maintenance reasons as
well as giving it an identity that indicates that it isn't iso19139.

To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with only the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is mandatory, would I just "overwrite" the declaration and include the base xml schema with an xml include?

Our iso19139 is basically plain iso19139, except that we have concluded that the gml/3.2 namespace is what is really required of the iso19139 specs (iso19139 requires iso19136, which is GML 3.2.1 having http://www.opengis.net/gml/3.2 as namespace). This requires new schema files and not only remapping of the namespace, since the namespace is also coded into the schema files. However, the changes are probably limited to the gml namespace, so we could use the base iso19139 for the rest.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing
the
sections that insert the uuid as fileIdentifier and make it impossible
to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is
the
base for e.G the OAI-PMH id).

We've also had this problem come up down here. The suggestion here was
that you could move the GeoNetwork uuid to a uuid attribute attached to
the gmd:MD_Metadata element. This would need changes to the relevant
templates of set-uuid.xsl, extract-uuid.xsl and update-fixed-info.xsl
for the schema/profile - which sounds similar to what you've done but
won't automatically provide the ability to refer to the record by the
fileIdentifier in OAIPMH or any other method.

I would support making the uuid a bit more flexible. Actually it is not clear to me (probably my ignorance) why the uuid is needed in the editor at all. The id seems sufficient to refer to the right MD record when interacting with the server.
For WMO/WIS purposes we would need a fileIdentifer flexible enough to be set by the user, since it can be pretty arbitrary, but you would probably check it for the right format with schematron.

4) custom templates
I would like to provide custom templates for the different WMO
datatypes.

[..]

I investigated, and saw that applying "xsl/portal-present.xsl"
Get.java:88, List list = Xml.transform(..) returns no templates,
although
some were found in the search above.
Any ideas where the error might be found?

I think the templates have probably been assigned to the iso19139 schema
rather than iso19139.wmo - probably because they go through the schema

nope.. my templates get assigned the "iso19139-wmo" schema. The problem should thus be in my custom presentation transformation (I copied the plain iso19139 one). I found indeed some potential problems. (I did not rename some "match" clauses, which still point to the plain iso19139 values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the templates and the xsd interact and the conventions that are assumed by GN. I have seen that there are at least two ways of referring to the schema in the "presentation" XSLTs. First, there is the parameter "mode" which is passed to the XSL templates, and which is "iso19139" even if it is a profile of iso19139. Second, there is the "match" and the names of the templates which seem to need to match the of the profile + something else, by convention?

I tried to apply the "portal-present.xsl" manually (oxygenxml with saxon 9.4EE and others) to the string I captured debugging with eclipse right before the XSL.Transform(..) in "Get.java:88", once for my iso19139-wmo templates and once for the templates coming with the included iso19139. I hoped that this would help me debugging the XSL, but I get the same result consisting only of the "<summary>..</summary>" with no "<metadata>" elements no matter if I the profile is "iso19139" or "iso19139-wmo". I am afraid that this is due to some problem in ogxygenXML, since in GN I get "<metadata>" elements when applying the transformation for plain "iso19139". (if the profile is "iso19139-wmo" I dont get the "<metadata>" elements, which I think is the cause of my not seeing any "iso19139-wmo" templates.)

I will keep debugging and trying to deepen my udnerstanding, but would appreciate any hint, especially on how the "mode", "match" and the XML schema work together in the presentation template.

best regards and sorry for the long mail
Timo

Let's work some more on this to make sure that we can make it happen for
2.8!

Cheers and thanks,
Simon

[sppigot]

On 06/20/2011 06:18 AM, Timo Proescholdt wrote:

<snip>
To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the
structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with only
the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is
mandatory, would I just "overwrite" the declaration and include the base
xml schema with an xml include?

Ok, so maybe some comments on how you can extend and restrict xml schemas for use in GeoNetwork.

Extensions: You can add elements to the base schema type using the XML schema 'extension' mechanism (see for example http://geonetwork.svn.sourceforge.net/viewvc/geonetwork/schemaPlugins/trunk/iso19139.mcp/schema/extensions/mcpExtensions.xsd).

Restrictions: If you want to implement a restriction (such as your example above) on an element in a base schema type then you should define and implement a schematron rule that enforces your restriction - see https://www.seegrid.csiro.au/wiki/AppSchemas/MetadataProfiles#Restrictions

Our iso19139 is basically plain iso19139, except that we have concluded
that the gml/3.2 namespace is what is really required of the iso19139
specs (iso19139 requires iso19136, which is GML 3.2.1 having
http://www.opengis.net/gml/3.2 as namespace). This requires new schema
files and not only remapping of the namespace, since the namespace is
also coded into the schema files. However, the changes are probably
limited to the gml namespace, so we could use the base iso19139 for the
rest.

So you could define a new profile with the new* schema files and your restrictions/constraints implemented as schematrons. In the presentation xslt, you could process all your metadata elements in iso19139 mode (and thus take advantage of the work already done on the base iso19139 schema) and add additional templates to handle elements in the new gml namespace.

* - by which I think you mean one of the sets of iso19139 schema files that are more recent than the ones we are currently using in the GeoNetwork iso19139 schema?

<snip>

I think the templates have probably been assigned to the iso19139 schema
rather than iso19139.wmo - probably because they go through the schema

nope.. my templates get assigned the "iso19139-wmo" schema. The problem
should thus be in my custom presentation transformation (I copied the
plain iso19139 one). I found indeed some potential problems. (I did not
rename some "match" clauses, which still point to the plain iso19139
values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the
templates and the xsd interact and the conventions that are assumed by
GN. I have seen that there are at least two ways of referring to the
schema in the "presentation" XSLTs. First, there is the parameter "mode"
which is passed to the XSL templates, and which is "iso19139" even if it
is a profile of iso19139. Second, there is the "match" and the names of
the templates which seem to need to match the of the profile + something
else, by convention?

Rather than put it down here in an ever expanding and probably difficult to follow email, I'll make it a priority this week to get the documentation on schemaPlugins (including the way in which the xslt processing works for profiles) into the GeoNetwork developers manual.

Cheers,
Simon

[Timo_Proescholdt]

Dear all,

I have made some progress with the WMO core profile schemaPlugin and I am
at a point where I now have a working core profile plugin that can be
loaded into GN.

Contentwise and currently, I am wrestling with the codelists. We have some
elements, e.G.:
/MD_Metadata/identificationInfo/MD_DataIdentification/resourceConstraints/MD_LegalConstraints/useLimitation,
that we would like to be populated using a dropdown and a fixed vocabulary
(codelist?), rather than a free text.
I can match the element in my "metadata-iso19139.wmocp.xsl", but I am not
sure what to do then. I can of course generate my own dropdown in the xsl,
but I'm not sure if this is the way to go.
Is there a more generic way of turning an element into a dropdown whose
values come from a codelist?

Also, is there a recommended way of disabling the editing of certain fields?

best regards and thanks
Timo

On 06/20/2011 06:18 AM, Timo Proescholdt wrote:

<snip>
To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the
structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with only
the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is
mandatory, would I just "overwrite" the declaration and include the base
xml schema with an xml include?

Ok, so maybe some comments on how you can extend and restrict xml
schemas for use in GeoNetwork.

Extensions: You can add elements to the base schema type using the XML
schema 'extension' mechanism (see for example
http://geonetwork.svn.sourceforge.net/viewvc/geonetwork/schemaPlugins/trunk/iso19139.mcp/schema/extensions/mcpExtensions.xsd).

Restrictions: If you want to implement a restriction (such as your
example above) on an element in a base schema type then you should
define and implement a schematron rule that enforces your restriction -
see
https://www.seegrid.csiro.au/wiki/AppSchemas/MetadataProfiles#Restrictions

Our iso19139 is basically plain iso19139, except that we have concluded
that the gml/3.2 namespace is what is really required of the iso19139
specs (iso19139 requires iso19136, which is GML 3.2.1 having
http://www.opengis.net/gml/3.2 as namespace). This requires new schema
files and not only remapping of the namespace, since the namespace is
also coded into the schema files. However, the changes are probably
limited to the gml namespace, so we could use the base iso19139 for the
rest.

So you could define a new profile with the new* schema files and your
restrictions/constraints implemented as schematrons. In the presentation
xslt, you could process all your metadata elements in iso19139 mode (and
thus take advantage of the work already done on the base iso19139
schema) and add additional templates to handle elements in the new gml
namespace.

* - by which I think you mean one of the sets of iso19139 schema files
that are more recent than the ones we are currently using in the
GeoNetwork iso19139 schema?

<snip>

I think the templates have probably been assigned to the iso19139
schema
rather than iso19139.wmo - probably because they go through the schema

nope.. my templates get assigned the "iso19139-wmo" schema. The problem
should thus be in my custom presentation transformation (I copied the
plain iso19139 one). I found indeed some potential problems. (I did not
rename some "match" clauses, which still point to the plain iso19139
values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the
templates and the xsd interact and the conventions that are assumed by
GN. I have seen that there are at least two ways of referring to the
schema in the "presentation" XSLTs. First, there is the parameter "mode"
which is passed to the XSL templates, and which is "iso19139" even if it
is a profile of iso19139. Second, there is the "match" and the names of
the templates which seem to need to match the of the profile + something
else, by convention?

Rather than put it down here in an ever expanding and probably difficult
to follow email, I'll make it a priority this week to get the
documentation on schemaPlugins (including the way in which the xslt
processing works for profiles) into the GeoNetwork developers manual.

Cheers,
Simon

[Francois_Prunayre]

Hi Timo,

2011/7/5 Timo Proescholdt <timo@anonymised.com>:

Dear all,

I have made some progress with the WMO core profile schemaPlugin and I am
at a point where I now have a working core profile plugin that can be
loaded into GN.

Contentwise and currently, I am wrestling with the codelists. We have some
elements, e.G.:
/MD_Metadata/identificationInfo/MD_DataIdentification/resourceConstraints/MD_LegalConstraints/useLimitation,
that we would like to be populated using a dropdown and a fixed vocabulary
(codelist?), rather than a free text.

At least, 2 options here I think :
a) replace the input by a dropdown. A good example for this is the
gmd:protocol list in ISO19139. The main issue of that approach is if
you get a record (by harvesting or import) which has a value not
available in your restricted list, then when opening the editor for
that record, this value will be probably lost.

b) use the suggestion mechanism which provide a dropdown next to the
input field (eg. scale denominator). For that search for helper in
metadata.xsl

HTH.

Francois

I can match the element in my "metadata-iso19139.wmocp.xsl", but I am not
sure what to do then. I can of course generate my own dropdown in the xsl,
but I'm not sure if this is the way to go.
Is there a more generic way of turning an element into a dropdown whose
values come from a codelist?

Also, is there a recommended way of disabling the editing of certain fields?

best regards and thanks
Timo

On 06/20/2011 06:18 AM, Timo Proescholdt wrote:

<snip>
To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the
structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with only
the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is
mandatory, would I just "overwrite" the declaration and include the base
xml schema with an xml include?

Ok, so maybe some comments on how you can extend and restrict xml
schemas for use in GeoNetwork.

Extensions: You can add elements to the base schema type using the XML
schema 'extension' mechanism (see for example
http://geonetwork.svn.sourceforge.net/viewvc/geonetwork/schemaPlugins/trunk/iso19139.mcp/schema/extensions/mcpExtensions.xsd).

Restrictions: If you want to implement a restriction (such as your
example above) on an element in a base schema type then you should
define and implement a schematron rule that enforces your restriction -
see
https://www.seegrid.csiro.au/wiki/AppSchemas/MetadataProfiles#Restrictions

Our iso19139 is basically plain iso19139, except that we have concluded
that the gml/3.2 namespace is what is really required of the iso19139
specs (iso19139 requires iso19136, which is GML 3.2.1 having
http://www.opengis.net/gml/3.2 as namespace). This requires new schema
files and not only remapping of the namespace, since the namespace is
also coded into the schema files. However, the changes are probably
limited to the gml namespace, so we could use the base iso19139 for the
rest.

So you could define a new profile with the new* schema files and your
restrictions/constraints implemented as schematrons. In the presentation
xslt, you could process all your metadata elements in iso19139 mode (and
thus take advantage of the work already done on the base iso19139
schema) and add additional templates to handle elements in the new gml
namespace.

* - by which I think you mean one of the sets of iso19139 schema files
that are more recent than the ones we are currently using in the
GeoNetwork iso19139 schema?

<snip>

I think the templates have probably been assigned to the iso19139
schema
rather than iso19139.wmo - probably because they go through the schema

nope.. my templates get assigned the "iso19139-wmo" schema. The problem
should thus be in my custom presentation transformation (I copied the
plain iso19139 one). I found indeed some potential problems. (I did not
rename some "match" clauses, which still point to the plain iso19139
values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the
templates and the xsd interact and the conventions that are assumed by
GN. I have seen that there are at least two ways of referring to the
schema in the "presentation" XSLTs. First, there is the parameter "mode"
which is passed to the XSL templates, and which is "iso19139" even if it
is a profile of iso19139. Second, there is the "match" and the names of
the templates which seem to need to match the of the profile + something
else, by convention?

Rather than put it down here in an ever expanding and probably difficult
to follow email, I'll make it a priority this week to get the
documentation on schemaPlugins (including the way in which the xslt
processing works for profiles) into the GeoNetwork developers manual.

Cheers,
Simon

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

[Timo_Proescholdt]

Hi Francois,

thanks for your email. I have looked at your suggestions, but have some
further questions.
It seems that GN automatically generates dropdowns for elements associated
with codelists when they are reflected as sequence(?) in the XML schema.
This seems to be the case for gmd:presentationForm for example, which has
no special handling in metadata-iso19139.xsl but has a dropdown in the
editor. (my xml is: <gmd:CI_PresentationFormCode
codeListValue="mapDigital"
codeList="http://www.isotc211.org/2005/resources/codeList.xml#CI_PresentationFormCode&quot;/&gt;\)

Does this mean that if I extend/restrict the XML schema so that my
element's children are a sequence, would GN generate the dropdown
automatically? Since the codelist would be separate/external, how would it
be "registered". Does it work via the oasis registry?

I think I lack a clear understanding how codelists and the editor interact.

b.t.w: I am happy to provide some sort of documentation on all thus, as
this will be valuable for people wanting to further improve the core
profile, but I need a better understanding first, I feel.

best regards
Timo

Hi Timo,

2011/7/5 Timo Proescholdt <timo@anonymised.com>:

Dear all,

I have made some progress with the WMO core profile schemaPlugin and I
am
at a point where I now have a working core profile plugin that can be
loaded into GN.

Contentwise and currently, I am wrestling with the codelists. We have
some
elements, e.G.:
/MD_Metadata/identificationInfo/MD_DataIdentification/resourceConstraints/MD_LegalConstraints/useLimitation,
that we would like to be populated using a dropdown and a fixed
vocabulary
(codelist?), rather than a free text.

At least, 2 options here I think :
a) replace the input by a dropdown. A good example for this is the
gmd:protocol list in ISO19139. The main issue of that approach is if
you get a record (by harvesting or import) which has a value not
available in your restricted list, then when opening the editor for
that record, this value will be probably lost.

b) use the suggestion mechanism which provide a dropdown next to the
input field (eg. scale denominator). For that search for helper in
metadata.xsl

HTH.

Francois

I can match the element in my "metadata-iso19139.wmocp.xsl", but I am
not
sure what to do then. I can of course generate my own dropdown in the
xsl,
but I'm not sure if this is the way to go.
Is there a more generic way of turning an element into a dropdown whose
values come from a codelist?

Also, is there a recommended way of disabling the editing of certain
fields?

best regards and thanks
Timo

On 06/20/2011 06:18 AM, Timo Proescholdt wrote:

<snip>
To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the
structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with
only
the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is
mandatory, would I just "overwrite" the declaration and include the
base
xml schema with an xml include?

Ok, so maybe some comments on how you can extend and restrict xml
schemas for use in GeoNetwork.

Extensions: You can add elements to the base schema type using the XML
schema 'extension' mechanism (see for example
http://geonetwork.svn.sourceforge.net/viewvc/geonetwork/schemaPlugins/trunk/iso19139.mcp/schema/extensions/mcpExtensions.xsd).

Restrictions: If you want to implement a restriction (such as your
example above) on an element in a base schema type then you should
define and implement a schematron rule that enforces your restriction -
see
https://www.seegrid.csiro.au/wiki/AppSchemas/MetadataProfiles#Restrictions

Our iso19139 is basically plain iso19139, except that we have
concluded
that the gml/3.2 namespace is what is really required of the iso19139
specs (iso19139 requires iso19136, which is GML 3.2.1 having
http://www.opengis.net/gml/3.2 as namespace). This requires new
schema
files and not only remapping of the namespace, since the namespace is
also coded into the schema files. However, the changes are probably
limited to the gml namespace, so we could use the base iso19139 for
the
rest.

So you could define a new profile with the new* schema files and your
restrictions/constraints implemented as schematrons. In the
presentation
xslt, you could process all your metadata elements in iso19139 mode
(and
thus take advantage of the work already done on the base iso19139
schema) and add additional templates to handle elements in the new gml
namespace.

* - by which I think you mean one of the sets of iso19139 schema files
that are more recent than the ones we are currently using in the
GeoNetwork iso19139 schema?

<snip>

I think the templates have probably been assigned to the iso19139
schema
rather than iso19139.wmo - probably because they go through the
schema

nope.. my templates get assigned the "iso19139-wmo" schema. The
problem
should thus be in my custom presentation transformation (I copied the
plain iso19139 one). I found indeed some potential problems. (I did
not
rename some "match" clauses, which still point to the plain iso19139
values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the
templates and the xsd interact and the conventions that are assumed by
GN. I have seen that there are at least two ways of referring to the
schema in the "presentation" XSLTs. First, there is the parameter
"mode"
which is passed to the XSL templates, and which is "iso19139" even if
it
is a profile of iso19139. Second, there is the "match" and the names
of
the templates which seem to need to match the of the profile +
something
else, by convention?

Rather than put it down here in an ever expanding and probably
difficult
to follow email, I'll make it a priority this week to get the
documentation on schemaPlugins (including the way in which the xslt
processing works for profiles) into the GeoNetwork developers manual.

Cheers,
Simon

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously
valuable.
Why? It contains a definitive record of application performance,
security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

[sppigot]

Hi Timo,

I'll chip in on this with Francois as I'm writing the doco for all this now...(see below)

For the ISO schemas and profiles: Codelists like the one you're referring to come from the loc/<language_code>/codelists.xml in the GeoNetwork schema directory (that file is built from the codelists provided with the ISO19139 schemas as referred to in the codeList attribute of these elements ie. http://www.isotc211.org/2005/resources/codeList.xml).

The loc/<language_code>/codelists.xml file is made available to the presentation xslts (eg. metadata-iso19139.xsl) on an XPath eg. /root/gui/schemas/iso19139/codelist for iso19139. The handling of these codelists (ie. building select list/drop downs for editing) starts with a template in the presentation xslt that matches any element that has a codeList attribute on one of its children (see the template that does this just before the template named iso19139Codelist in metadata-iso19139.xsl). This covers codelists for elements in the ISO schema that have type gco:CodeListValue_Type.

The other way to supply automatic codeList behaviour is to do what is done with the ISO topicCategory (see http://www.isotc211.org/2005/gmd/identification.xsd). That is define a type for an element that is a restriction of the XSD string and enumerates the values that the element can have. GeoNetwork will read those from the schema and make them available to the presentation xslts which will build a select list/drop down menu (see template getElementText for example in web/geonetwork/xsl/metadata.xsl). These values can also be localized by putting the values into loc/<language_code>/codelists.xml.

Finally, if you're defining a profile of ISO and you use the naming convention for your profile schema in GeoNetwork eg. iso19139.wmocp then you need only define the codelists etc you need for your profile, GeoNetwork should pull the base iso19139 ones out of the iso19139 schema so no need to duplicate all those in your profile.

BTW As promised I have begun and progressed the schema plugins documentation as part of the Developer Manual in 2.7 - some of this description (including an example of Francois' helper/suggestion function) is already there and/or will be enhanced by this discussion shortly so feel free to check it out and give feedback/contribute more.

Cheers and thanks for stimulating this discussion - hopefully we'll be able to get it into a more permanent form in the doco,
Simon
________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Friday, 8 July 2011 12:29 AM
To: Francois Prunayre
Cc: Pigot, Simon (CMAR, Hobart); geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] bug with schemaPlugins and schemaLocation? / questions

Hi Francois,

thanks for your email. I have looked at your suggestions, but have some
further questions.
It seems that GN automatically generates dropdowns for elements associated
with codelists when they are reflected as sequence(?) in the XML schema.
This seems to be the case for gmd:presentationForm for example, which has
no special handling in metadata-iso19139.xsl but has a dropdown in the
editor. (my xml is: <gmd:CI_PresentationFormCode
codeListValue="mapDigital"
codeList="http://www.isotc211.org/2005/resources/codeList.xml#CI_PresentationFormCode&quot;/&gt;\)

Does this mean that if I extend/restrict the XML schema so that my
element's children are a sequence, would GN generate the dropdown
automatically? Since the codelist would be separate/external, how would it
be "registered". Does it work via the oasis registry?

I think I lack a clear understanding how codelists and the editor interact.

b.t.w: I am happy to provide some sort of documentation on all thus, as
this will be valuable for people wanting to further improve the core
profile, but I need a better understanding first, I feel.

best regards
Timo

Hi Timo,

2011/7/5 Timo Proescholdt <timo@anonymised.com>:

Dear all,

I have made some progress with the WMO core profile schemaPlugin and I
am
at a point where I now have a working core profile plugin that can be
loaded into GN.

Contentwise and currently, I am wrestling with the codelists. We have
some
elements, e.G.:
/MD_Metadata/identificationInfo/MD_DataIdentification/resourceConstraints/MD_LegalConstraints/useLimitation,
that we would like to be populated using a dropdown and a fixed
vocabulary
(codelist?), rather than a free text.

At least, 2 options here I think :
a) replace the input by a dropdown. A good example for this is the
gmd:protocol list in ISO19139. The main issue of that approach is if
you get a record (by harvesting or import) which has a value not
available in your restricted list, then when opening the editor for
that record, this value will be probably lost.

b) use the suggestion mechanism which provide a dropdown next to the
input field (eg. scale denominator). For that search for helper in
metadata.xsl

HTH.

Francois

I can match the element in my "metadata-iso19139.wmocp.xsl", but I am
not
sure what to do then. I can of course generate my own dropdown in the
xsl,
but I'm not sure if this is the way to go.
Is there a more generic way of turning an element into a dropdown whose
values come from a codelist?

Also, is there a recommended way of disabling the editing of certain
fields?

best regards and thanks
Timo

On 06/20/2011 06:18 AM, Timo Proescholdt wrote:

<snip>
To clarify my understanding.
a) The GN editor uses the xml schema of a profile to know about the
structure of the metadata.
b) If a schema is iso19139 based, the base schema can be used, with
only
the additional/changed elements needing declaration in the profile.
How does this work on the xml layer? If in my profile an element is
mandatory, would I just "overwrite" the declaration and include the
base
xml schema with an xml include?

Ok, so maybe some comments on how you can extend and restrict xml
schemas for use in GeoNetwork.

Extensions: You can add elements to the base schema type using the XML
schema 'extension' mechanism (see for example
GeoNetwork - Geographic Metadata Catalog download | SourceForge.net).

Restrictions: If you want to implement a restriction (such as your
example above) on an element in a base schema type then you should
define and implement a schematron rule that enforces your restriction -
see
https://www.seegrid.csiro.au/wiki/AppSchemas/MetadataProfiles#Restrictions

Our iso19139 is basically plain iso19139, except that we have
concluded
that the gml/3.2 namespace is what is really required of the iso19139
specs (iso19139 requires iso19136, which is GML 3.2.1 having
http://www.opengis.net/gml/3.2 as namespace). This requires new
schema
files and not only remapping of the namespace, since the namespace is
also coded into the schema files. However, the changes are probably
limited to the gml namespace, so we could use the base iso19139 for
the
rest.

So you could define a new profile with the new* schema files and your
restrictions/constraints implemented as schematrons. In the
presentation
xslt, you could process all your metadata elements in iso19139 mode
(and
thus take advantage of the work already done on the base iso19139
schema) and add additional templates to handle elements in the new gml
namespace.

* - by which I think you mean one of the sets of iso19139 schema files
that are more recent than the ones we are currently using in the
GeoNetwork iso19139 schema?

<snip>

I think the templates have probably been assigned to the iso19139
schema
rather than iso19139.wmo - probably because they go through the
schema

nope.. my templates get assigned the "iso19139-wmo" schema. The
problem
should thus be in my custom presentation transformation (I copied the
plain iso19139 one). I found indeed some potential problems. (I did
not
rename some "match" clauses, which still point to the plain iso19139
values. I probably have to add "-wmo" to them).

I lack full understanding of the way "mode", "match", the names of the
templates and the xsd interact and the conventions that are assumed by
GN. I have seen that there are at least two ways of referring to the
schema in the "presentation" XSLTs. First, there is the parameter
"mode"
which is passed to the XSL templates, and which is "iso19139" even if
it
is a profile of iso19139. Second, there is the "match" and the names
of
the templates which seem to need to match the of the profile +
something
else, by convention?

Rather than put it down here in an ever expanding and probably
difficult
to follow email, I'll make it a priority this week to get the
documentation on schemaPlugins (including the way in which the xslt
processing works for profiles) into the GeoNetwork developers manual.

Cheers,
Simon

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously
valuable.
Why? It contains a definitive record of application performance,
security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

[Timo_Proescholdt]

Hi Simon,

this picks up on the uuid point again.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing
the
sections that insert the uuid as fileIdentifier and make it impossible
to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is
the
base for e.G the OAI-PMH id).

We've also had this problem come up down here. The suggestion here was
that you could move the GeoNetwork uuid to a uuid attribute attached to
the gmd:MD_Metadata element. This would need changes to the relevant
templates of set-uuid.xsl, extract-uuid.xsl and update-fixed-info.xsl
for the schema/profile - which sounds similar to what you've done but
won't automatically provide the ability to refer to the record by the
fileIdentifier in OAIPMH or any other method.

(On a related note I thought perhaps we could also define a new class to
handle uuid generation - default behaviour would be the existing one ie.
generate a type 4 random uuid. This behaviour could then be
replaced/overridden with other behaviour (eg. we have a requirement to
support type 1 uuids) maybe on a per schema basis as part of a plugin.
Presumably this could also be implemented to use whatever process is
used to generate a unique identifier for gmd:fileIdentifier in your
schema?).

It would be good if the uuid generation could be handled by a xslt, but
even this would not solve the problem of a user supplied one, if GN needs
the uuid at creation time.

GN creates the uuid and saves the metadata record directly upon creation,
so there is no time for the user to interact. It seems insensible to
change the uuid, e.G if a different fileIdentifier is input, since this
contradicts the maxim of a uuid.

We at WMO need the gmd:fileIdentifier to be set manually, which I have
managed creating a schema profile. However, we have this requirement that
the OAI identifier correspond to the fileIdentifier, for ease of
debugging.
There is a bit of a catch22 here, because GN creates the uuid before the
user can even think of editing. At the same time, the uuid is the base of
the identifier in OAI, with little prospect of tuning the OAI so that it
work on the base of the gmd:fileIdentifier encoded in the XML only.
(cannot look up the fileIdentifier in the DB)

It seems that it boils down to delaying the creation/storage of the uuid
to the point in time when the user (first) saves the record.
Would it be perceivable to have a mechanism that, if a certain element
(say metadata_uuid) is part of the request, updates the uuid in the DB?
Maybe in conjunction with your idea of allowing different uuid generation
schemata?

what do you/the others think?
best regards
Timo

[andrew_walsh]

Hi Timo,

As long as the new uuid's that you wish to replace are proper
uuid's i.e type 4 I think
[see http://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_.28random.29 ]
Then I guess this is OK. However if those uuid are in any way
non-random and follow some system then its probably not OK.

Regards,

Andrew

----- Original Message ----- From: "Timo Proescholdt" <timo@anonymised.com>
To: <simon.pigot@anonymised.com>
Cc: <geonetwork-devel@lists.sourceforge.net>
Sent: Tuesday, July 12, 2011 8:26 PM
Subject: Re: [GeoNetwork-devel] bug with schemaPlugins and schemaLocation? / questions

Hi Simon,

this picks up on the uuid point again.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing
the
sections that insert the uuid as fileIdentifier and make it impossible
to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is
the
base for e.G the OAI-PMH id).

We've also had this problem come up down here. The suggestion here was
that you could move the GeoNetwork uuid to a uuid attribute attached to
the gmd:MD_Metadata element. This would need changes to the relevant
templates of set-uuid.xsl, extract-uuid.xsl and update-fixed-info.xsl
for the schema/profile - which sounds similar to what you've done but
won't automatically provide the ability to refer to the record by the
fileIdentifier in OAIPMH or any other method.

(On a related note I thought perhaps we could also define a new class to
handle uuid generation - default behaviour would be the existing one ie.
generate a type 4 random uuid. This behaviour could then be
replaced/overridden with other behaviour (eg. we have a requirement to
support type 1 uuids) maybe on a per schema basis as part of a plugin.
Presumably this could also be implemented to use whatever process is
used to generate a unique identifier for gmd:fileIdentifier in your
schema?).

It would be good if the uuid generation could be handled by a xslt, but
even this would not solve the problem of a user supplied one, if GN needs
the uuid at creation time.

GN creates the uuid and saves the metadata record directly upon creation,
so there is no time for the user to interact. It seems insensible to
change the uuid, e.G if a different fileIdentifier is input, since this
contradicts the maxim of a uuid.

We at WMO need the gmd:fileIdentifier to be set manually, which I have
managed creating a schema profile. However, we have this requirement that
the OAI identifier correspond to the fileIdentifier, for ease of
debugging.
There is a bit of a catch22 here, because GN creates the uuid before the
user can even think of editing. At the same time, the uuid is the base of
the identifier in OAI, with little prospect of tuning the OAI so that it
work on the base of the gmd:fileIdentifier encoded in the XML only.
(cannot look up the fileIdentifier in the DB)

It seems that it boils down to delaying the creation/storage of the uuid
to the point in time when the user (first) saves the record.
Would it be perceivable to have a mechanism that, if a certain element
(say metadata_uuid) is part of the request, updates the uuid in the DB?
Maybe in conjunction with your idea of allowing different uuid generation
schemata?

what do you/the others think?
best regards
Timo

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

[sppigot]

Hi Timo,

Maybe this reduces down to offering the user a dialog that has uuid creation options for the 'create' and 'new metadata' functions *before* the existing record/template is cloned. For example: select: uuid (in whatever schemata is currently the default eg. either type 1 date/time based or type 4 random) or user specified. Seems to me that it would be a bit better to do this rather than let the user enter a fileIdentifier (presumably they can edit the fileIdentifer field only if it is empty/missing?) eg. we can reject non-unique uuids etc before the existing record/template is cloned.

Cheers,
Simon

________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Tuesday, 12 July 2011 8:26 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] bug with schemaPlugins and schemaLocation? / questions

Hi Simon,

this picks up on the uuid point again.

2) make fileIdentifier editable
If I modify update-fixed-info.xsl and metadata-iso19139.xsl, removing
the
sections that insert the uuid as fileIdentifier and make it impossible
to
edit it respectively, I can modify the fileIdentifier, using the normal
editor. Is there any problem with this approach I might have overseen?
Unfortunately there seems to be no way to also edit the uuid (which is
the
base for e.G the OAI-PMH id).

We've also had this problem come up down here. The suggestion here was
that you could move the GeoNetwork uuid to a uuid attribute attached to
the gmd:MD_Metadata element. This would need changes to the relevant
templates of set-uuid.xsl, extract-uuid.xsl and update-fixed-info.xsl
for the schema/profile - which sounds similar to what you've done but
won't automatically provide the ability to refer to the record by the
fileIdentifier in OAIPMH or any other method.

(On a related note I thought perhaps we could also define a new class to
handle uuid generation - default behaviour would be the existing one ie.
generate a type 4 random uuid. This behaviour could then be
replaced/overridden with other behaviour (eg. we have a requirement to
support type 1 uuids) maybe on a per schema basis as part of a plugin.
Presumably this could also be implemented to use whatever process is
used to generate a unique identifier for gmd:fileIdentifier in your
schema?).

It would be good if the uuid generation could be handled by a xslt, but
even this would not solve the problem of a user supplied one, if GN needs
the uuid at creation time.

GN creates the uuid and saves the metadata record directly upon creation,
so there is no time for the user to interact. It seems insensible to
change the uuid, e.G if a different fileIdentifier is input, since this
contradicts the maxim of a uuid.

We at WMO need the gmd:fileIdentifier to be set manually, which I have
managed creating a schema profile. However, we have this requirement that
the OAI identifier correspond to the fileIdentifier, for ease of
debugging.
There is a bit of a catch22 here, because GN creates the uuid before the
user can even think of editing. At the same time, the uuid is the base of
the identifier in OAI, with little prospect of tuning the OAI so that it
work on the base of the gmd:fileIdentifier encoded in the XML only.
(cannot look up the fileIdentifier in the DB)

It seems that it boils down to delaying the creation/storage of the uuid
to the point in time when the user (first) saves the record.
Would it be perceivable to have a mechanism that, if a certain element
(say metadata_uuid) is part of the request, updates the uuid in the DB?
Maybe in conjunction with your idea of allowing different uuid generation
schemata?

what do you/the others think?
best regards
Timo