[GeoNetwork-devel] oai-provider changes schemalocation

Dear all,

we have observed that geonetwork changes the schemaLocation of a metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
http://trac.osgeo.org/geonetwork/ticket/373

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be of
poor coding style, since the schemaLocation is hardcoded in the code.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?
Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
  String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
  String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
  md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
  if (Lib.existsConverter(schemaDir, prefix)) {
     md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
  } else {
    throw new CannotDisseminateFormatException("Unknown prefix : "+ prefix);
  }
}

On 15/03/11 04:07, Timo Proescholdt wrote:

Dear all,

we have observed that geonetwork changes the schemaLocation of a metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
#373 (GeoNetwork changes schemaLocation, potentially invalidating metadata) – GeoNetwork opensource Developer website

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be of
poor coding style, since the schemaLocation is hardcoded in the code.

I suspect the reason it has been hardcoded is that GeoNetwork bundles and comes with the schemas that are shown in that schemaLocation - if you create or edit records then those are the schemas against which your record should be validated - so the intent seems to me to be correct even if the method of doing it is not so elegant (see below for how this might be done in 2.7 onwards). Another reason this might be here is that the older, slightly nicer way of doing this in update-fixed-info.xsl on a per schema basis is not necessarily available as the application of that xslt can be switched off in the system config.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?

I doubt this would cause an issue but see above unless you are thinking about the fact that the same schema may be available from a few different schema locations (which last time I looked was true!).

Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

With the schemaPlugin changes, each schema has its own identity file. Seems to me that this could also contain the values for the schemaLocation(s) that should be used with that schema. The SchemaManager could be enhanced to lookup these values and provide them to whatever mechanism we use to add them to the metadata record.

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
   String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
   String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
   md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
   if (Lib.existsConverter(schemaDir, prefix)) {
      md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
   } else {
     throw new CannotDisseminateFormatException("Unknown prefix : "+ prefix);
   }
}

?Because the resumptionToken fixes weren't in 2.6.3? I don't think they made it in there until after the release of 2.6.3.

Cheers.
Simon

Dear all,

here comes a patch for the problem. (eclipse WS patch)

Semantic-wise, the schemaLocation is now only added to an outgoing OAI
response (ListRecords,GetRecord) if there is no schemaLocation present in
the metadata record in GN.

Syntactic-wise I have removed the code duplication of buildRecord,
introducing a new, slightly modified static buildRecord in the class
GetRecord, which is called from ListRecords.

Introducing a common superclass would also have been possible, but I
discarded it due to the fact that ListRecords has a superclass already,
but GetRecord does not.

hope this is useful
best regards
Timo

Dear all,

we have observed that geonetwork changes the schemaLocation of a metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
http://trac.osgeo.org/geonetwork/ticket/373

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be of
poor coding style, since the schemaLocation is hardcoded in the code.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?
Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
  String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
  String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
  md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
  if (Lib.existsConverter(schemaDir, prefix)) {
     md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
  } else {
    throw new CannotDisseminateFormatException("Unknown prefix : "+
prefix);
  }
}

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

(attachments)

patch.txt (6.24 KB)

I'll take a look at it as I think I have just found another issue to address with this harvester - on some occasions it appears to be sending a resumptionToken when it doesn't have any records to send?

Cheers and thanks,
Simon
________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Thursday, 17 March 2011 12:04 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] oai-provider changes schemalocation

Dear all,

here comes a patch for the problem. (eclipse WS patch)

Semantic-wise, the schemaLocation is now only added to an outgoing OAI
response (ListRecords,GetRecord) if there is no schemaLocation present in
the metadata record in GN.

Syntactic-wise I have removed the code duplication of buildRecord,
introducing a new, slightly modified static buildRecord in the class
GetRecord, which is called from ListRecords.

Introducing a common superclass would also have been possible, but I
discarded it due to the fact that ListRecords has a superclass already,
but GetRecord does not.

hope this is useful
best regards
Timo

Dear all,

we have observed that geonetwork changes the schemaLocation of a metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
#373 (GeoNetwork changes schemaLocation, potentially invalidating metadata) – GeoNetwork opensource Developer website

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be of
poor coding style, since the schemaLocation is hardcoded in the code.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?
Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
  String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
  String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
  md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
  if (Lib.existsConverter(schemaDir, prefix)) {
     md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
  } else {
    throw new CannotDisseminateFormatException("Unknown prefix : "+
prefix);
  }
}

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

Hi Simon,

I have also seen that behaviour, but right now I just cannot reproduce it.
I think I used to see it when I issued a ListIdentifiers against an empty
provider, but this is no longer the case.

Checking the code, especially, listRecordsResponse:toXml ,
listRecordsResponse:toXml show that the token element is only added if !=
null, which tells us that a tokenelements somehow gets generated in
AbsttractTokenListener:98. However, this should only happen when the list
size is greater than the maximum allowed records for a response.

let me know when you manage to reproduce.

best regards
Timo

I'll take a look at it as I think I have just found another issue to
address with this harvester - on some occasions it appears to be sending a
resumptionToken when it doesn't have any records to send?

Cheers and thanks,
Simon
________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Thursday, 17 March 2011 12:04 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] oai-provider changes schemalocation

Dear all,

here comes a patch for the problem. (eclipse WS patch)

Semantic-wise, the schemaLocation is now only added to an outgoing OAI
response (ListRecords,GetRecord) if there is no schemaLocation present in
the metadata record in GN.

Syntactic-wise I have removed the code duplication of buildRecord,
introducing a new, slightly modified static buildRecord in the class
GetRecord, which is called from ListRecords.

Introducing a common superclass would also have been possible, but I
discarded it due to the fact that ListRecords has a superclass already,
but GetRecord does not.

hope this is useful
best regards
Timo

Dear all,

we have observed that geonetwork changes the schemaLocation of a
metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
http://trac.osgeo.org/geonetwork/ticket/373

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is
stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be
of
poor coding style, since the schemaLocation is hardcoded in the code.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?
Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
  String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
  String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
  md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
  if (Lib.existsConverter(schemaDir, prefix)) {
     md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
  } else {
    throw new CannotDisseminateFormatException("Unknown prefix : "+
prefix);
  }
}

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

Hi Timo,

AbstractTokenLister was making decisions about when to add the resumptionToken as though all records returned from the search would be in the ListRecords response. This would have been fine if an external harvester was asking for a prefix (or schema) that all records in GeoNetwork belonged to. This assumption isn't true in many cases.

I think the most straightforward change to AbstractTokenLister is to add a _schema parameter with the prefix as the search value to restrict the search so that it only returns the records requested. This would be in line with the assumptions in AbstractTokenLister. I've tested this in the BlueNetMEST/ANZMEST sandbox with joai doing the harvesting and so far it seems to work well.

Cheers,
Simon
________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Friday, 18 March 2011 4:46 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: geonetwork-devel@lists.sourceforge.net
Subject: RE: [GeoNetwork-devel] oai-provider changes schemalocation

Hi Simon,

I have also seen that behaviour, but right now I just cannot reproduce it.
I think I used to see it when I issued a ListIdentifiers against an empty
provider, but this is no longer the case.

Checking the code, especially, listRecordsResponse:toXml ,
listRecordsResponse:toXml show that the token element is only added if !=
null, which tells us that a tokenelements somehow gets generated in
AbsttractTokenListener:98. However, this should only happen when the list
size is greater than the maximum allowed records for a response.

let me know when you manage to reproduce.

best regards
Timo

I'll take a look at it as I think I have just found another issue to
address with this harvester - on some occasions it appears to be sending a
resumptionToken when it doesn't have any records to send?

Cheers and thanks,
Simon
________________________________________
From: Timo Proescholdt [timo@anonymised.com]
Sent: Thursday, 17 March 2011 12:04 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] oai-provider changes schemalocation

Dear all,

here comes a patch for the problem. (eclipse WS patch)

Semantic-wise, the schemaLocation is now only added to an outgoing OAI
response (ListRecords,GetRecord) if there is no schemaLocation present in
the metadata record in GN.

Syntactic-wise I have removed the code duplication of buildRecord,
introducing a new, slightly modified static buildRecord in the class
GetRecord, which is called from ListRecords.

Introducing a common superclass would also have been possible, but I
discarded it due to the fact that ListRecords has a superclass already,
but GetRecord does not.

hope this is useful
best regards
Timo

Dear all,

we have observed that geonetwork changes the schemaLocation of a
metadata
record when it outputs it via the OAI-PHM provider.
Interestingly, this even happens if there is already a perfectly valid
schemaLocation for the schema in question.

The issue is related to, but was not fixed by
#373 (GeoNetwork changes schemaLocation, potentially invalidating metadata) – GeoNetwork opensource Developer website

I have identified the buildRecord function in GetRecord.java and
ListRecords.java as the culprits.
GN replaces the schemaLocation with a local one, if the schema requested
by the OAI harvester is equal to the schema the metadata record is
stored
in. A transformation is attempted otherwise.
You will fine the relevant code below.

A solution would probably only replace the schemaLocation if there was
none already, along the lines of the fix for ticket 373.

For the forthcoming fix, it would be good if the 100% code duplication
between buildRecord in Getrecord and ListRecords, was addressed and I
would also like to point out that the fix in Show.java:131 seems to be
of
poor coding style, since the schemaLocation is hardcoded in the code.

Questions to the community. Would a fix that only added a schemaLocation
if the metadata record has not one already break something?
Is there a way to get the schemaLocation from a centrally configurable
place rather than hardcoding it?

best regards
Timo

P.S.: I have checked out GN 2.6.3 to attempt a fix but I am getting
complication errors, due to missing geonetworkResumptionToken. Any hints
why that is?

if (prefix.equals(schema)) {
  String schemaUrl = Lib.getSchemaUrl(context, schemaDir);
  String schemaLoc = md.getNamespace().getURI() +" "+ schemaUrl;
  md.setAttribute("schemaLocation", schemaLoc, OaiPmh.Namespaces.XSI);
} else {
  if (Lib.existsConverter(schemaDir, prefix)) {
     md = Lib.transform(schemaDir, md, uuid, changeDate, prefix,
context.getBaseUrl(), dm.getSiteURL(), gc.getSiteName());
  } else {
    throw new CannotDisseminateFormatException("Unknown prefix : "+
prefix);
  }
}

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork