[GeoNetwork-devel] Question on regarding dateStamp in iso 19139

Hi Andrew et al,

The definition of dateStamp is the "date the metadata was created". Yu
*must* not change that definition. ISO 19115 is *not* concerned with
implementation. Hence using the logic of "how else do you know the
change date" or "everyone needs that change date" are not good reasons
to change the definition of dateStamp. If you need a change date for the
metadata then you must implement it in the back end of the database, as
suggested by Andrew, or create a profile which contains a new element
for the metadata change date. You *must not* change the definition of
and existing ISO 19115 metadata element.

I'm extremely surprised that the people that I worked with to develop
ISO TC 211 standards allowed this misuse of dateStamp to be implemented
in national profiles.

John Hockaday

On Wed, 2012-10-31 at 18:06 +1100, andrew walsh wrote:

Hello Heikki, Ian, Simon, John, Annina and All,

Yes, Heikki to confirm I was talking about the 'Geonetwork' protocol harvester.

I had always thought the Geonetwork had used the database 'changeDate' in the
metadata table as a means to decide to harvest or not. This seemed logical to
me.

My reasoning for this thought is the BlueNet MEST version of Geonetwork
we have been using since August 2010 DOES NOT update the gmd:dateStamp when
records are edited/saved.

Looking at the update-fixed-info.xsl from the Bluenet MEST version 1.4.9
(14/10/2011) of GN
for the ISO19139-mcp schema it says:

<xsl:template match="gmd:dateStamp">
  <xsl:choose>
   <xsl:when test="/root/env/updateDateStamp='yes'">
    <gmd:dateStamp>
     <gco:DateTime><xsl:value-of select="/root/env/changeDate"/></gco:DateTime>
    </gmd:dateStamp>
   </xsl:when>
   <xsl:otherwise>
    <xsl:copy-of select="."/>
   </xsl:otherwise>
  </xsl:choose>
</xsl:template>

That is, only update the gmd:DateStamp if ="/root/env/updateDateStamp='yes'
which I never found occurring. How would the /root/env/updateDateStamp be set to
yes?
Simon, maybe you could help here.

Given that I never observed the gmd:DateStamp updating it never occurred to me
that this
would used by the harvester as check whether to harvest a record. So I assumed
the
harvester was looking rather at the database changeDate to decide whether to
harvest or not.

Whereas the core (International version) of Geonetwork from
v2.6.3 up until the current V2.8.0RC1 DOES update the gmd:dateStamp on edit/save
as recently noted by Ian Allen (thanks! Ian).

Looking at update-fixed-info.xsl in all these situations:

- GN v2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139-mcp1.4 schema
they all have:

<xsl:template match="gmd:dateStamp">
    <xsl:choose>
        <xsl:when test="/root/env/changeDate">
            <xsl:copy>
                    <gco:DateTime>
                        <xsl:value-of select="/root/env/changeDate"/>
                    </gco:DateTime>
            </xsl:copy>
        </xsl:when>
        <xsl:otherwise>
            <xsl:copy-of select="."/>
        </xsl:otherwise>
    </xsl:choose>
</xsl:template>

Hi Simon,

Could you let me know your understanding of how the 'Geonetwork' protocol
harvester
decides whether to harvest or not in the older Geonetwork+BlueNet and more
recent
Geonetwork+ANZMEST versions?

It seems to me that the Geonetwork protocol harvester and CSW protocol harvester
could use the
database changeDate but maybe I am missing some point?

Regards All,

Andrew

> Tuesday, 30 October 2012 21:22 Heikki Doeleman wrote"

>hi Andrew,

>by GN harvesters do you mean, GeoNetwork protocol harvesters ?

>Those could use the database changedate, but currently they don't do that; if a
>metadata with the same uuid as one in the >harvesting result exists, it is
>overwritten, with no regard to timestamp.

>CSW harvesting compares local client's dateStamp to the the metadata in the
>harvesting result to decide whether to update or >to never mind, but if the
>date element in the ISO19139 metadata is not guaranteed to be a
>LastModificationDate (which seems >to be so from your previous emails in this
>thread), then that update system should be considered broken, I guess, and we
> >should rather ignore datestamps altogether and just replace metadata having
>the same uuid as metadata in the harvest results.

>Kind regards
>Heikki Doeleman

----- Original Message -----
From: heikki
To: Annina Hirschi Wyss
Cc: ianwallen@anonymised.com ; geonetwork-devel@lists.sourceforge.net
Sent: Wednesday, October 31, 2012 12:48 AM
Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

.. just to clarify: I do not propose here to change the behaviour that
GeoNetwork uses this element as a last modification date; only to make the check
in CSW (and maybe other) harvester fool-proof to this in case the harvested
catalog does not use the element in this way.

On Tue, Oct 30, 2012 at 2:43 PM, heikki <tropicano@anonymised.com> wrote:

hi,

discussed briefly with François and created a ticket to change the behaviour:
http://trac.osgeo.org/geonetwork/ticket/1124. Set for 2.8.0 as the change is
very small.

If people don't agree to this change, please comment in the ticket asap.

Kind regards
Heikki Doeleman

On Tue, Oct 30, 2012 at 2:15 PM, Annina Hirschi Wyss <annina.hirschi@anonymised.com>
wrote:

Hi,

Some remarks from my side:

You are right with the ISO "restrictions" about the creation date, but where
would you actually store the revision/update information for the metadata set? I
don't think there is a possibility.
Within the maintenance class, there is only an attribute for the next scheduled
update.

I think this missing element is the reason, why many implementations (not only
GeoNetwork) use the field to store the (IMHO more useful and meaningful)
information about the last modification (automatic update on saving the
metadata).

When we worked on the German translation of ISO 19115[*], we also discussed this
issue with several participants from different offices and enterprises in
Germany and Switzerland and finally agreed to include the "change" in the
attribute description to sort of allow storing the change date.
One reason to allow change dates were also the INSPIRE technical guidelines[**]
where the following comment is made on dateStamp:

'ISO is more restrictive because this element
shall contain the "date that the metadata
was created" and INSPIRE may contain the
"date when the metadata record was
created or updated"'

Annina

[*]resulting in an official Document of the German SDI:
http://www.gdi-de.org/download/AK/ISO19115_GermanTranslation_GDIDE.pdf

[**]http://inspire.jrc.ec.europa.eu/documents/Metadata/INSPIRE_MD_IR_and_ISO_v1_2_20100616.pdf

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Hi John,

Totally agree, the definition of the gmd:dateStamp shouldn't have been changed.
Would have been better in hindsight to create another metadata element to track the change date
such as was done with the ISO1939-mcp profile i.e the mcp:RevisonDate but too late
now as we have seen with the NAP, USGIN and INSPIRE profiles.

Heikki and Simon,

Thank you for clarifying how the Geonetwork harvester and CSW harvester
keep track of which records to re-harvest. It's a real surprise to me how the Geonetwork
protocol harvester is working in the Aligner.java code sample you provided.
It seems not efficient that the harvest client does an update of ALL the metadata
records even though they had not changed on the server.

Could we have an enhancement to the Geonetwork protocol harvester so that only
records with a newer database changeDate on the server get updated on the client?

Andrew

----- Original Message ----- From: "john.hockaday" <john.hockaday@anonymised.com>
To: "andrew walsh" <awalsh@anonymised.com>
Cc: <tropicano@anonymised.com>; <geonetwork-devel@lists.sourceforge.net>
Sent: Thursday, November 01, 2012 1:12 PM
Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

Hi Andrew et al,

The definition of dateStamp is the "date the metadata was created". Yu
*must* not change that definition. ISO 19115 is *not* concerned with
implementation. Hence using the logic of "how else do you know the
change date" or "everyone needs that change date" are not good reasons
to change the definition of dateStamp. If you need a change date for the
metadata then you must implement it in the back end of the database, as
suggested by Andrew, or create a profile which contains a new element
for the metadata change date. You *must not* change the definition of
and existing ISO 19115 metadata element.

I'm extremely surprised that the people that I worked with to develop
ISO TC 211 standards allowed this misuse of dateStamp to be implemented
in national profiles.

John Hockaday

On Wed, 2012-10-31 at 18:06 +1100, andrew walsh wrote:

Hello Heikki, Ian, Simon, John, Annina and All,

Yes, Heikki to confirm I was talking about the 'Geonetwork' protocol harvester.

I had always thought the Geonetwork had used the database 'changeDate' in the
metadata table as a means to decide to harvest or not. This seemed logical to
me.

My reasoning for this thought is the BlueNet MEST version of Geonetwork
we have been using since August 2010 DOES NOT update the gmd:dateStamp when
records are edited/saved.

Looking at the update-fixed-info.xsl from the Bluenet MEST version 1.4.9
(14/10/2011) of GN
for the ISO19139-mcp schema it says:

<xsl:template match="gmd:dateStamp">
  <xsl:choose>
   <xsl:when test="/root/env/updateDateStamp='yes'">
    <gmd:dateStamp>
     <gco:DateTime><xsl:value-of select="/root/env/changeDate"/></gco:DateTime>
    </gmd:dateStamp>
   </xsl:when>
   <xsl:otherwise>
    <xsl:copy-of select="."/>
   </xsl:otherwise>
  </xsl:choose>
</xsl:template>

That is, only update the gmd:DateStamp if ="/root/env/updateDateStamp='yes'
which I never found occurring. How would the /root/env/updateDateStamp be set to
yes?
Simon, maybe you could help here.

Given that I never observed the gmd:DateStamp updating it never occurred to me
that this
would used by the harvester as check whether to harvest a record. So I assumed
the
harvester was looking rather at the database changeDate to decide whether to
harvest or not.

Whereas the core (International version) of Geonetwork from
v2.6.3 up until the current V2.8.0RC1 DOES update the gmd:dateStamp on edit/save
as recently noted by Ian Allen (thanks! Ian).

Looking at update-fixed-info.xsl in all these situations:

- GN v2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139-mcp1.4 schema
they all have:

<xsl:template match="gmd:dateStamp">
    <xsl:choose>
        <xsl:when test="/root/env/changeDate">
            <xsl:copy>
                    <gco:DateTime>
                        <xsl:value-of select="/root/env/changeDate"/>
                    </gco:DateTime>
            </xsl:copy>
        </xsl:when>
        <xsl:otherwise>
            <xsl:copy-of select="."/>
        </xsl:otherwise>
    </xsl:choose>
</xsl:template>

Hi Simon,

Could you let me know your understanding of how the 'Geonetwork' protocol
harvester
decides whether to harvest or not in the older Geonetwork+BlueNet and more
recent
Geonetwork+ANZMEST versions?

It seems to me that the Geonetwork protocol harvester and CSW protocol harvester
could use the
database changeDate but maybe I am missing some point?

Regards All,

Andrew

> Tuesday, 30 October 2012 21:22 Heikki Doeleman wrote"

>hi Andrew,

>by GN harvesters do you mean, GeoNetwork protocol harvesters ?

>Those could use the database changedate, but currently they don't do that; >if a
>metadata with the same uuid as one in the >harvesting result exists, it is
>overwritten, with no regard to timestamp.

>CSW harvesting compares local client's dateStamp to the the metadata in the
>harvesting result to decide whether to update or >to never mind, but if the
>date element in the ISO19139 metadata is not guaranteed to be a
>LastModificationDate (which seems >to be so from your previous emails in >this
>thread), then that update system should be considered broken, I guess, and >we
> >should rather ignore datestamps altogether and just replace metadata > >having
>the same uuid as metadata in the harvest results.

>Kind regards
>Heikki Doeleman

----- Original Message ----- From: heikki
To: Annina Hirschi Wyss
Cc: ianwallen@anonymised.com ; geonetwork-devel@lists.sourceforge.net
Sent: Wednesday, October 31, 2012 12:48 AM
Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

.. just to clarify: I do not propose here to change the behaviour that
GeoNetwork uses this element as a last modification date; only to make the check
in CSW (and maybe other) harvester fool-proof to this in case the harvested
catalog does not use the element in this way.

On Tue, Oct 30, 2012 at 2:43 PM, heikki <tropicano@anonymised.com> wrote:

hi,

discussed briefly with François and created a ticket to change the behaviour:
http://trac.osgeo.org/geonetwork/ticket/1124. Set for 2.8.0 as the change is
very small.

If people don't agree to this change, please comment in the ticket asap.

Kind regards
Heikki Doeleman

On Tue, Oct 30, 2012 at 2:15 PM, Annina Hirschi Wyss <annina.hirschi@anonymised.com458...>
wrote:

Hi,

Some remarks from my side:

You are right with the ISO "restrictions" about the creation date, but where
would you actually store the revision/update information for the metadata set? I
don't think there is a possibility.
Within the maintenance class, there is only an attribute for the next scheduled
update.

I think this missing element is the reason, why many implementations (not only
GeoNetwork) use the field to store the (IMHO more useful and meaningful)
information about the last modification (automatic update on saving the
metadata).

When we worked on the German translation of ISO 19115[*], we also discussed this
issue with several participants from different offices and enterprises in
Germany and Switzerland and finally agreed to include the "change" in the
attribute description to sort of allow storing the change date.
One reason to allow change dates were also the INSPIRE technical guidelines[**]
where the following comment is made on dateStamp:

'ISO is more restrictive because this element
shall contain the "date that the metadata
was created" and INSPIRE may contain the
"date when the metadata record was
created or updated"'

Annina

[*]resulting in an official Document of the German SDI:
http://www.gdi-de.org/download/AK/ISO19115_GermanTranslation_GDIDE.pdf

[**]http://inspire.jrc.ec.europa.eu/documents/Metadata/INSPIRE_MD_IR_and_ISO_v1_2_20100616.pdf

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Hello Again Heikki,

I just had another look into geonet20/Aligner.java and looks like it is using
the database changeDate in deciding whether to update or not. Looking further into
the updateMetadata(..) method it appears to get the changeDate from the remote site:

private void updateMetadata(String siteId, Element info, String id) throws Exception
{
  String remoteId = info.getChildText("id");
  String remoteUuid= info.getChildText("uuid");
  String changeDate= info.getChildText("changeDate");

  if (localUuids.getID(remoteUuid) == null)
  {
   log.error(" - Warning! The remote uuid '"+ remoteUuid +"' does not belong to site '"+ siteId+"'");
   log.error(" - The site id of this metadata has been changed.");
   log.error(" - The metadata update will be skipped.");

   result.uuidSkipped++;
  }
  else
  {
   updateMetadata(id, remoteId, remoteUuid, changeDate);
   updateCategories(id, info);
  }
}

Then an overloaded updateMetadata() is called which gets the changeDate on the local site:

private void updateMetadata(String id, String remoteId, String remoteUuid, String changeDate) throws Exception
{
  String date = localUuids.getChangeDate(remoteUuid);

<Comment: "date" = local change date, "changeDate" = remote change Date >

  if (!updateCondition(date, changeDate))
  {
            if(log.isDebugEnabled()) log.debug(" - XML not changed to local metadata with id="+ id);
   result.unchangedMetadata++;
  }
  else
  {
            if(log.isDebugEnabled()) log.debug(" - Updating local metadata with id="+ id);

   Element md = getRemoteMetadata(req, remoteId);

   if (md == null)
    log.warning(" - Cannot get metadata (possibly bad XML) with remote id="+ remoteId);
   else {
                //
                // update metadata
                //
                boolean validate = false;
                boolean ufo = false;
                boolean index = false;
                String language = context.getLanguage();
                dataMan.updateMetadata(context, dbms, id, md, validate, ufo, index, language, changeDate, false);

    result.updatedMetadata++;
   }
  }
}

Above method uses updateCondition() below which compares the local and remote date.
Returning true (1) if the remote date is greater (newer) than the local date
and if true running dataMan.updateMetadata(..) and setting result.updatedMetadata++;

private boolean updateCondition(String localDate, String remoteDate)
{
  ISODate local = new ISODate(localDate);
  ISODate remote= new ISODate(remoteDate);

  //--- accept if remote date is greater than local date

  return (remote.sub(local) > 0);
}

This is how I thought the harvester was working.

Andrew

----- Original Message ----- From: heikki
To: andrew walsh
Cc: geonetwork-devel@lists.sourceforge.net
Sent: Wednesday, October 31, 2012 8:37 PM
Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

hi Andrew,

the code for the GeoNetwork 2.0 protocol harvester, where the client decides whether to update a metadata if it already exists, is here, from line 99: https://github.com/geonetwork/core-geonetwork/blob/master/web/src/main/java/org/fao/geonet/kernel/harvest/harvester/geonet20/Aligner.java.

As you can see, although changeDate seems to be available, it is only used to debug-log. The update-or-not decision is this

                    String id = dataMan.getMetadataId(dbms, remoteUuid);
                    if (id == null) {
                        id = addMetadata(info);
                    }
                    else {
                        updateMetadata(siteId, info, id);
                    }
                    dbms.commit();

which doesn't look at the date.

Anyway I don't think this is all that important, because the end result should be the same whether it looks at the date or not -- it may just prevent some unnecessary updates if it would look at the date.

It also explains why things seem to work for you even if you see that the dateStamp inside your metadata never gets updated. It just doesn't look at any date, neither from the database table, nor from the metadata element, and will always update existing metadata.

To me it's more important to remove the date check from the CSW harvester, which does use the dateStamp field from inside the metadata, when the content of that field may not be the last modification date, in all cases.

Does this make sense ?

Kind regards
Heikki Doeleman

On Wed, Oct 31, 2012 at 8:06 AM, andrew walsh <awalsh@anonymised.com> wrote:

Hello Heikki, Ian, Simon, John, Annina and All,

Yes, Heikki to confirm I was talking about the 'Geonetwork' protocol harvester.

I had always thought the Geonetwork had used the database 'changeDate' in the
metadata table as a means to decide to harvest or not. This seemed logical to me.

My reasoning for this thought is the BlueNet MEST version of Geonetwork
we have been using since August 2010 DOES NOT update the gmd:dateStamp when
records are edited/saved.

Looking at the update-fixed-info.xsl from the Bluenet MEST version 1.4.9 (14/10/2011) of GN
for the ISO19139-mcp schema it says:

<xsl:template match="gmd:dateStamp">
<xsl:choose>
  <xsl:when test="/root/env/updateDateStamp='yes'">
   <gmd:dateStamp>
    <gco:DateTime><xsl:value-of select="/root/env/changeDate"/></gco:DateTime>
   </gmd:dateStamp>
  </xsl:when>
  <xsl:otherwise>
   <xsl:copy-of select="."/>
  </xsl:otherwise>
</xsl:choose>
</xsl:template>

That is, only update the gmd:DateStamp if ="/root/env/updateDateStamp='yes'
which I never found occurring. How would the /root/env/updateDateStamp be set to yes?
Simon, maybe you could help here.

Given that I never observed the gmd:DateStamp updating it never occurred to me that this
would used by the harvester as check whether to harvest a record. So I assumed the
harvester was looking rather at the database changeDate to decide whether to harvest or not.

Whereas the core (International version) of Geonetwork from
v2.6.3 up until the current V2.8.0RC1 DOES update the gmd:dateStamp on edit/save
as recently noted by Ian Allen (thanks! Ian).

Looking at update-fixed-info.xsl in all these situations:

- GN v2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139 schema
- GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139-mcp1.4 schema
they all have:

<xsl:template match="gmd:dateStamp">
   <xsl:choose>
       <xsl:when test="/root/env/changeDate">
           <xsl:copy>
                   <gco:DateTime>
                       <xsl:value-of select="/root/env/changeDate"/>
                   </gco:DateTime>
           </xsl:copy>
       </xsl:when>
       <xsl:otherwise>
           <xsl:copy-of select="."/>
       </xsl:otherwise>
   </xsl:choose>
</xsl:template>

Hi Simon,

Could you let me know your understanding of how the 'Geonetwork' protocol harvester
decides whether to harvest or not in the older Geonetwork+BlueNet and more recent
Geonetwork+ANZMEST versions?

It seems to me that the Geonetwork protocol harvester and CSW protocol harvester could use the
database changeDate but maybe I am missing some point?

Regards All,

Andrew

Tuesday, 30 October 2012 21:22 Heikki Doeleman wrote"

hi Andrew,

by GN harvesters do you mean, GeoNetwork protocol harvesters ?

Those could use the database changedate, but currently they don't do that; if a metadata with the same uuid as one in the >harvesting result exists, it is overwritten, with no regard to timestamp.

CSW harvesting compares local client's dateStamp to the the metadata in the harvesting result to decide whether to update or >to never mind, but if the date element in the ISO19139 metadata is not guaranteed to be a LastModificationDate (which seems >to be so from your previous emails in this thread), then that update system should be considered broken, I guess, and we >should rather ignore datestamps altogether and just replace metadata having the same uuid as metadata in the harvest results.

Kind regards
Heikki Doeleman

----- Original Message ----- From: heikki
To: Annina Hirschi Wyss
Cc: ianwallen@anonymised.com ; geonetwork-devel@lists.sourceforge.net
Sent: Wednesday, October 31, 2012 12:48 AM

Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

.. just to clarify: I do not propose here to change the behaviour that GeoNetwork uses this element as a last modification date; only to make the check in CSW (and maybe other) harvester fool-proof to this in case the harvested catalog does not use the element in this way.

On Tue, Oct 30, 2012 at 2:43 PM, heikki <tropicano@anonymised.com> wrote:

hi,

discussed briefly with François and created a ticket to change the behaviour: http://trac.osgeo.org/geonetwork/ticket/1124. Set for 2.8.0 as the change is very small.

If people don't agree to this change, please comment in the ticket asap.

Kind regards
Heikki Doeleman

On Tue, Oct 30, 2012 at 2:15 PM, Annina Hirschi Wyss <annina.hirschi@anonymised.com...> wrote:

Hi,

Some remarks from my side:

You are right with the ISO "restrictions" about the creation date, but where would you actually store the revision/update information for the metadata set? I don't think there is a possibility.
Within the maintenance class, there is only an attribute for the next scheduled update.

I think this missing element is the reason, why many implementations (not only GeoNetwork) use the field to store the (IMHO more useful and meaningful) information about the last modification (automatic update on saving the metadata).

When we worked on the German translation of ISO 19115[*], we also discussed this issue with several participants from different offices and enterprises in Germany and Switzerland and finally agreed to include the "change" in the attribute description to sort of allow storing the change date.
One reason to allow change dates were also the INSPIRE technical guidelines[**] where the following comment is made on dateStamp:

'ISO is more restrictive because this element
shall contain the "date that the metadata
was created" and INSPIRE may contain the
"date when the metadata record was
created or updated"'

Annina

[*]resulting in an official Document of the German SDI: http://www.gdi-de.org/download/AK/ISO19115_GermanTranslation_GDIDE.pdf

[**]http://inspire.jrc.ec.europa.eu/documents/Metadata/INSPIRE_MD_IR_and_ISO_v1_2_20100616.pdf

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct

Hi Andrew et al,

A long time ago Doug Nebert gae me some PERL script that I further
worked on that harvested remote metadata. I think it used something like
the HTML updated or last-updated condition of a plain text file. This
worked really well because the metadata file was only harvested if the
HTML condition was met.

I don't think that this condition could be decided on in a database but
using a database element to decide on the need to harvest would be much
more efficient.

Thanks.

John

On Fri, 2012-11-02 at 11:59 +1100, andrew walsh wrote:

Hi John,

Totally agree, the definition of the gmd:dateStamp shouldn't have been changed.
Would have been better in hindsight to create another metadata element to track
the change date
such as was done with the ISO1939-mcp profile i.e the mcp:RevisonDate but too
late
now as we have seen with the NAP, USGIN and INSPIRE profiles.

Heikki and Simon,

Thank you for clarifying how the Geonetwork harvester and CSW harvester
keep track of which records to re-harvest. It's a real surprise to me how the
Geonetwork
protocol harvester is working in the Aligner.java code sample you provided.
It seems not efficient that the harvest client does an update of ALL the
metadata
records even though they had not changed on the server.

Could we have an enhancement to the Geonetwork protocol harvester so that only
records with a newer database changeDate on the server get updated on the
client?

Andrew

----- Original Message -----
From: "john.hockaday" <john.hockaday@anonymised.com>
To: "andrew walsh" <awalsh@anonymised.com>
Cc: <tropicano@anonymised.com>; <geonetwork-devel@lists.sourceforge.net>
Sent: Thursday, November 01, 2012 1:12 PM
Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139

> Hi Andrew et al,
>
> The definition of dateStamp is the "date the metadata was created". Yu
> *must* not change that definition. ISO 19115 is *not* concerned with
> implementation. Hence using the logic of "how else do you know the
> change date" or "everyone needs that change date" are not good reasons
> to change the definition of dateStamp. If you need a change date for the
> metadata then you must implement it in the back end of the database, as
> suggested by Andrew, or create a profile which contains a new element
> for the metadata change date. You *must not* change the definition of
> and existing ISO 19115 metadata element.
>
> I'm extremely surprised that the people that I worked with to develop
> ISO TC 211 standards allowed this misuse of dateStamp to be implemented
> in national profiles.
>
> John Hockaday
>
> On Wed, 2012-10-31 at 18:06 +1100, andrew walsh wrote:
>> Hello Heikki, Ian, Simon, John, Annina and All,
>>
>> Yes, Heikki to confirm I was talking about the 'Geonetwork' protocol
>> harvester.
>>
>> I had always thought the Geonetwork had used the database 'changeDate' in the
>> metadata table as a means to decide to harvest or not. This seemed logical to
>> me.
>>
>> My reasoning for this thought is the BlueNet MEST version of Geonetwork
>> we have been using since August 2010 DOES NOT update the gmd:dateStamp when
>> records are edited/saved.
>>
>> Looking at the update-fixed-info.xsl from the Bluenet MEST version 1.4.9
>> (14/10/2011) of GN
>> for the ISO19139-mcp schema it says:
>>
>> <xsl:template match="gmd:dateStamp">
>> <xsl:choose>
>> <xsl:when test="/root/env/updateDateStamp='yes'">
>> <gmd:dateStamp>
>> <gco:DateTime><xsl:value-of
>> select="/root/env/changeDate"/></gco:DateTime>
>> </gmd:dateStamp>
>> </xsl:when>
>> <xsl:otherwise>
>> <xsl:copy-of select="."/>
>> </xsl:otherwise>
>> </xsl:choose>
>> </xsl:template>
>>
>> That is, only update the gmd:DateStamp if ="/root/env/updateDateStamp='yes'
>> which I never found occurring. How would the /root/env/updateDateStamp be set
>> to
>> yes?
>> Simon, maybe you could help here.
>>
>> Given that I never observed the gmd:DateStamp updating it never occurred to
>> me
>> that this
>> would used by the harvester as check whether to harvest a record. So I
>> assumed
>> the
>> harvester was looking rather at the database changeDate to decide whether to
>> harvest or not.
>>
>> Whereas the core (International version) of Geonetwork from
>> v2.6.3 up until the current V2.8.0RC1 DOES update the gmd:dateStamp on
>> edit/save
>> as recently noted by Ian Allen (thanks! Ian).
>>
>> Looking at update-fixed-info.xsl in all these situations:
>>
>> - GN v2.8.0RC1 with ISO19139 schema
>> - GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139 schema
>> - GN v2.8.0RC1 + ANZMEST 2.8.0RC1 with ISO19139-mcp1.4 schema
>> they all have:
>>
>> <xsl:template match="gmd:dateStamp">
>> <xsl:choose>
>> <xsl:when test="/root/env/changeDate">
>> <xsl:copy>
>> <gco:DateTime>
>> <xsl:value-of select="/root/env/changeDate"/>
>> </gco:DateTime>
>> </xsl:copy>
>> </xsl:when>
>> <xsl:otherwise>
>> <xsl:copy-of select="."/>
>> </xsl:otherwise>
>> </xsl:choose>
>> </xsl:template>
>>
>> Hi Simon,
>>
>> Could you let me know your understanding of how the 'Geonetwork' protocol
>> harvester
>> decides whether to harvest or not in the older Geonetwork+BlueNet and more
>> recent
>> Geonetwork+ANZMEST versions?
>>
>> It seems to me that the Geonetwork protocol harvester and CSW protocol
>> harvester
>> could use the
>> database changeDate but maybe I am missing some point?
>>
>> Regards All,
>>
>> Andrew
>>
>>
>> > Tuesday, 30 October 2012 21:22 Heikki Doeleman wrote"
>>
>> >hi Andrew,
>>
>>
>> >by GN harvesters do you mean, GeoNetwork protocol harvesters ?
>>
>>
>> >Those could use the database changedate, but currently they don't do that;
>> >if a
>> >metadata with the same uuid as one in the >harvesting result exists, it is
>> >overwritten, with no regard to timestamp.
>>
>>
>> >CSW harvesting compares local client's dateStamp to the the metadata in the
>> >harvesting result to decide whether to update or >to never mind, but if the
>> >date element in the ISO19139 metadata is not guaranteed to be a
>> >LastModificationDate (which seems >to be so from your previous emails in
>> >this
>> >thread), then that update system should be considered broken, I guess, and
>> >we
>> > >should rather ignore datestamps altogether and just replace metadata
>> > >having
>> >the same uuid as metadata in the harvest results.
>>
>>
>> >Kind regards
>> >Heikki Doeleman
>>
>>
>> ----- Original Message -----
>> From: heikki
>> To: Annina Hirschi Wyss
>> Cc: ianwallen@anonymised.com ; geonetwork-devel@lists.sourceforge.net
>> Sent: Wednesday, October 31, 2012 12:48 AM
>> Subject: Re: [GeoNetwork-devel] Question on regarding dateStamp in iso 19139
>>
>>
>> .. just to clarify: I do not propose here to change the behaviour that
>> GeoNetwork uses this element as a last modification date; only to make the
>> check
>> in CSW (and maybe other) harvester fool-proof to this in case the harvested
>> catalog does not use the element in this way.
>>
>>
>>
>> On Tue, Oct 30, 2012 at 2:43 PM, heikki <tropicano@anonymised.com> wrote:
>>
>> hi,
>>
>>
>> discussed briefly with François and created a ticket to change the behaviour:
>> http://trac.osgeo.org/geonetwork/ticket/1124. Set for 2.8.0 as the change is
>> very small.
>>
>>
>> If people don't agree to this change, please comment in the ticket asap.
>>
>>
>> Kind regards
>> Heikki Doeleman
>>
>>
>>
>>
>>
>> On Tue, Oct 30, 2012 at 2:15 PM, Annina Hirschi Wyss <annina.hirschi@anonymised.com>
>> wrote:
>>
>> Hi,
>>
>> Some remarks from my side:
>>
>> You are right with the ISO "restrictions" about the creation date, but where
>> would you actually store the revision/update information for the metadata
>> set? I
>> don't think there is a possibility.
>> Within the maintenance class, there is only an attribute for the next
>> scheduled
>> update.
>>
>> I think this missing element is the reason, why many implementations (not
>> only
>> GeoNetwork) use the field to store the (IMHO more useful and meaningful)
>> information about the last modification (automatic update on saving the
>> metadata).
>>
>> When we worked on the German translation of ISO 19115[*], we also discussed
>> this
>> issue with several participants from different offices and enterprises in
>> Germany and Switzerland and finally agreed to include the "change" in the
>> attribute description to sort of allow storing the change date.
>> One reason to allow change dates were also the INSPIRE technical
>> guidelines[**]
>> where the following comment is made on dateStamp:
>>
>> 'ISO is more restrictive because this element
>> shall contain the "date that the metadata
>> was created" and INSPIRE may contain the
>> "date when the metadata record was
>> created or updated"'
>>
>>
>> Annina
>>
>>
>> [*]resulting in an official Document of the German SDI:
>> http://www.gdi-de.org/download/AK/ISO19115_GermanTranslation_GDIDE.pdf
>>
>> [**]http://inspire.jrc.ec.europa.eu/documents/Metadata/INSPIRE_MD_IR_and_ISO_v1_2_20100616.pdf
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> GeoNetwork-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>> _______________________________________________
>> GeoNetwork-devel mailing list
>> GeoNetwork-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
>> GeoNetwork OpenSource is maintained at
>> http://sourceforge.net/projects/geonetwork
>
>