[GeoNetwork-devel] OAI bug

Dear Geonetwork developers,

some months ago I wrote an email about an issue with the OAI provider of Geonetwork. Since the email has not sparked much discussion, but the
issue persists, I'm writing this second email.

The issue is that Geonetwork cannot be harvested with most OAI harvesters, since the handling of the "resumptionToken" is broken.
This can e.g be seen by using the attached code to harvest a GN node. (need oclc.oai.harvester2 jar)
If the GN installation contains more metadata records than fit into one
OAI response page, the harvesting will stop after the first page. This is because the resumptionToken sent by the harvester is considered illegal by GN.

I have written and committed a patch to Geonetwork BlueMestNet, which
has been operational (active OAI usage) in several nodes in the WMO
community. I would like to submit this patch to GN as well, as fix of
an essential feature.

Technically, the patch adds a OaiResumptionToken cache to store the resultSets associated to a resumption Token. The cache is periodically
purged by a thread. The interval as well as cachesize are added as parameters to the configuration options.

More details in my commit to BlueMestNet Sandbox (revision 6006
) 22/4/2010.

please let me know how we can proceed
Timo

(attachments)

Testharvest.java (1.76 KB)

Hi Timo,

I would suggest you to open a ticket on trac for that issue.
http://trac.osgeo.org/geonetwork/newticket

The idea could be also to provide a patch attached to the ticket in order to be integrated into the trunk.

I don’t think we need any proposal to add this fix / new functionality / improvement to the trunk.

Once the patch will be enclosed to the ticket we would be able to commit it into trunk.

Thank you for your contribution,

Best Regards,

Mathieu Coudert

On Wed, Jun 16, 2010 at 2:59 PM, Timo Pröscholdt <timo@anonymised.com> wrote:

Dear Geonetwork developers,

some months ago I wrote an email about an issue with the OAI provider of Geonetwork. Since the email has not sparked much discussion, but the
issue persists, I’m writing this second email.

The issue is that Geonetwork cannot be harvested with most OAI harvesters, since the handling of the “resumptionToken” is broken.
This can e.g be seen by using the attached code to harvest a GN node. (need oclc.oai.harvester2 jar)
If the GN installation contains more metadata records than fit into one
OAI response page, the harvesting will stop after the first page. This is because the resumptionToken sent by the harvester is considered illegal by GN.

I have written and committed a patch to Geonetwork BlueMestNet, which
has been operational (active OAI usage) in several nodes in the WMO
community. I would like to submit this patch to GN as well, as fix of
an essential feature.

Technically, the patch adds a OaiResumptionToken cache to store the resultSets associated to a resumption Token. The cache is periodically
purged by a thread. The interval as well as cachesize are added as parameters to the configuration options.

More details in my commit to BlueMestNet Sandbox (revision 6006
) 22/4/2010.

please let me know how we can proceed
Timo


ThinkGeek and WIRED’s GeekDad team up for the Ultimate
GeekDad Father’s Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo


GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Hi Mathieu and Timo,

Mathieu this has been in the BlueNetMEST for a while - rather than make Timo resubmit it, its probably my responsibility :slight_smile: to get it into the trunk - especially now that the trunk has gone to maven and changed the paths!. I'll get moving on this in the next few days.

Cheers,
Simon
________________________________________
From: Mathieu Coudert [mathieu.coudert@anonymised.com]
Sent: Wednesday, 16 June 2010 11:39 PM
To: Timo Pröscholdt
Cc: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] OAI bug

Hi Timo,

I would suggest you to open a ticket on trac for that issue.
http://trac.osgeo.org/geonetwork/newticket

The idea could be also to provide a patch attached to the ticket in order to be integrated into the trunk.

I don't think we need any proposal to add this fix / new functionality / improvement to the trunk.

Once the patch will be enclosed to the ticket we would be able to commit it into trunk.

Thank you for your contribution,

Best Regards,

Mathieu Coudert

On Wed, Jun 16, 2010 at 2:59 PM, Timo Pröscholdt <timo@anonymised.com<mailto:timo@anonymised.com>> wrote:
Dear Geonetwork developers,

some months ago I wrote an email about an issue with the OAI provider of Geonetwork. Since the email has not sparked much discussion, but the
issue persists, I'm writing this second email.

The issue is that Geonetwork cannot be harvested with most OAI harvesters, since the handling of the "resumptionToken" is broken.
This can e.g be seen by using the attached code to harvest a GN node. (need oclc.oai.harvester2 jar)
If the GN installation contains more metadata records than fit into one
OAI response page, the harvesting will stop after the first page. This is because the resumptionToken sent by the harvester is considered illegal by GN.

I have written and committed a patch to Geonetwork BlueMestNet, which
has been operational (active OAI usage) in several nodes in the WMO
community. I would like to submit this patch to GN as well, as fix of
an essential feature.

Technically, the patch adds a OaiResumptionToken cache to store the resultSets associated to a resumption Token. The cache is periodically
purged by a thread. The interval as well as cachesize are added as parameters to the configuration options.

More details in my commit to BlueMestNet Sandbox (revision 6006
) 22/4/2010.

please let me know how we can proceed
Timo

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

On Wed, Jun 16, 2010 at 4:30 PM, Simon.Pigot@anonymised.com wrote:

Hi Mathieu and Timo,

Mathieu this has been in the BlueNetMEST for a while - rather than make Timo resubmit it, its probably my responsibility :slight_smile: to get it into the trunk - especially now that the trunk has gone to maven and changed the paths!. I’ll get moving on this in the next few days.

Sorry for the extra work Simon! :wink:

However, this does not prevent us from opening a ticket to address this problem.
What do you think Simon?

Cheers,

Mathieu

Cheers,
Simon


From: Mathieu Coudert [mathieu.coudert@anonymised.com]
Sent: Wednesday, 16 June 2010 11:39 PM
To: Timo Pröscholdt
Cc: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] OAI bug

Hi Timo,

I would suggest you to open a ticket on trac for that issue.
http://trac.osgeo.org/geonetwork/newticket

The idea could be also to provide a patch attached to the ticket in order to be integrated into the trunk.

I don’t think we need any proposal to add this fix / new functionality / improvement to the trunk.

Once the patch will be enclosed to the ticket we would be able to commit it into trunk.

Thank you for your contribution,

Best Regards,

Mathieu Coudert

On Wed, Jun 16, 2010 at 2:59 PM, Timo Pröscholdt <timo@anonymised.commailto:[timo@anonymised.com](mailto:timo@anonymised.com)> wrote:
Dear Geonetwork developers,

some months ago I wrote an email about an issue with the OAI provider of Geonetwork. Since the email has not sparked much discussion, but the
issue persists, I’m writing this second email.

The issue is that Geonetwork cannot be harvested with most OAI harvesters, since the handling of the “resumptionToken” is broken.
This can e.g be seen by using the attached code to harvest a GN node. (need oclc.oai.harvester2 jar)
If the GN installation contains more metadata records than fit into one
OAI response page, the harvesting will stop after the first page. This is because the resumptionToken sent by the harvester is considered illegal by GN.

I have written and committed a patch to Geonetwork BlueMestNet, which
has been operational (active OAI usage) in several nodes in the WMO
community. I would like to submit this patch to GN as well, as fix of
an essential feature.

Technically, the patch adds a OaiResumptionToken cache to store the resultSets associated to a resumption Token. The cache is periodically
purged by a thread. The interval as well as cachesize are added as parameters to the configuration options.

More details in my commit to BlueMestNet Sandbox (revision 6006
) 22/4/2010.

please let me know how we can proceed
Timo


ThinkGeek and WIRED’s GeekDad team up for the Ultimate
GeekDad Father’s Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo


GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

hi all,

On 6/16/2010 4:55 PM, Mathieu Coudert wrote:

On Wed, Jun 16, 2010 at 4:30 PM, <Simon.Pigot@anonymised.com> wrote:

    Hi Mathieu and Timo,

    Mathieu this has been in the BlueNetMEST for a while - rather than
    make Timo resubmit it, its probably my responsibility :slight_smile: to get it
    into the trunk - especially now that the trunk has gone to maven and
    changed the paths!. I'll get moving on this in the next few days.

i'm grateful for this, especially since I tried getting the current repo
to work to no avail today..

Sorry for the extra work Simon! :wink:

However, this does not prevent us from opening a ticket to address this
problem.
What do you think Simon?

I will open the ticket tomorrow so that procedure is followed. I guess that's ok with you, Simon?

best!
Timo

Cheers,

Mathieu

    Cheers,
    Simon
    ________________________________________
    From: Mathieu Coudert [mathieu.coudert@anonymised.com
    <mailto:mathieu.coudert@anonymised.com>]
    Sent: Wednesday, 16 June 2010 11:39 PM
    To: Timo Pröscholdt
    Cc: geonetwork-devel@lists.sourceforge.net
    <mailto:geonetwork-devel@lists.sourceforge.net>
    Subject: Re: [GeoNetwork-devel] OAI bug

    Hi Timo,

    I would suggest you to open a ticket on trac for that issue.
    http://trac.osgeo.org/geonetwork/newticket

    The idea could be also to provide a patch attached to the ticket in
    order to be integrated into the trunk.

    I don't think we need any proposal to add this fix / new
    functionality / improvement to the trunk.

    Once the patch will be enclosed to the ticket we would be able to
    commit it into trunk.

    Thank you for your contribution,

    Best Regards,

    Mathieu Coudert

    On Wed, Jun 16, 2010 at 2:59 PM, Timo Pröscholdt
    <timo@anonymised.com
    <mailto:timo@anonymised.com><mailto:timo@anonymised.com
    <mailto:timo@anonymised.com>>> wrote:
    Dear Geonetwork developers,

    some months ago I wrote an email about an issue with the OAI
    provider of Geonetwork. Since the email has not sparked much
    discussion, but the
    issue persists, I'm writing this second email.

    The issue is that Geonetwork cannot be harvested with most OAI
    harvesters, since the handling of the "resumptionToken" is broken.
    This can e.g be seen by using the attached code to harvest a GN
    node. (need oclc.oai.harvester2 jar)
    If the GN installation contains more metadata records than fit into one
    OAI response page, the harvesting will stop after the first page.
    This is because the resumptionToken sent by the harvester is
    considered illegal by GN.

    I have written and committed a patch to Geonetwork BlueMestNet, which
    has been operational (active OAI usage) in several nodes in the WMO
    community. I would like to submit this patch to GN as well, as fix of
    an essential feature.

    Technically, the patch adds a OaiResumptionToken cache to store the
    resultSets associated to a resumption Token. The cache is periodically
    purged by a thread. The interval as well as cachesize are added as
    parameters to the configuration options.

    More details in my commit to BlueMestNet Sandbox (revision 6006
    ) 22/4/2010.

    please let me know how we can proceed
    Timo

    ------------------------------------------------------------------------------
    ThinkGeek and WIRED's GeekDad team up for the Ultimate
    GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
    lucky parental unit. See the prize list and enter to win:
    http://p.sf.net/sfu/thinkgeek-promo
    _______________________________________________
    GeoNetwork-devel mailing list
    GeoNetwork-devel@lists.sourceforge.net
    <mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net
    <mailto:GeoNetwork-devel@lists.sourceforge.net>>
    https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
    GeoNetwork OpenSource is maintained at
    http://sourceforge.net/projects/geonetwork

No worries - go ahead and open the ticket.

Cheers,
Simon
________________________________________
From: Timo Pröscholdt [timo@anonymised.com]
Sent: Thursday, 17 June 2010 2:05 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] OAI bug

hi all,

On 6/16/2010 4:55 PM, Mathieu Coudert wrote:

On Wed, Jun 16, 2010 at 4:30 PM, <Simon.Pigot@anonymised.com> wrote:

    Hi Mathieu and Timo,

    Mathieu this has been in the BlueNetMEST for a while - rather than
    make Timo resubmit it, its probably my responsibility :slight_smile: to get it
    into the trunk - especially now that the trunk has gone to maven and
    changed the paths!. I'll get moving on this in the next few days.

i'm grateful for this, especially since I tried getting the current repo
to work to no avail today..

Sorry for the extra work Simon! :wink:

However, this does not prevent us from opening a ticket to address this
problem.
What do you think Simon?

I will open the ticket tomorrow so that procedure is followed. I guess
that's ok with you, Simon?

best!
Timo

Cheers,

Mathieu

    Cheers,
    Simon
    ________________________________________
    From: Mathieu Coudert [mathieu.coudert@anonymised.com
    <mailto:mathieu.coudert@anonymised.com>]
    Sent: Wednesday, 16 June 2010 11:39 PM
    To: Timo Pröscholdt
    Cc: geonetwork-devel@lists.sourceforge.net
    <mailto:geonetwork-devel@lists.sourceforge.net>
    Subject: Re: [GeoNetwork-devel] OAI bug

    Hi Timo,

    I would suggest you to open a ticket on trac for that issue.
    http://trac.osgeo.org/geonetwork/newticket

    The idea could be also to provide a patch attached to the ticket in
    order to be integrated into the trunk.

    I don't think we need any proposal to add this fix / new
    functionality / improvement to the trunk.

    Once the patch will be enclosed to the ticket we would be able to
    commit it into trunk.

    Thank you for your contribution,

    Best Regards,

    Mathieu Coudert

    On Wed, Jun 16, 2010 at 2:59 PM, Timo Pröscholdt
    <timo@anonymised.com
    <mailto:timo@anonymised.com><mailto:timo@anonymised.com
    <mailto:timo@anonymised.com>>> wrote:
    Dear Geonetwork developers,

    some months ago I wrote an email about an issue with the OAI
    provider of Geonetwork. Since the email has not sparked much
    discussion, but the
    issue persists, I'm writing this second email.

    The issue is that Geonetwork cannot be harvested with most OAI
    harvesters, since the handling of the "resumptionToken" is broken.
    This can e.g be seen by using the attached code to harvest a GN
    node. (need oclc.oai.harvester2 jar)
    If the GN installation contains more metadata records than fit into one
    OAI response page, the harvesting will stop after the first page.
    This is because the resumptionToken sent by the harvester is
    considered illegal by GN.

    I have written and committed a patch to Geonetwork BlueMestNet, which
    has been operational (active OAI usage) in several nodes in the WMO
    community. I would like to submit this patch to GN as well, as fix of
    an essential feature.

    Technically, the patch adds a OaiResumptionToken cache to store the
    resultSets associated to a resumption Token. The cache is periodically
    purged by a thread. The interval as well as cachesize are added as
    parameters to the configuration options.

    More details in my commit to BlueMestNet Sandbox (revision 6006
    ) 22/4/2010.

    please let me know how we can proceed
    Timo

    ------------------------------------------------------------------------------
    ThinkGeek and WIRED's GeekDad team up for the Ultimate
    GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
    lucky parental unit. See the prize list and enter to win:
    http://p.sf.net/sfu/thinkgeek-promo
    _______________________________________________
    GeoNetwork-devel mailing list
    GeoNetwork-devel@lists.sourceforge.net
    <mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net
    <mailto:GeoNetwork-devel@lists.sourceforge.net>>
    geonetwork-devel List Signup and Options
    GeoNetwork OpenSource is maintained at
    http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork