[GeoNetwork-users] OAI/PMH Harvesting

Holmgrimur_Helgason · November 9, 2016, 1:08pm

Hello

I'm trying to harvest using oai-pmh protocol with no results.

I'm using this url for the harvesting and it gives a result, but Geonetwork
isn't receiving anything.

<http://osgeo-org.1560.x6.nabble.com/file/n5295006/harvest.jpg>
I always get java error in the log fle:

2016-11-08 14:06:24,075 WARN - Raised exception while getting metadata
file : java.lang.NullPointerException
2016-11-08 14:06:24,075 WARN - Check with your administrator the error.
2016-11-08 14:06:24,075 ERROR -
java.lang.NullPointerException
  at
org.fao.oaipmh.responses.GetRecordResponse.build(GetRecordResponse.java:85)
  at
org.fao.oaipmh.responses.GetRecordResponse.<init>(GetRecordResponse.java:47)
  at
org.fao.oaipmh.requests.GetRecordRequest.execute(GetRecordRequest.java:84)
  at
org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.retrieveMetadata(Harvester.java:372)
  at
org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.addMetadata(Harvester.java:316)
  at
org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.align(Harvester.java:284)
  at
org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.harvest(Harvester.java:173)
  at
org.fao.geonet.kernel.harvest.harvester.oaipmh.OaiPmhHarvester.doHarvest(OaiPmhHarvester.java:154)
  at
org.fao.geonet.kernel.harvest.harvester.AbstractHarvester$HarvestWithIndexProcessor.process(AbstractHarvester.java:497)
  at
org.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:566)
  at
org.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:28)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
  at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

Holmgrimur_Helgason · November 11, 2016, 1:14pm

Hi
please anyone

I'm wondering if it is possible to use this URL for harvesting:
http://api.gbif.org/v1/oai-pmh/registry?verb=ListIdentifiers&set=installation:b494d108-871b-4dfa-bffe-c7949c7400e4&metadataPrefix=oai_dc

<http://osgeo-org.1560.x6.nabble.com/file/n5295335/harvest2.jpg>

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295335.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

Jorge_Samuel_Mendes · November 11, 2016, 1:18pm

Hi

It should be ok with the URL:

http://api.gbif.org/v1/oai-pmh/registry

I have been using GN for OAI harvesting without any problems

Jorge

On 11-11-16 14:14, hoddih wrote:

Hi
please anyone

I'm wondering if it is possible to use this URL for harvesting:
http://api.gbif.org/v1/oai-pmh/registry?verb=ListIdentifiers&set=installation:b494d108-871b-4dfa-bffe-c7949c7400e4&metadataPrefix=oai_dc

<http://osgeo-org.1560.x6.nabble.com/file/n5295335/harvest2.jpg>

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295335.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
ISRIC - World Soil Information
Post: PO box 353, 6700 AJ, Wageningen,
The Netherlands Visiting Address: Droevendaalsesteeg 3, 6708 PB
Wageningen (Bdg. 101),
Office: C.013
Office Phone: +31 (0) 317 4 83715
Mobile Phone: +31 (0) 613 9 06950
Skype: jorge.de.jesus (Jorge Samuel Mendes de Jesus)
OpenPGPKey: 0xA3D0065A

Holmgrimur_Helgason · November 11, 2016, 2:50pm

Thanks Jorge

It runs normally but I always get the messages that that I showed in the
first post in this thread.

total: 62
*
unretrievable: 62
*

And in the log file is some Java NullPointerException! Any clue?

p.s. My hair is starting be little gray

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295359.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

Jose_Garcia1 · November 11, 2016, 3:18pm

Hi

I've done a quick test using the same set as you are using with a 3.2.x
branch setup and get the 62 records harvested properly.

I see 62 records harvested. Which version are you using?

Regards,
Jose García

On Fri, Nov 11, 2016 at 4:15 PM, Jose Garcia <jose.garcia@anonymised.com> wrote:

Hi

I've done a quick test using the same set as you are using with a 3.2.x
branch setup and get the 62 records harvested properly.

!harvester-config.png|746x611

!result.png|806x225

Which version are you using?

Regards,
Jose García

On Fri, Nov 11, 2016 at 3:50 PM, hoddih <hoddi@anonymised.com> wrote:

Thanks Jorge

It runs normally but I always get the messages that that I showed in the
first post in this thread.

total: 62
*
unretrievable: 62
*

And in the log file is some Java NullPointerException! Any clue?

p.s. My hair is starting be little gray

--
View this message in context: http://osgeo-org.1560.x6.nabbl
e.com/OAI-PMH-Harvesting-tp5295006p5295359.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------
------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/project
s/geonetwork

--

*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
<https://twitter.com/geocat_bv>
<https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
environment before printing this email.*

--

*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
<https://twitter.com/geocat_bv>
<https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
environment before printing this email.*

Jorge_Samuel_Mendes · November 11, 2016, 3:58pm

Hi

I am not certain about GN log verbose (can't remember how to increase
verbose of logs), but I think the geonetwork log contains the requests
done during harvest. It is very likely that you have a URL list of the
request made to the OAI, copy a few requests and see what the OAI
returns or does, maybe there is something misconfigured

I think NullPointerException happens because you are not getting any
content. Looking at your last email, have you configured GN to use:

set --> installation:b494d108-871b-4dfa-bffe-c7949c7400e4
metadataPrefix --> oai_dc

Also, after setting the URL on the GN do you see available sets ???
(manual list of sets)
http://api.gbif.org/v1/oai-pmh/registry?verb=ListSets

and see the metadata options
http://api.gbif.org/v1/oai-pmh/registry?verb=ListMetadataFormats

Cheers
Jorge

On 11-11-16 15:50, hoddih wrote:

Thanks Jorge

It runs normally but I always get the messages that that I showed in the
first post in this thread.

total: 62
*
unretrievable: 62
*

And in the log file is some Java NullPointerException! Any clue?

p.s. My hair is starting be little gray

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295359.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
ISRIC - World Soil Information
Post: PO box 353, 6700 AJ, Wageningen,
The Netherlands Visiting Address: Droevendaalsesteeg 3, 6708 PB
Wageningen (Bdg. 101),
Office: C.013
Office Phone: +31 (0) 317 4 83715
Mobile Phone: +31 (0) 613 9 06950
Skype: jorge.de.jesus (Jorge Samuel Mendes de Jesus)
OpenPGPKey: 0xA3D0065A

Holmgrimur_Helgason · November 11, 2016, 4:01pm

Thanks a millions Jose

It works now.

I was doing it in a wrong way. I putted the"set" and "prefix" in the URL
not in the "search Filter"

You guys are great

On Fri, Nov 11, 2016 at 3:17 PM, Jose Garcia <jose.garcia@anonymised.com> wrote:

Hi

I've done a quick test using the same set as you are using with a 3.2.x
branch setup and get the 62 records harvested properly.

I see 62 records harvested. Which version are you using?

Regards,
Jose García

On Fri, Nov 11, 2016 at 4:15 PM, Jose Garcia <jose.garcia@anonymised.com>
wrote:

Hi

I've done a quick test using the same set as you are using with a 3.2.x
branch setup and get the 62 records harvested properly.

!result.png|806x225

!harvester-config.png|746x611

Which version are you using?

Regards,
Jose García

On Fri, Nov 11, 2016 at 3:50 PM, hoddih <hoddi@anonymised.com> wrote:

Thanks Jorge

It runs normally but I always get the messages that that I showed in the
first post in this thread.

total: 62
*
unretrievable: 62
*

And in the log file is some Java NullPointerException! Any clue?

p.s. My hair is starting be little gray

--
View this message in context: http://osgeo-org.1560.x6.nabbl
e.com/OAI-PMH-Harvesting-tp5295006p5295359.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.

------------------------------------------------------------
------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/project
s/geonetwork

--

*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
<https://twitter.com/geocat_bv>
<https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
environment before printing this email.*

--

*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
<https://twitter.com/geocat_bv>
<https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
environment before printing this email.*

--
Hólmgrímur Helgason (Hóddi)
CAFF - Data Manager
hoddi@anonymised.com
+354 462 3357

Holmgrimur_Helgason · November 14, 2016, 12:32pm

Thanks a millions

It works now.

I was doing it in a wrong way. I putted the"set" and "prefix" in the URL
not in the "search Filter"

You guys are great

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295660.html
Sent from the GeoNetwork users mailing list archive at Nabble.com.