[GeoNetwork-devel] harvesting from 2.10 to 2.6

Hi All,

I need to harvest records from GN 2.10 to GN 2.6. I followed the harvesting docs and set up a GN2.10 server and ran into some issues while harvesting. following are excerpts from log file. What do I need to take care while harvesting from higher version to lower version?

2014-05-29 11:44:25,257 INFO [jeeves.service] - → writing xml for : xml.harvesting.run
2014-05-29 11:44:25,257 DEBUG [jeeves.service] - Service xml is :

1059

2014-05-29 11:44:25,257 INFO [jeeves.service] - → output ended for : xml.harvesting.run
2014-05-29 11:44:25,257 INFO [jeeves.service] - → dispatch ended for : xml.harvesting.run
2014-05-29 11:44:30,195 WARN [geonetwork.harvester] - Raised exception when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception while harvesting from : gdeserver12 (CswHarvester)
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Class : OperationAbortedEx
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Message : Raised exception when searching

Thanks a lot

Regards

Ameet

Hi Ameet

Just to be sure about the harvesting direction: you’re harvesting with CSW protocol metadata from a 2.10.X server into a 2.6.X server?

I have done a similar test with GeoNetwork instances running locally, but could not reproduce it. Can you check the tomcat log? probably contains more details about the error. The previous snippet doesn’t clarifies the issue.

Regards,
Jose García

···

On Thu, May 29, 2014 at 2:19 PM, Ameet Chaudhari <ameet.chaudhari@anonymised.com> wrote:

Hi All,

I need to harvest records from GN 2.10 to GN 2.6. I followed the harvesting docs and set up a GN2.10 server and ran into some issues while harvesting. following are excerpts from log file. What do I need to take care while harvesting from higher version to lower version?

2014-05-29 11:44:25,257 INFO [jeeves.service] - → writing xml for : xml.harvesting.run
2014-05-29 11:44:25,257 DEBUG [jeeves.service] - Service xml is :

1059

2014-05-29 11:44:25,257 INFO [jeeves.service] - → output ended for : xml.harvesting.run
2014-05-29 11:44:25,257 INFO [jeeves.service] - → dispatch ended for : xml.harvesting.run
2014-05-29 11:44:30,195 WARN [geonetwork.harvester] - Raised exception when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception while harvesting from : gdeserver12 (CswHarvester)
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Class : OperationAbortedEx
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Message : Raised exception when searching

Thanks a lot

Regards

Ameet


Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet


GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork


GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on GeoServer and GeoNetwork. Visit http://geocat.net for details.


Jose García
GeoCat bv
Veenderweg 13
6721 WD Bennekom
The Netherlands
http://GeoCat.net

Hi Jose,

here are the detail logs after I enabled more detailed logging.
Note:
1. I'm able to Add and Activate 2.10.x server in Harvesting
management. While doing a Run, this exception occurs.
2. If I use geonetwork url for the same server (instead of CSW based),
then harvesting works but we want it for CSW ISO Profile 2.0

following are log file contents of server to which I'm harvesting
(2.6.5) [looks like occurring during search, like something is
blocking but there is no firewall turned ON on either servers I
checked]:

2014-05-30 13:20:03,968 INFO [jeeves.service] - -> output ended
for : xml.harvesting.get
2014-05-30 13:20:03,968 INFO [jeeves.service] - -> dispatch ended
for : xml.harvesting.get
2014-05-30 13:20:05,843 WARN [geonetwork.harvester] - Raised
exception when searching : java.lang.NullPointerException
2014-05-30 13:20:05,843 DEBUG [geonetwork.harvester] - Raised
exception when searching
2014-05-30 13:20:05,843 DEBUG [geonetwork.harvester] - Changing to CSW POST
2014-05-30 13:20:05,843 INFO [geonetwork.harvester] - Searching on :
gdeserver12 (1..21)
2014-05-30 13:20:10,328 WARN [geonetwork.harvester] - Raised
exception when searching : java.lang.NullPointerException
2014-05-30 13:20:10,328 DEBUG [geonetwork.index] - Closing Writer from
endIndexGroup
2014-05-30 13:20:10,328 INFO [geonetwork.index] - Closing
Index_writer, ref _count 0 ram in use 0 docs buffered 0
2014-05-30 13:20:10,328 WARN [geonetwork.harvester] - Raised
exception while harvesting from : gdeserver12 (CswHarvester)
2014-05-30 13:20:10,328 WARN [geonetwork.harvester] - (C) Class :
OperationAbortedEx
2014-05-30 13:20:10,328 WARN [geonetwork.harvester] - (C) Message :
Raised exception when searching
2014-05-30 13:20:10,328 DEBUG [jeeves.dbmspool] - Aborting
jeeves.resources.dbms.Dbms@anonymised.com
2014-05-30 13:20:32,296 INFO [jeeves.request] -

2014-05-30 13:20:32,296 INFO [jeeves.request] - HTML Request (from
127.0.0.1) : /geonetwork/srv/en/xml.harvesting.get
2014-05-30 13:20:32,296 DEBUG [jeeves.request] - Method : POST
2014-05-30 13:20:32,296 DEBUG [jeeves.request] - Content type :
application/xml; charset=UTF-8
2014-05-30 13:20:32,296 DEBUG [jeeves.request] - Accept :
text/javascript, text/html, application/xml, text/xml, */*
2014-05-30 13:20:32,296 DEBUG [jeeves.request] - Session id is
E28A0D3A3879BEA5A082C28F4EE46491
2014-05-30 13:20:32,312 INFO [jeeves.service] - Dispatching :
xml.harvesting.get
2014-05-30 13:20:32,312 DEBUG [jeeves.service] - -> no input parameters
2014-05-30 13:20:32,312 DEBUG [jeeves.engine] -
TransformerFactoryFactory:
de.fzi.dbs.xml.transform.CachingTransformerFactory
2014-05-30 13:20:32,312 DEBUG [jeeves.engine] -
TransformerFactoryFactory: produces transformer implementation
net.sf.saxon.IdentityTransformer
2014-05-30 13:20:32,312 INFO [jeeves.service] - -> dispatching to
output for : xml.harvesting.get
2014-05-30 13:20:32,312 INFO [jeeves.service] - -> writing xml
for : xml.harvesting.get
2014-05-30 13:20:32,312 DEBUG [jeeves.service] - Service xml is :
<nodes>
  <node id="916" type="csw">
    <site>
      <name>gdeserver12</name>
      <uuid>e858cfd5-66be-4b8e-8075-099ccb616d56</uuid>
      <account>
        <use>false</use>
        <username />
        <password />
      </account>
      <capabilitiesUrl>http://192.168.1.222:8081/geonetwork/srv/eng/csw?request=GetCapabilities&amp;service=CSW&amp;acceptVersions=2.0.2&lt;/capabilitiesUrl&gt;
      <icon>default.gif</icon>
    </site>
    <content>
      <validate>false</validate>
      <importxslt>none</importxslt>
    </content>
    <options>
      <every>1440</every>
      <oneRunOnly>false</oneRunOnly>
      <status>active</status>
    </options>
    <searches />
    <owner>
      <user>1</user>
      <group>2</group>
    </owner>
    <privileges>
      <group id="1">
        <operation name="view" />
        <operation name="dynamic" />
        <operation name="featured" />
      </group>
    </privileges>
    <info>
      <lastRun />
      <running>false</running>
    </info>
    <error id="operation-aborted">
      <message>Raised exception when searching</message>
      <class>OperationAbortedEx</class>
      <stack>
        <at class="org.fao.geonet.kernel.harvest.harvester.csw.Harvester"
file="Harvester.java" line="440" method="doSearch" />
        <at class="org.fao.geonet.kernel.harvest.harvester.csw.Harvester"
file="Harvester.java" line="203" method="search" />
        <at class="org.fao.geonet.kernel.harvest.harvester.csw.Harvester"
file="Harvester.java" line="93" method="harvest" />
        <at class="org.fao.geonet.kernel.harvest.harvester.csw.CswHarvester"
file="CswHarvester.java" line="213" method="doHarvest" />
        <at class="org.fao.geonet.kernel.harvest.harvester.AbstractHarvester$HarvestWithIndexProcessor"
file="AbstractHarvester.java" line="379" method="process" />
        <at class="org.fao.geonet.kernel.MetadataIndexerProcessor"
file="MetadataIndexerProcessor.java" line="39"
method="processWithFastIndexing" />
        <at class="org.fao.geonet.kernel.harvest.harvester.AbstractHarvester"
file="AbstractHarvester.java" line="406" method="harvest" />
        <at class="org.fao.geonet.kernel.harvest.harvester.Executor"
file="Executor.java" line="87" method="run" />
      </stack>
      <object>java.lang.NullPointerException</object>
    </error>
  </node>
</nodes>
2014-05-30 13:20:32,312 INFO [jeeves.service] - -> output ended
for : xml.harvesting.get
2014-05-30 13:20:32,312 INFO [jeeves.service] - -> dispatch ended
for : xml.harvesting.get

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

and following are log contents of the server from which I'm harvesting (2.10.3):

2014-05-30 13:36:41,688 INFO [jeeves.request] - HTML Request (from
192.168.1.152) : /geonetwork/srv/eng/csw
2014-05-30 13:36:41,688 INFO [jeeves.service] - Dispatching : csw
2014-05-30 13:36:41,688 INFO [jeeves.webapp.csw] - Received:
<request>
  <service>CSW</service>
  <acceptVersions>2.0.2</acceptVersions>
  <request>GetCapabilities</request>
</request>
2014-05-30 13:36:41,688 INFO [jeeves.webapp.csw] - Dispatching
operation : GetCapabilities
2014-05-30 13:36:41,688 DEBUG [geonetwork.csw] - Handling property
names '[keyword]' with max records of 1000
2014-05-30 13:36:41,688 DEBUG [geonetwork.index] - Ask for new reader
2014-05-30 13:36:41,688 DEBUG [jeeves.resources] - Opening: main-db in
thread: 98
2014-05-30 13:36:41,688 DEBUG [jeeves.resources] - Null resource,
opening a new one from the resource provider.
2014-05-30 13:36:41,688 DEBUG [jeeves.resources] - Open connection: 24091185
2014-05-30 13:36:41,688 DEBUG [jeeves.resources] - Returning:
jeeves.resources.dbms.Dbms@anonymised.com
2014-05-30 13:36:41,688 DEBUG [geonetwork.search] - Sorting by :
[relevance,true]
2014-05-30 13:36:41,688 DEBUG [geonetwork.search] - Build summary: false
2014-05-30 13:36:41,688 DEBUG [geonetwork.search] - Setting up the TFC
with numHits 1000
2014-05-30 13:36:41,688 DEBUG [geonetwork.search] - Lucene query: _op0:1
2014-05-30 13:36:41,688 DEBUG [geonetwork.search] - Get top docs from
0 ... 1000 (total: 251)
2014-05-30 13:36:41,720 DEBUG [jeeves.resources] - Opening: main-db in
thread: 98
2014-05-30 13:36:41,720 DEBUG [jeeves.resources] - Returning:
jeeves.resources.dbms.Dbms@anonymised.com
2014-05-30 13:36:41,720 DEBUG [jeeves.dbms.select] - Query: SELECT *
FROM Languages WHERE isInspire='y'
2014-05-30 13:36:41,720 DEBUG [jeeves.dbms.select] - Connection: 24091185
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Found 2 records
in 0.015 secs
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Query: SELECT *
FROM Languages WHERE isDefault='y'
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Connection: 24091185
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Found 1 records in 0.0 secs
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Query: SELECT *
FROM CswServerCapabilitiesInfo WHERE langId = ?
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Connection: 24091185
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Args : eng
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Found 4 records in 0.0 secs
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Query: SELECT *
FROM Users WHERE id = ?
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Connection: 24091185
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Args : -1
2014-05-30 13:36:41,735 DEBUG [jeeves.dbms.select] - Found 0 records in 0.0 secs
2014-05-30 13:36:41,735 DEBUG [jeeves.resources] - Close connection: 24091185
2014-05-30 13:36:41,735 INFO [jeeves.service] - -> dispatching to
output for : csw
2014-05-30 13:36:41,735 INFO [jeeves.service] - -> writing xml for : csw
2014-05-30 13:36:41,735 INFO [jeeves.service] - -> output ended for : csw
2014-05-30 13:36:41,735 INFO [jeeves.service] - -> dispatch ended for : csw
2014-05-30 13:36:58,629 INFO [geonetwork.lucene] - Done running
PurgeExpiredSearchersTask. 1 versions still cached.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks a lot in advance for going through longish logs.. :slight_smile:

Regards
Ameet

On 5/30/14, Ameet Chaudhari <ameet.chaudhari@anonymised.com> wrote:

Thanks Jose,

Just to be sure about the harvesting direction: you're harvesting with CSW

protocol metadata from a 2.10.X server into a 2.6.X server?

Yes I am trying to harvest with CSW ISO Profile 2.0 from 2.10.x server into
a 2.6.x

I have done a similar test with GeoNetwork instances running locally, but

could not reproduce it. Can you check the tomcat log? probably contains
more details about the error. The previous snippet doesn't clarifies the
issue.

I'll check again and try to give more info on the error I am getting.
(Outside the Harvester management, I checked the csw URL of 2.10.x server
and I'm getting the appropriate response). I'll get back with more info.

Ameet

On Thu, May 29, 2014 at 2:19 PM, Ameet Chaudhari
<ameet.chaudhari@anonymised.com>
wrote:

Hi All,

I need to harvest records from GN 2.10 to GN 2.6. I followed the
harvesting docs and set up a GN2.10 server and ran into some issues
while
harvesting. following are excerpts from log file. What do I need to take
care while harvesting from higher version to lower version?

2014-05-29 11:44:25,257 INFO [jeeves.service] - -> writing xml for
: xml.harvesting.run
2014-05-29 11:44:25,257 DEBUG [jeeves.service] - Service xml is :
<response>
  <id status="ok">1059</id>
</response>
2014-05-29 11:44:25,257 INFO [jeeves.service] - -> output ended for
:
xml.harvesting.run
2014-05-29 11:44:25,257 INFO [jeeves.service] - -> dispatch ended for
:
xml.harvesting.run
2014-05-29 11:44:30,195 WARN [geonetwork.harvester] - Raised exception
when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception
when searching : java.lang.NullPointerException
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - Raised exception
while harvesting from : gdeserver12 (CswHarvester)
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Class
: OperationAbortedEx
2014-05-29 11:44:34,571 WARN [geonetwork.harvester] - (coffee) Message
: Raised exception when searching

Thanks a lot
Regards
Ameet

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

--

* GeoCat Bridge for ArcGIS allows instant publishing of data and metadata
on GeoServer and GeoNetwork. Visit http://geocat.net
<http://geocat.net/&gt; for details. _________________________Jose
GarcíaGeoCat bvVeenderweg 13 6721 WD BennekomThe
Netherlandshttp://GeoCat.net/> *

--
Systems Programmer
kCube Consultancy Services

--
Systems Programmer
kCube Consultancy Services