[GeoNetwork-users] selective harvesting

Terry_Rankine · April 26, 2012, 1:52am

Hi Guys

I am working with some guys who have a multi node mess, and we wish to start to 'untangle' some of the 'node graph' into a more structured hierarchy.

Before we get to that - I have some harvesting questions, as I am a little confused about the current setup, and want to make sure I have it right.

Scenario:
1. node3 has a static 19115 record
2. node2->harvest->node3.
3. node1 ->harvest-> node2

Questions:
1. Origin of harvest record? will the record make it to node1? what is the origin listed as? is the 'path taken' recorded? Do the UUID's change? I appreciate that the internal GN ID's will, but what about the UUIDs?
2. Update propagation. If node2 is allowing harvested records to be altered. when node1 harvests, does it get the original or the updated record? does node1 know the record was altered along the way?
3. What happens if some 'turkey' makes node3 harvest node1...... remember - I told you this was a multi node mess....
4. When node2 is down and node1 harvests - what happens? does it check the original location of the records or only node2

Regards,
Terry Rankine

andrew_walsh · April 26, 2012, 2:29am

Hi Terry,

A while back I did some experimenting with the GN harvester with 3 nodes harvesting off
each other so I could answer some of your questions as below.

Andrew

----- Original Message ----- From: <Terry.Rankine@anonymised.com>
To: <geonetwork-users@lists.sourceforge.net>
Sent: Thursday, April 26, 2012 11:52 AM
Subject: [GeoNetwork-users] selective harvesting

Hi Guys

I am working with some guys who have a multi node mess, and we wish to start to 'untangle' some of the 'node graph' into a more structured hierarchy.

Before we get to that - I have some harvesting questions, as I am a little confused about the current setup, and want to make sure I have it right.

Scenario:
1. node3 has a static 19115 record
2. node2->harvest->node3.
3. node1 ->harvest-> node2

Questions:
1. Origin of harvest record? will the record make it to node1? what is the origin listed as? is the 'path taken' recorded? Do the UUID's change? I appreciate that the internal GN ID's will, but what about the UUIDs?

- Record should make it to node1,
- Origin should be listed in 'source' column in the metadata table should be the uuid (GN siteId)
i.e node 3 in your scenario.
- Don't think 'path taken' would be known

2. Update propagation. If node2 is allowing harvested records to be altered. when node1 harvests, does it get the original or the updated record? does node1 know the record was altered along the way?

Node 2 altering the harvest records sounds like a bad idea, this would muck up the
integrity of the harvest process. No idea what would happen at node 1.

3. What happens if some 'turkey' makes node3 harvest node1...... remember - I told you this was a multi node mess....

Then it gets more complicated, as you say a mess.

4. When node2 is down and node1 harvests - what happens? does it check the original location of the records or only node2

Probably the harvest from node1 to node2 would fail, I suspect node1 wouldn't check the source node3
as node1 has no knowlege of the harvest chain.

Regards,
Terry Rankine

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Terry_Rankine · April 26, 2012, 2:45am

This comes very close to all my answers
http://www.geonetwork-opensource.org/manuals/trunk/users/admin/harvesting/index.html

but doesn't cover the origin / history of the harvest, or the hacking of harvested content.... (yes I realise its a little evil...)

Terry

________________________________________
From: Terry.Rankine@anonymised.com [Terry.Rankine@anonymised.com]
Sent: Thursday, 26 April 2012 9:52 AM
To: geonetwork-users@lists.sourceforge.net
Subject: [ExternalEmail] [GeoNetwork-users] selective harvesting

Hi Guys

I am working with some guys who have a multi node mess, and we wish to start to 'untangle' some of the 'node graph' into a more structured hierarchy.

Before we get to that - I have some harvesting questions, as I am a little confused about the current setup, and want to make sure I have it right.

Scenario:
1. node3 has a static 19115 record
2. node2->harvest->node3.
3. node1 ->harvest-> node2

Questions:
1. Origin of harvest record? will the record make it to node1? what is the origin listed as? is the 'path taken' recorded? Do the UUID's change? I appreciate that the internal GN ID's will, but what about the UUIDs?
2. Update propagation. If node2 is allowing harvested records to be altered. when node1 harvests, does it get the original or the updated record? does node1 know the record was altered along the way?
3. What happens if some 'turkey' makes node3 harvest node1...... remember - I told you this was a multi node mess....
4. When node2 is down and node1 harvests - what happens? does it check the original location of the records or only node2

Regards,
Terry Rankine

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Heikki_Doeleman · April 30, 2012, 12:12pm

hi Terry,

when node 3 harvests node 1, you'll have a circular harvesting setup, which
is prone to fun effects as described here:
http://osgeo-org.1560.n6.nabble.com/Harvesting-delete-logic-td3877120.html.

Kind regards
Heikki doeleman

On Thu, Apr 26, 2012 at 4:45 AM, <Terry.Rankine@anonymised.com> wrote:

This comes very close to all my answers

http://www.geonetwork-opensource.org/manuals/trunk/users/admin/harvesting/index.html

but doesn't cover the origin / history of the harvest, or the hacking of
harvested content.... (yes I realise its a little evil...)

Terry

________________________________________
From: Terry.Rankine@anonymised.com [Terry.Rankine@anonymised.com]
Sent: Thursday, 26 April 2012 9:52 AM
To: geonetwork-users@lists.sourceforge.net
Subject: [ExternalEmail] [GeoNetwork-users] selective harvesting

Hi Guys

I am working with some guys who have a multi node mess, and we wish to
start to 'untangle' some of the 'node graph' into a more structured
hierarchy.

Before we get to that - I have some harvesting questions, as I am a little
confused about the current setup, and want to make sure I have it right.

Scenario:
1. node3 has a static 19115 record
2. node2->harvest->node3.
3. node1 ->harvest-> node2

Questions:
1. Origin of harvest record? will the record make it to node1? what is the
origin listed as? is the 'path taken' recorded? Do the UUID's change? I
appreciate that the internal GN ID's will, but what about the UUIDs?
2. Update propagation. If node2 is allowing harvested records to be
altered. when node1 harvests, does it get the original or the updated
record? does node1 know the record was altered along the way?
3. What happens if some 'turkey' makes node3 harvest node1...... remember
- I told you this was a multi node mess....
4. When node2 is down and node1 harvests - what happens? does it check the
original location of the records or only node2

Regards,
Terry Rankine

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork