[GeoNetwork-users] Efficient way to make changes against volumes of records

GeoNetwork GeoNetwork User
1

Hi,

I have GN (v2.6.4) running in conjunction with Postgresql.
Soon, I am expecting the need of importing millions of records into it.
Problem is, there might be a situation where I have to change some
record values at once over the entire record entries at some point
after the import, but to do that, it would be tedious to import the
entire records again after making the changes to the source record
files.

So my question is, is it possible to alter a metadata record directly
in the database through SQL ? If yes, how would I know where to change
without breaking the GN ? Are the specs of each database table
available somewhere?
Thank you so much for your help in advance.

Yoshi

2

Hi,

I've performed some batch processing of both GeoNetwork records (inside the
database) and on the file system (GeoServer). The specs of the tables are
quite easy to figure out by hand if you get a viewing tool like pgAdmin. Do
you want to make changes to some record fields like e.g. record user/owner,
privleges etc. or to the metadata itself? The metadata itself is usually
stored as XML, so once you get that "out" of the database and make the
changes you want, it's easy to write it back in.

In general, if you make changes just to the XML data, it's unlikely you'll
break something critical (at most you'll get a schema failure during
GeoNetwork WebUI visualization and/or serving of metadata through CSW). If
you alter record data like owner, id etc. you may have to be more careful
though.

Victor

On Tue, Jun 12, 2012 at 2:11 AM, Yoshi NA <tombitake1979@anonymised.com> wrote:

Hi,

I have GN (v2.6.4) running in conjunction with Postgresql.
Soon, I am expecting the need of importing millions of records into it.
Problem is, there might be a situation where I have to change some
record values at once over the entire record entries at some point
after the import, but to do that, it would be tedious to import the
entire records again after making the changes to the source record
files.

So my question is, is it possible to alter a metadata record directly
in the database through SQL ? If yes, how would I know where to change
without breaking the GN ? Are the specs of each database table
available somewhere?
Thank you so much for your help in advance.

Yoshi

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

3

Hi all,

Keep in mind that you must be consistent at anytime with the Lucene index...

Cheers,
Mathieu

On Tue, Jun 12, 2012 at 11:20 AM, Victor Epitropou <
vepitrop@anonymised.com> wrote:

Hi,

I've performed some batch processing of both GeoNetwork records (inside the
database) and on the file system (GeoServer). The specs of the tables are
quite easy to figure out by hand if you get a viewing tool like pgAdmin. Do
you want to make changes to some record fields like e.g. record user/owner,
privleges etc. or to the metadata itself? The metadata itself is usually
stored as XML, so once you get that "out" of the database and make the
changes you want, it's easy to write it back in.

In general, if you make changes just to the XML data, it's unlikely you'll
break something critical (at most you'll get a schema failure during
GeoNetwork WebUI visualization and/or serving of metadata through CSW). If
you alter record data like owner, id etc. you may have to be more careful
though.

Victor

On Tue, Jun 12, 2012 at 2:11 AM, Yoshi NA <tombitake1979@anonymised.com> wrote:

> Hi,
>
> I have GN (v2.6.4) running in conjunction with Postgresql.
> Soon, I am expecting the need of importing millions of records into it.
> Problem is, there might be a situation where I have to change some
> record values at once over the entire record entries at some point
> after the import, but to do that, it would be tedious to import the
> entire records again after making the changes to the source record
> files.
>
> So my question is, is it possible to alter a metadata record directly
> in the database through SQL ? If yes, how would I know where to change
> without breaking the GN ? Are the specs of each database table
> available somewhere?
> Thank you so much for your help in advance.
>
> Yoshi
>
>
>
------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

4

hi,

just to clarify Mathieu's comment:

sure, you can edit directly in the database like that. But after you've
done that, GeoNetwork's Lucene index will no longer be in sync with the
contents of the database (this means for example that if you made changes
to metadata, you won't be able to find those changes by using the search
function).

You can sync it again by logging in as an Administrator in GeoNetwork, and
run 'Re-build Lucene index" in the Admin page.

Hope this helps
Kind regards
Heikki Doeleman

On Tue, Jun 12, 2012 at 11:56 AM, Mathieu Coudert <mathieu.coudert@anonymised.com

wrote:

Hi all,

Keep in mind that you must be consistent at anytime with the Lucene
index...

Cheers,
Mathieu

On Tue, Jun 12, 2012 at 11:20 AM, Victor Epitropou <
vepitrop@anonymised.com> wrote:

> Hi,
>
> I've performed some batch processing of both GeoNetwork records (inside
the
> database) and on the file system (GeoServer). The specs of the tables are
> quite easy to figure out by hand if you get a viewing tool like pgAdmin.
Do
> you want to make changes to some record fields like e.g. record
user/owner,
> privleges etc. or to the metadata itself? The metadata itself is usually
> stored as XML, so once you get that "out" of the database and make the
> changes you want, it's easy to write it back in.
>
> In general, if you make changes just to the XML data, it's unlikely
you'll
> break something critical (at most you'll get a schema failure during
> GeoNetwork WebUI visualization and/or serving of metadata through CSW).
If
> you alter record data like owner, id etc. you may have to be more careful
> though.
>
> Victor
>
> On Tue, Jun 12, 2012 at 2:11 AM, Yoshi NA <tombitake1979@anonymised.com>
wrote:
>
> > Hi,
> >
> > I have GN (v2.6.4) running in conjunction with Postgresql.
> > Soon, I am expecting the need of importing millions of records into it.
> > Problem is, there might be a situation where I have to change some
> > record values at once over the entire record entries at some point
> > after the import, but to do that, it would be tedious to import the
> > entire records again after making the changes to the source record
> > files.
> >
> > So my question is, is it possible to alter a metadata record directly
> > in the database through SQL ? If yes, how would I know where to change
> > without breaking the GN ? Are the specs of each database table
> > available somewhere?
> > Thank you so much for your help in advance.
> >
> > Yoshi
> >
> >
> >
>
------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond.
Discussions
> > will include endpoint security, mobile security and the latest in
malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > GeoNetwork-users mailing list
> > GeoNetwork-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

5

Thanks, Victor, Mathieu.

It's the metadata itself I want to edit.
I did some changes, followed by Lucene index rebuild, and it looks it's working.

Yoshi

On Tue, Jun 12, 2012 at 6:56 PM, Mathieu Coudert
<mathieu.coudert@anonymised.com> wrote:

Hi all,

Keep in mind that you must be consistent at anytime with the Lucene index...

Cheers,
Mathieu

On Tue, Jun 12, 2012 at 11:20 AM, Victor Epitropou
<vepitrop@anonymised.com> wrote:

Hi,

I've performed some batch processing of both GeoNetwork records (inside
the
database) and on the file system (GeoServer). The specs of the tables are
quite easy to figure out by hand if you get a viewing tool like pgAdmin.
Do
you want to make changes to some record fields like e.g. record
user/owner,
privleges etc. or to the metadata itself? The metadata itself is usually
stored as XML, so once you get that "out" of the database and make the
changes you want, it's easy to write it back in.

In general, if you make changes just to the XML data, it's unlikely you'll
break something critical (at most you'll get a schema failure during
GeoNetwork WebUI visualization and/or serving of metadata through CSW). If
you alter record data like owner, id etc. you may have to be more careful
though.

Victor

On Tue, Jun 12, 2012 at 2:11 AM, Yoshi NA <tombitake1979@anonymised.com> wrote:

> Hi,
>
> I have GN (v2.6.4) running in conjunction with Postgresql.
> Soon, I am expecting the need of importing millions of records into it.
> Problem is, there might be a situation where I have to change some
> record values at once over the entire record entries at some point
> after the import, but to do that, it would be tedious to import the
> entire records again after making the changes to the source record
> files.
>
> So my question is, is it possible to alter a metadata record directly
> in the database through SQL ? If yes, how would I know where to change
> without breaking the GN ? Are the specs of each database table
> available somewhere?
> Thank you so much for your help in advance.
>
> Yoshi
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.
> Discussions
> will include endpoint security, mobile security and the latest in
> malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

6

Hi Heikki,
I did my change as you instructed and it seems working correctly.
Thanks a lot for the clarification.
Yoshi

On Wed, Jun 13, 2012 at 2:14 AM, heikki <tropicano@anonymised.com> wrote:

hi,

just to clarify Mathieu's comment:

sure, you can edit directly in the database like that. But after you've
done that, GeoNetwork's Lucene index will no longer be in sync with the
contents of the database (this means for example that if you made changes
to metadata, you won't be able to find those changes by using the search
function).

You can sync it again by logging in as an Administrator in GeoNetwork, and
run 'Re-build Lucene index" in the Admin page.

Hope this helps
Kind regards
Heikki Doeleman

On Tue, Jun 12, 2012 at 11:56 AM, Mathieu Coudert <mathieu.coudert@anonymised.com..

wrote:

Hi all,

Keep in mind that you must be consistent at anytime with the Lucene
index...

Cheers,
Mathieu

On Tue, Jun 12, 2012 at 11:20 AM, Victor Epitropou <
vepitrop@anonymised.com> wrote:

> Hi,
>
> I've performed some batch processing of both GeoNetwork records (inside
the
> database) and on the file system (GeoServer). The specs of the tables are
> quite easy to figure out by hand if you get a viewing tool like pgAdmin.
Do
> you want to make changes to some record fields like e.g. record
user/owner,
> privleges etc. or to the metadata itself? The metadata itself is usually
> stored as XML, so once you get that "out" of the database and make the
> changes you want, it's easy to write it back in.
>
> In general, if you make changes just to the XML data, it's unlikely
you'll
> break something critical (at most you'll get a schema failure during
> GeoNetwork WebUI visualization and/or serving of metadata through CSW).
If
> you alter record data like owner, id etc. you may have to be more careful
> though.
>
> Victor
>
> On Tue, Jun 12, 2012 at 2:11 AM, Yoshi NA <tombitake1979@anonymised.com>
wrote:
>
> > Hi,
> >
> > I have GN (v2.6.4) running in conjunction with Postgresql.
> > Soon, I am expecting the need of importing millions of records into it.
> > Problem is, there might be a situation where I have to change some
> > record values at once over the entire record entries at some point
> > after the import, but to do that, it would be tedious to import the
> > entire records again after making the changes to the source record
> > files.
> >
> > So my question is, is it possible to alter a metadata record directly
> > in the database through SQL ? If yes, how would I know where to change
> > without breaking the GN ? Are the specs of each database table
> > available somewhere?
> > Thank you so much for your help in advance.
> >
> > Yoshi
> >
> >
> >
>
------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond.
Discussions
> > will include endpoint security, mobile security and the latest in
malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > GeoNetwork-users mailing list
> > GeoNetwork-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> GeoNetwork-users mailing list
> GeoNetwork-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork