[GRASS-dev] <gsoc2014> <grass metadata> <week 6>

GRASS GIS GRASS GIS developer
1

Hi all,
Below is my weekly report:

  • worked on GUI editor
  • implemented generator of metadata items (control text, label, button) in main editor
  • fixed some bugs in editor- implemented interface for parse information from jinja template
  • added additional information about metadata to jinja template.


Continue with working on GUI editor:

  • to finish version 1.1 (by wikipage) of g.gui.editor
  • to connect GUI editor (middle panel) with OWSLib and Jinja template

no

Best,
Matej

2

On 29 June 2014 21:57, Matej Krejci <matejkrejci@gmail.com> wrote:

Hi all,

Hi Matej

to finish version 1.1 (by wikipage) of g.gui.editor

I think that g.gui.editor it is not so clear, could you rename it to
g.gui.metadata or g.gui.metaeditor?

to connect GUI editor (middle panel) with OWSLib and Jinja template

Best,
Matej

--
ciao
Luca

http://gis.cri.fmach.it/delucchi/
www.lucadelu.org

3

Hi,

2014-06-30 9:37 GMT+02:00 Luca Delucchi <lucadeluge@gmail.com>:

I think that g.gui.editor it is not so clear, could you rename it to
g.gui.metadata or g.gui.metaeditor?

right, `g.gui.metadata` seems to be the most appropriate, Martin

--
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa

4

g.gui.metadata is the most clear to me.

Doug

···

On Mon, Jun 30, 2014 at 4:30 AM, Martin Landa <landa.martin@gmail.com> wrote:

Hi,

2014-06-30 9:37 GMT+02:00 Luca Delucchi <lucadeluge@gmail.com>:

I think that g.gui.editor it is not so clear, could you rename it to
g.gui.metadata or g.gui.metaeditor?

right, g.gui.metadata seems to be the most appropriate, Martin

–
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

–

Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb@fws.gov

The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.

5

Hi,
2014-06-30 9:37 GMT+02:00 Luca Delucchi <lucadeluge@gmail.com>:

On 29 June 2014 21:57, Matej Krejci <matejkrejci@gmail.com> wrote:
> Hi all,

Hi Matej

>
> to finish version 1.1 (by wikipage) of g.gui.editor
>

I think that g.gui.editor it is not so clear, could you rename it to
g.gui.metadata or g.gui.metaeditor?

Thanks for notice. The right/current name is g.gui.metadata. I just wrote

name wrongly in weekly report. Sorry for confusion. However, the final name
of module is open.
Matej

6

Hi Matej,

Thanks for working on metadata support in GRASS!

I am very much looking forward to having the modules you are developing in core. The documentation about your work which you have on the Wiki looks very, very promising!

I was wondering, did you consider storing keyword lists, points of contacts, and so on in a central (SQLite) database (comparable to what the t.*-modules do with time series)? For organizations it would be useful to have tools which produce consistent meta-data (same set of key-words used by all members of staff…). Furthermore, efficiency might be improved by having e.g. table with contact persons within an organization (where all contact details are saved) one can pick from in a combo-box.

Such a DB should probably be accessible from all locations and mapsets…

Just an idea, and I do not know how much work that would be…

BTW, is the metadata publishing functionality based on owslib + pycsw, which you mention on your wiki-page, somehow related to GeoNetwork?

Cheers and thanks again for your excellent work!

Stefan

7

Hi Stefan,

2014-07-02 9:35 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:

Hi Matej,
I was wondering, did you consider storing keyword lists, points of contacts, and > so on in a central (SQLite) database (comparable to what the t.*-modules do with > time series)? For organizations it would be useful to have tools which produce
consistent meta-data (same set of key-words used by all members of staff…).

Current possibility how to produce consistent metadata (same set of key) is by modifcation jinja template [1]. Values in jinja template are represented by Python objects. You can just replace these objects by desire values for creating pattern. What do you think?

Furthermore, efficiency might be improved by having e.g. table with contact
pers ons within an organization (where all contact details are saved) one can
pick from in a combo-box.

Such a DB should probably be accessible from all locations and mapsets…

It makes sense to me. This task should not be problem, but is more than essential. Probably I’ll do it after the GSOC term.

BTW, is the metadata publishing functionality based on owslib + pycsw, which
you mention on your wiki-page, somehow related to GeoNetwork?
No, I dont know this project in detail…

Thank you for useful hints!
Cheers
Matej

[1] https://svn.osgeo.org/grass/sandbox/krejcmat/src/data/grassInspireTemplate.xml

8

Hi Matej,

Thanks for your reply.

Metadata is a current topic I my organization at the moment, so your work comes exactly at the right time for me. Here in NINA people are using many different kinds of GIS software and we are considering introducing GeoNetwork as a central metadata storage solution, which makes metadata management to some extend independent from the GIS software used. Among public and environmental organisations, GeoNetwork is quite popular for metadata management (amongst others Mapping authorities from Netherlands, Switzerland, and the Nordic countries are using it, see also: http://geonetwork-opensource.org/gallery/gallery.html).

I have no real personal experience with GeoNetwor (yet), but from what I read in the documentation I could imagine two ways how GRASS metadata could make their way into such a central metadata catalogue:

  1. XML import (+ this option is in place when GRASS metadata is written to XML, - import requires manual labor)

  2. Metadata harvesting (GeoNetwork can “harvest” metadata from other CSWs, so maybe pycsw already provides automatic an interface in that direction(?). That would be great!)

When it comes to consistent metadata, I guess e.g. lists with values for metadata fields will in practice have to be more or less dynamic (as new staff member are being employed, new topics for spatial analysis arise (requiring new keywords), new types of data being developed and so on). Not sure if manipulating the templates is convenient enough for average users (seems to be mainly suitable for users with some degree of programming experience). However, good to know that providing a custom set of values is already possible through the templates.

If you start working on a DB for metadata related content, feel free to let me know if I can support you by any means (yet I will be on leave for almost the rest of the year, which means I can spend only a limited amount of time).

However, I had a second look on GeoNetwork in this regard, and GeoNetwork is (naturally) using a database for metadata management (one option here is PostgreSQL). If it would be possible to just connect to that same DB (e.g. for fetching and storing key words, data on contact persons…), that would be really, really great, as this ensures that metadata produced in GRASS are 100% compliant to metadata standards used in GeoNetwork (if GRASS users also use GeoNetwork).

Anyway, Rome was not built in a day and I am 100% sure that your work will be very, very useful already from what you planned for GSoC.

All the best,

Stefan

9

Hi Stefan,
On Thu, Jul 3, 2014 at 9:46 AM, Blumentrath, Stefan <Stefan.Blumentrath@nina.no> wrote:

Hi Matej,

Thanks for your reply.

Metadata is a current topic I my organization at the moment, so your work comes exactly at the right time for me. Here in NINA people are using many different kinds of GIS software and we are considering introducing GeoNetwork as a central metadata storage solution, which makes metadata management to some extend independent from the GIS software used. Among public and environmental organisations, GeoNetwork is quite popular for metadata management (amongst others Mapping authorities from Netherlands, Switzerland, and the Nordic countries are using it, see also: http://geonetwork-opensource.org/gallery/gallery.html).

I have no real personal experience with GeoNetwor (yet), but from what I read in the documentation I could imagine two ways how GRASS metadata could make their way into such a central metadata catalogue:

  1. XML import (+ this option is in place when GRASS metadata is written to XML, - import requires manual labor)

  2. Metadata harvesting (GeoNetwork can “harvest” metadata from other CSWs, so maybe pycsw already provides automatic an interface in that direction(?). That would be great!)

pycsw is a CSW server that allows to harvest information from other OGC services, as well as Geonetwork. This latter offers also the editor, while pycsw doesn’t, but is used internally by other applications which offer full metadata management (and a GUI), such as CKAN, GeoNode, etc. and the idea is to add GRASS to this list as well :slight_smile:
If you want to stick with geonetwork, you can harvest the metadata from the pycsw, there’s no need of manual work.

When it comes to consistent metadata, I guess e.g. lists with values for metadata fields will in practice have to be more or less dynamic (as new staff member are being employed, new topics for spatial analysis arise (requiring new keywords), new types of data being developed and so on). Not sure if manipulating the templates is convenient enough for average users (seems to be mainly suitable for users with some degree of programming experience). However, good to know that providing a custom set of values is already possible through the templates.

If you start working on a DB for metadata related content, feel free to let me know if I can support you by any means (yet I will be on leave for almost the rest of the year, which means I can spend only a limited amount of time).

The idea is to allow user to generate a custom template using the gui, that can be reused for any metadata generated afterwards. Your idea of a database deserves consideration as well

Thanks
Madi

However, I had a second look on GeoNetwork in this regard, and GeoNetwork is (naturally) using a database for metadata management (one option here is PostgreSQL). If it would be possible to just connect to that same DB (e.g. for fetching and storing key words, data on contact persons…), that would be really, really great, as this ensures that metadata produced in GRASS are 100% compliant to metadata standards used in GeoNetwork (if GRASS users also use GeoNetwork).

Anyway, Rome was not built in a day and I am 100% sure that your work will be very, very useful already from what you planned for GSoC.

All the best,

Stefan

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

–
Best regards,

Dr. Margherita DI LEO
Scientific / technical project officer

European Commission - DG JRC
Institute for Environment and Sustainability (IES)
Via Fermi, 2749
I-21027 Ispra (VA) - Italy - TP 261

Tel. +39 0332 78 3600
margherita.di-leo@jrc.ec.europa.eu

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.

Hi Matej,

Thanks for your reply.

Metadata is a current topic I my organization at the moment, so your work comes exactly at the right time for me. Here in NINA people are using many different kinds of GIS software and we are considering introducing GeoNetwork as a central metadata storage solution, which makes metadata management to some extend independent from the GIS software used. Among public and environmental organisations, GeoNetwork is quite popular for metadata management (amongst others Mapping authorities from Netherlands, Switzerland, and the Nordic countries are using it, see also: http://geonetwork-opensource.org/gallery/gallery.html).

I have no real personal experience with GeoNetwor (yet), but from what I read in the documentation I could imagine two ways how GRASS metadata could make their way into such a central metadata catalogue:

  1. XML import (+ this option is in place when GRASS metadata is written to XML, - import requires manual labor)

  2. Metadata harvesting (GeoNetwork can “harvest” metadata from other CSWs, so maybe pycsw already provides automatic an interface in that direction(?). That would be great!)

When it comes to consistent metadata, I guess e.g. lists with values for metadata fields will in practice have to be more or less dynamic (as new staff member are being employed, new topics for spatial analysis arise (requiring new keywords), new types of data being developed and so on). Not sure if manipulating the templates is convenient enough for average users (seems to be mainly suitable for users with some degree of programming experience). However, good to know that providing a custom set of values is already possible through the templates.

If you start working on a DB for metadata related content, feel free to let me know if I can support you by any means (yet I will be on leave for almost the rest of the year, which means I can spend only a limited amount of time).

However, I had a second look on GeoNetwork in this regard, and GeoNetwork is (naturally) using a database for metadata management (one option here is PostgreSQL). If it would be possible to just connect to that same DB (e.g. for fetching and storing key words, data on contact persons…), that would be really, really great, as this ensures that metadata produced in GRASS are 100% compliant to metadata standards used in GeoNetwork (if GRASS users also use GeoNetwork).

Anyway, Rome was not built in a day and I am 100% sure that your work will be very, very useful already from what you planned for GSoC.

All the best,

Stefan

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

10

Hei Madi,

Thanks for your clarification.

Very interesting! I see I have to take a closer look on pycsw and it`s front-ends (as a possible alternative to GeoNetwork).

The pycsw documentation [1] says that also pycsw Metadata repositories are set up with a database backend (SQLite, PostgreSQL, and even PostGIS support). However, integration with data portal solutions like e.g. GeoNode or Open Data Catalogue is unfortunately “read-only” meaning that meta-data can be only queried [2].

Would be nice if metadata from GRASS could somehow go directly into a geodata portal (and that the other way around e.g. people, keywords … could be fetched from there)… (BTW, in the proprietary world such a functionality requires yet another extension: https://www.geocat.net/bridge/)

For organization with a geodata portal solution it is maybe possible to just sync pycsw`s DB with a DB from e.g. GeoNode or so, if one wants to have that consistent…

If you allow me two more questions in this regard:

Will pycsw become a dependency for GRASS (with metadata support)?

And will it be necessary, that pycsw runs on the same computer as GRASS (meaning also one pycsw for each GRASS installation) or could that somehow be centralized (in other words that GRASS connects to a central pycsw)?

Cheers

Stefan

[1] http://docs.pycsw.org/en/1.8.3/administration.html#metadata-repository-setup

[2] http://docs.pycsw.org/en/1.8.3/geonode.html

···

Hi Matej,

Thanks for your reply.

Metadata is a current topic I my organization at the moment, so your work comes exactly at the right time for me. Here in NINA people are using many different kinds of GIS software and we are considering introducing GeoNetwork as a central metadata storage solution, which makes metadata management to some extend independent from the GIS software used. Among public and environmental organisations, GeoNetwork is quite popular for metadata management (amongst others Mapping authorities from Netherlands, Switzerland, and the Nordic countries are using it, see also: http://geonetwork-opensource.org/gallery/gallery.html).

I have no real personal experience with GeoNetwor (yet), but from what I read in the documentation I could imagine two ways how GRASS metadata could make their way into such a central metadata catalogue:

  1. XML import (+ this option is in place when GRASS metadata is written to XML, - import requires manual labor)

  2. Metadata harvesting (GeoNetwork can “harvest” metadata from other CSWs, so maybe pycsw already provides automatic an interface in that direction(?). That would be great!)

When it comes to consistent metadata, I guess e.g. lists with values for metadata fields will in practice have to be more or less dynamic (as new staff member are being employed, new topics for spatial analysis arise (requiring new keywords), new types of data being developed and so on). Not sure if manipulating the templates is convenient enough for average users (seems to be mainly suitable for users with some degree of programming experience). However, good to know that providing a custom set of values is already possible through the templates.

If you start working on a DB for metadata related content, feel free to let me know if I can support you by any means (yet I will be on leave for almost the rest of the year, which means I can spend only a limited amount of time).

However, I had a second look on GeoNetwork in this regard, and GeoNetwork is (naturally) using a database for metadata management (one option here is PostgreSQL). If it would be possible to just connect to that same DB (e.g. for fetching and storing key words, data on contact persons…), that would be really, really great, as this ensures that metadata produced in GRASS are 100% compliant to metadata standards used in GeoNetwork (if GRASS users also use GeoNetwork).

Anyway, Rome was not built in a day and I am 100% sure that your work will be very, very useful already from what you planned for GSoC.

All the best,

Stefan

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

11

Hi Stefan,

···

2014-07-03 18:22 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:

Hei Madi,

Thanks for your clarification.

Very interesting! I see I have to take a closer look on pycsw and it`s front-ends (as a possible alternative to GeoNetwork).

The pycsw documentation [1] says that also pycsw Metadata repositories are set up with a database backend (SQLite, PostgreSQL, and even PostGIS support). However, integration with data portal solutions like e.g. GeoNode or Open Data Catalogue is unfortunately “read-only” meaning that meta-data can be only queried [2].

Would be nice if metadata from GRASS could somehow go directly into a geodata portal (and that the other way around e.g. people, keywords … could be fetched from there)… (BTW, in the proprietary world such a functionality requires yet another extension: https://www.geocat.net/bridge/)

For organization with a geodata portal solution it is maybe possible to just sync pycsw`s DB with a DB from e.g. GeoNode or so, if one wants to have that consistent…

If you allow me two more questions in this regard:

Will pycsw become a dependency for GRASS (with metadata support)?

Yes it is. Currently there are no packages for Debian. Pycsw has dependency on SQLalchemy, shapely and pyproj. I have info that debian package will probably be created during a few months.

And will it be necessary, that pycsw runs on the same computer as GRASS (meaning also one pycsw for each GRASS installation) or could that somehow be centralized (in other words that GRASS connects to a central pycsw)?

Yes of course, the pycsw is providing csw catalogue which can run on a local pc (localhost) or on a server. Our vision is to make an interface based on OWSLib and pycsw which produces essential functions like browsing based on filtering, harvesting and publishing metadata to csw catalogue. The user will be able to choose csw catalogue target like a localhost(probably SQLite) or catalogue on internet. This task will part of GSOC term and after…

Thank you,
Matej

12

pycsw is an OGC Reference Implementation of CSW, which means it supports all the features in the spec, including the harvesting and transactions. Best, Angelos

···

Hi Stefan,

On 07/03/2014 10:46 AM, Blumentrath, Stefan wrote:

Hi Matej,

Thanks for your reply.
Metadata is a current topic I my organization at the moment, so your work comes exactly at the right time for me. Here in NINA people are using many different kinds of GIS software and we are considering introducing GeoNetwork as a central metadata storage solution, which makes metadata management to some extend independent from the GIS software used. Among public and environmental organisations, GeoNetwork is quite popular for metadata management (amongst others Mapping authorities from Netherlands, Switzerland, and the Nordic countries are using it, see also: [http://geonetwork-opensource.org/gallery/gallery.html](http://geonetwork-opensource.org/gallery/gallery.html)).

I have no real personal experience with GeoNetwor (yet), but from what I read in the documentation I could imagine two ways how GRASS metadata could make their way into such a central metadata catalogue:

1)      XML import (+ this option is in place when GRASS metadata is written to XML, - import requires manual labor)

2)      Metadata harvesting (GeoNetwork can “harvest” metadata from other CSWs, so maybe pycsw already provides automatic an interface in that direction(?). That would be great!)

When it comes to consistent metadata, I guess e.g. lists with values for metadata fields will in practice have to be more or less dynamic (as new staff member are being employed, new topics for spatial analysis arise (requiring new keywords), new types of data being developed and so on). Not sure if manipulating the templates is convenient enough for average users (seems to be mainly suitable for users with some degree of programming experience). However, good to know that providing a custom set of values is already possible through the templates.
If you start working on a DB for metadata related content, feel free to let me know if I can support you by any means (yet I will be on leave for almost the rest of the year, which means I can spend only a limited amount of time).
However, I had a second look on GeoNetwork in this regard, and GeoNetwork is (naturally) using a database for metadata management (one option here is PostgreSQL). If it would be possible to just connect to that same DB (e.g. for fetching and storing key words, data on contact persons…), that would be really, really great, as this ensures that metadata produced in GRASS are 100% compliant to metadata standards used in GeoNetwork (if GRASS users also use GeoNetwork).

Anyway, Rome was not built in a day and I am 100% sure that your work will be very, very useful already from what you planned for GSoC.

All the best,
Stefan


_______________________________________________
grass-dev mailing list
[grass-dev@lists.osgeo.org](mailto:grass-dev@lists.osgeo.org)
[http://lists.osgeo.org/mailman/listinfo/grass-dev](http://lists.osgeo.org/mailman/listinfo/grass-dev)

-- 
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
[http://users.ntua.gr/tzotsos](http://users.ntua.gr/tzotsos)
13

Almost every CSW server out there has its own implementation of such a database schema (GeoNetwork, pycsw, deegree, MDweb etc…). How are we going to select which is best to support as native? IMHO there is not one solution to this problem. If you take a close look into MDweb for example, there is an almost full implementation of ISO 19115 in a PostgreSQL schema…

···

On 07/03/2014 02:40 PM, Margherita Di Leo wrote:

Hi Stefan,
On Thu, Jul 3, 2014 at 9:46 AM, Blumentrath, Stefan <
[Stefan.Blumentrath@nina.no](mailto:Stefan.Blumentrath@nina.no)> wrote:


However, I had a second look on GeoNetwork in this regard, and GeoNetwork


is (naturally) using a database for metadata management (one option here is
PostgreSQL). If it would be possible to just connect to that same DB (e.g.
for fetching and storing key words, data on contact persons…), that would
be really, really great, as this ensures that metadata produced in GRASS
are 100% compliant to metadata standards used in GeoNetwork (if GRASS users
also use GeoNetwork).




Anyway, Rome was not built in a day and I am 100% sure that your work


will be very, very useful already from what you planned for GSoC.



All the best,

Stefan

_______________________________________________
grass-dev mailing list
[grass-dev@lists.osgeo.org](mailto:grass-dev@lists.osgeo.org)
[http://lists.osgeo.org/mailman/listinfo/grass-dev](http://lists.osgeo.org/mailman/listinfo/grass-dev)




_______________________________________________
grass-dev mailing list
[grass-dev@lists.osgeo.org](mailto:grass-dev@lists.osgeo.org)
[http://lists.osgeo.org/mailman/listinfo/grass-dev](http://lists.osgeo.org/mailman/listinfo/grass-dev)

-- 
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
[http://users.ntua.gr/tzotsos](http://users.ntua.gr/tzotsos)
14

There are pycsw deb packages for UbuntuGIS and OSGeoLive, but those were not accepted for Debian and need some more work. On a side note, there is work in progress for read/write integration of pycsw with CKAN. We are hopping to have a working demo for FOSS4GE in 2 weeks Angelos

···

On 07/04/2014 10:15 PM, Matej Krejci wrote:

Hi Stefan,

2014-07-03 18:22 GMT+02:00 Blumentrath, Stefan [<Stefan.Blumentrath@nina.no>](mailto:Stefan.Blumentrath@nina.no):


 Hei Madi,

Thanks for your clarification.

Very interesting! I see I have to take a closer look on pycsw and it`s
front-ends (as a possible alternative to GeoNetwork).

The pycsw documentation [1] says that also pycsw Metadata repositories are
set up with a database backend (SQLite, PostgreSQL, and even PostGIS
support). However, integration with data portal solutions like e.g. GeoNode
or Open Data Catalogue is unfortunately “read-only” meaning that meta-data
can be only queried [2].

Would be nice if metadata from GRASS could somehow go directly into a
geodata portal (and that the other way around e.g. people, keywords … could
be fetched from there)…  (BTW, in the proprietary world such a
functionality requires yet another extension:
[https://www.geocat.net/bridge/](https://www.geocat.net/bridge/))

For organization with a geodata portal solution it is maybe possible to
just sync pycsw`s DB with a DB from e.g. GeoNode or so, if one wants to
have that consistent…

If you allow me two more questions in this regard:

Will pycsw become a dependency for GRASS (with metadata support)?


Yes it is. Currently there are no packages for Debian. Pycsw has dependency
on SQLalchemy, shapely and pyproj. I have info that debian package
will probably  be created during a few months.

http://publicamundi.eu/



 And will it be necessary, that pycsw runs on the same computer as GRASS
(meaning also one pycsw for each GRASS installation) or could that somehow
be centralized (in other words that GRASS connects to a central pycsw)?


Yes of course, the pycsw is providing csw catalogue which can run on a
local pc (localhost) or on a server. Our vision is to make an interface
based on OWSLib  and pycsw which produces essential functions like browsing
based on filtering, harvesting and publishing metadata to csw catalogue.
The user will be able to choose csw catalogue target like a
localhost(probably SQLite) or catalogue on internet. This task will part of
GSOC term and after...

Thank you,
 Matej


_______________________________________________
grass-dev mailing list
[grass-dev@lists.osgeo.org](mailto:grass-dev@lists.osgeo.org)
[http://lists.osgeo.org/mailman/listinfo/grass-dev](http://lists.osgeo.org/mailman/listinfo/grass-dev)

-- 
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
[http://users.ntua.gr/tzotsos](http://users.ntua.gr/tzotsos)