[GRASS-dev] Heading towards unified dataset

Hi,

at FOSS4G we were talking about the need of unified dataset, a GRASS location in our GRASS case, to enable easy writing of examples and also tests.

The location would have maps with unified names such as “elevation” and these names can be used in the examples and tests so that both examples and tests can, to certain extent, work in different locations. For examples in manual pages or other educational material, this would mean that it could be used better across more countries or areas. For tests, this would mean that different projection or data can be tested with the algorithms.

This has of course its limits, for example the result of statistical analysis would be different in different locations but the point is that the analysis can be done.

We already have NC (full) location and NC basic location. They have these raster maps in common:

elevation
elevation_shade
lakes

These maps are in NC basic:

basins
geology
landuse
soils

But in full NC they have different names:

basin_50K
geology_30m
landuse96_28m
soilsID

Of course the longer names have their reasoning in differences with similar raster maps in the same location but I would say that having unified names is more advantageous for teaching/test datasets then absolute clearness of names. This should be in metadata anyway.

One can also argue about the unified names themselves (e.g. elevation vs dtm or usage of underscore) but most of it is pretty clear since it has to be the most general names possible.

The names must be obviously in English. If somebody would like to have data in different language, derived dataset must be created. Perhaps it would be possible to provide some batch version of g.rename (but there are also attribute columns and others).

The last issue might be what if there is nothing in the area which can be part of the map or if dam or pond are lakes. But we can allow for some inaccuracies when creating a training dataset.

The other locations which can be unified are Piemonte and Spearfish.

So, what are the next steps? Decide about which maps to include and which names to use? Let’s start from the NC basic location.

Raster maps

basins
elevation
elevation_shade
geology
lakes
landuse
soils

I’m not sure if geology and soils would be available in other locations, so we could leave out them. However, they are available for Spearfish and maybe for Piemonte (my Italian is not really usable).

Vector maps

boundary_region
boundary_state
census
elev_points
firestations
geology
geonames
hospitals
points_of_interest
railroads
roadsmajor
schools
streams
streets
zipcodes

We would need to have at at least one map for each type. I’m not sure what are the crucial ones and broadly available but it seems that training datasets are usually near some civilization, so roads or schools might be available. Buildings would be nice to have.

Attribute data, time series and 3D rasters and (real) 3D vectors are of course whole new level. So, I would start with rasters and (mostly 2D) vectors.

Vaclav

To better coordinate creation of the new sample dataset, I created a page on GRASS Trac wiki:

http://trac.osgeo.org/grass/wiki/SampleDataset

I invite all to join the effort.

Please see also the discussions about the sample spatio-temporal data:

http://lists.osgeo.org/pipermail/grass-dev/2014-October/071114.html
http://osgeo-org.1560.x6.nabble.com/sample-vector-temporal-data-td5166407.html
http://comments.gmane.org/gmane.comp.gis.grass.devel/60441

http://lists.osgeo.org/pipermail/grass-dev/2014-May/068601.html
http://osgeo-org.1560.x6.nabble.com/sample-dataset-for-temporal-data-td5137870.html

···

On Tue, Sep 30, 2014 at 2:18 PM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

Hi,

at FOSS4G we were talking about the need of unified dataset, a GRASS location in our GRASS case, to enable easy writing of examples and also tests.

The location would have maps with unified names such as “elevation” and these names can be used in the examples and tests so that both examples and tests can, to certain extent, work in different locations. For examples in manual pages or other educational material, this would mean that it could be used better across more countries or areas. For tests, this would mean that different projection or data can be tested with the algorithms.

This has of course its limits, for example the result of statistical analysis would be different in different locations but the point is that the analysis can be done.

We already have NC (full) location and NC basic location. They have these raster maps in common:

elevation
elevation_shade
lakes

These maps are in NC basic:

basins
geology
landuse
soils

But in full NC they have different names:

basin_50K
geology_30m
landuse96_28m
soilsID

Of course the longer names have their reasoning in differences with similar raster maps in the same location but I would say that having unified names is more advantageous for teaching/test datasets then absolute clearness of names. This should be in metadata anyway.

One can also argue about the unified names themselves (e.g. elevation vs dtm or usage of underscore) but most of it is pretty clear since it has to be the most general names possible.

The names must be obviously in English. If somebody would like to have data in different language, derived dataset must be created. Perhaps it would be possible to provide some batch version of g.rename (but there are also attribute columns and others).

The last issue might be what if there is nothing in the area which can be part of the map or if dam or pond are lakes. But we can allow for some inaccuracies when creating a training dataset.

The other locations which can be unified are Piemonte and Spearfish.

So, what are the next steps? Decide about which maps to include and which names to use? Let’s start from the NC basic location.

Raster maps

basins
elevation
elevation_shade
geology
lakes
landuse
soils

I’m not sure if geology and soils would be available in other locations, so we could leave out them. However, they are available for Spearfish and maybe for Piemonte (my Italian is not really usable).

Vector maps

boundary_region
boundary_state
census
elev_points
firestations
geology
geonames
hospitals
points_of_interest
railroads
roadsmajor
schools
streams
streets
zipcodes

We would need to have at at least one map for each type. I’m not sure what are the crucial ones and broadly available but it seems that training datasets are usually near some civilization, so roads or schools might be available. Buildings would be nice to have.

Attribute data, time series and 3D rasters and (real) 3D vectors are of course whole new level. So, I would start with rasters and (mostly 2D) vectors.

Vaclav

On Sun, Dec 28, 2014 at 2:55 AM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

To better coordinate creation of the new sample dataset, I created a page on
GRASS Trac wiki:

http://trac.osgeo.org/grass/wiki/SampleDataset

The lists (I have added some comments) looks very good to me.
It is also (now) needed for OSGeoLive8.5 which cannot fit the "old"
140MB NC dataset.

See
http://trac.osgeo.org/osgeo/ticket/1446#comment:7

Markus