[GRASS-dev] sample dataset update

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M

Markus,

I have a TODO list for data set update already as well as several data sets, so if you set up a wiki page,
I will put there what I have planned with the links to the data.
There are numerous excellent public data sets available that we can
add. The current data set's PERMANENT mapset should stay as is so that
old material such as GRASS book works, but the new data could be added as separate mapsets that people
can insert into the location, such as here:
http://courses.ncsu.edu/mea582/common/GIS_anal_grass/GIS_Anal_grgeomorph2.html
and/or we can prepare a completely new location.

Helena

On Jun 20, 2011, at 11:15 AM, Markus Metz wrote:

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M

Below are some plans for data updates. It would be useful to put this on wiki
so that others can comment, add, modify. I was thinking about
http://grass.osgeo.org/wiki/Sample_datasets
but this is really for downloads, so perhaps as a new section here?
http://grass.osgeo.org/wiki/GRASS_7_ideas_collection

Feel free to post it on wiki (or suggest where it would fit and I will post it).
After I get some feedback I will prepare the first version of the new mapsets for networks and timeseries
to test whether they are suitable,

thanks, Helena

------------------------------------------------------------------------------------------------

Prepare new data sets (grass7 vector format?):
- provided as separate MAPSETS that can be inserted into nc_spm_08 or gisdemo_ncspm (or their grass7 versions)
- packaged all into single location (nc_spm_11) which could reach 1GB.
- each MAPSET will have an example of application

proposed new MAPSETS:

Road and street networks (nconemap, ncdot)
http://www.ncdot.org/it/gis/DataDistribution/DOTData/default.html
LRS road network, Integrated Statewide Road network, Bike Paths
- streets for wake http://www.wakegov.com/gis/services/data.htm\`
-add boundaries and POI for reference?
-include emergency centers ? hurricane evacuation routes? potential emergency shelters?
-include stream network?

Time_series_coast
coastal lidar point clouds, 1m res DEMs, orthos, 15 years in 10-15 snapshots
extended version of this
http://courses.ncsu.edu/mea582/common/media/01/NagsHead_series.zip
also used for coastal analysis toolbox add-ons

Landuse_img
new 4 channel 1m res 2009 imagery NAID
new NLCD 2006 from USGS and NLCD2001
0.15m res ortho?
include any post storm, post hurricane, post flood imagery for training purposes?
see example here:
http://www.ncdot.org/it/gis/DataDistribution/SpecialData/NCStormDamageAndCleanup.html

Time_series_census
this may take more time to prepare,
data are available for 1970,80,90,2000,2010
combine with election districts?

LOCATIONS:
- cleanup and provide nc_ll, nc_utm, nc_spf for testing projections

EXTERNAL data sets for testing external data formats
- point cloud asci, las
- shape
- geodatabase?
- MRsid, geotiff?

Anything else from here?
http://courses.ncsu.edu/mea582/common/GIS_anal_lecture/GIS_Anal_webdata.html
http://www.nconemap.com/Default.aspx?tabid=286
http://www.wakegov.com/gis/services/data.htm
(we provide link to this so only data to be used for GRASS tutorials and testing should be in the data set)

On Jun 20, 2011, at 11:15 AM, Markus Metz wrote:

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M

On Thu, Jun 23, 2011 at 6:44 AM, Helena Mitasova <hmitaso@ncsu.edu> wrote:

Below are some plans for data updates. It would be useful to put this on wiki
so that others can comment, add, modify. I was thinking about
http://grass.osgeo.org/wiki/Sample_datasets
but this is really for downloads, so perhaps as a new section here?
http://grass.osgeo.org/wiki/GRASS_7_ideas_collection

I would suggest to keep GRASS 7 ideas and the sample datasets separate
because 1) we do not know when GRASS 7 will be released and 2) the
additional datasets you listed below are very interesting, also for
GRASS 6.x. Therefore I would tend to add any new datasets to
http://grass.osgeo.org/wiki/Sample_datasets
even though this is currently for downloads only, but the two main
sites for sample data download are
http://grass.osgeo.org/download/data.php
and
http://www.grassbook.org/data_menu3rd.php
so it could be ok to add new ideas and TODO's to the wiki?

I like the idea of having separately downloadable mapsets for the two
existing sample locations. This keeps download size manageable and
users can pick the sample data they are interested in. Of course
additional sample locations are also an option and I think that there
are already some new sample locations in preparation, e.g. for Italy.

Another issue are the examples based on the sample data, both in the
book and in the documentation. All the examples for vector network
analysis that use cost columns are wrong. I am busy updating the
v.net.* manuals for GRASS 7 and will (hopefully) soon backport the
manuals to 6.5 and 6.4. Other examples in the manuals, and the manuals
themselves, may need a closer look too...

With regard to GRASS 7 vector format, I want to change it further and
would therefore postpone a GRASS 7 sample dataset a bit.
Forward/backward compatibility will be limited: rebuilding topology
will be required when switching from 6.x to 7 or back (currently GRASS
6.x can read GRASS 7 vectors as they are). For the existing sample
datasets spearfish and nc_spm, v.build.all takes less than a minute on
my laptop (6.x and 7), so this is not really an obstacle.

For external file formats, we could also provide more links to free
datasources, however, examples based on these data may not work when
these datasources are updated.

Markus M

Feel free to post it on wiki (or suggest where it would fit and I will post it).
After I get some feedback I will prepare the first version of the new mapsets for networks and timeseries
to test whether they are suitable,

thanks, Helena

------------------------------------------------------------------------------------------------

Prepare new data sets (grass7 vector format?):
- provided as separate MAPSETS that can be inserted into nc_spm_08 or gisdemo_ncspm (or their grass7 versions)
- packaged all into single location (nc_spm_11) which could reach 1GB.
- each MAPSET will have an example of application

proposed new MAPSETS:

Road and street networks (nconemap, ncdot)
http://www.ncdot.org/it/gis/DataDistribution/DOTData/default.html
LRS road network, Integrated Statewide Road network, Bike Paths
- streets for wake http://www.wakegov.com/gis/services/data.htm\`
-add boundaries and POI for reference?
-include emergency centers ? hurricane evacuation routes? potential emergency shelters?
-include stream network?

Time_series_coast
coastal lidar point clouds, 1m res DEMs, orthos, 15 years in 10-15 snapshots
extended version of this
http://courses.ncsu.edu/mea582/common/media/01/NagsHead_series.zip
also used for coastal analysis toolbox add-ons

Landuse_img
new 4 channel 1m res 2009 imagery NAID
new NLCD 2006 from USGS and NLCD2001
0.15m res ortho?
include any post storm, post hurricane, post flood imagery for training purposes?
see example here:
http://www.ncdot.org/it/gis/DataDistribution/SpecialData/NCStormDamageAndCleanup.html

Time_series_census
this may take more time to prepare,
data are available for 1970,80,90,2000,2010
combine with election districts?

LOCATIONS:
- cleanup and provide nc_ll, nc_utm, nc_spf for testing projections

EXTERNAL data sets for testing external data formats
- point cloud asci, las
- shape
- geodatabase?
- MRsid, geotiff?

Anything else from here?
http://courses.ncsu.edu/mea582/common/GIS_anal_lecture/GIS_Anal_webdata.html
http://www.nconemap.com/Default.aspx?tabid=286
http://www.wakegov.com/gis/services/data.htm
(we provide link to this so only data to be used for GRASS tutorials and testing should be in the data set)

On Jun 20, 2011, at 11:15 AM, Markus Metz wrote:

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M

On Fri, Jun 24, 2011 at 11:13 AM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:

On Thu, Jun 23, 2011 at 6:44 AM, Helena Mitasova <hmitaso@ncsu.edu> wrote:

Below are some plans for data updates. It would be useful to put this on wiki
so that others can comment, add, modify. I was thinking about
http://grass.osgeo.org/wiki/Sample_datasets
but this is really for downloads, so perhaps as a new section here?
http://grass.osgeo.org/wiki/GRASS_7_ideas_collection

I would suggest to keep GRASS 7 ideas and the sample datasets separate
because 1) we do not know when GRASS 7 will be released and 2) the
additional datasets you listed below are very interesting, also for
GRASS 6.x. Therefore I would tend to add any new datasets to
http://grass.osgeo.org/wiki/Sample_datasets
even though this is currently for downloads only, but the two main
sites for sample data download are
http://grass.osgeo.org/download/data.php
and
http://www.grassbook.org/data_menu3rd.php
so it could be ok to add new ideas and TODO's to the wiki?

I have updated http://grass.osgeo.org/wiki/Sample_datasets

OK?

Markus M

I like the idea of having separately downloadable mapsets for the two
existing sample locations. This keeps download size manageable and
users can pick the sample data they are interested in. Of course
additional sample locations are also an option and I think that there
are already some new sample locations in preparation, e.g. for Italy.

Another issue are the examples based on the sample data, both in the
book and in the documentation. All the examples for vector network
analysis that use cost columns are wrong. I am busy updating the
v.net.* manuals for GRASS 7 and will (hopefully) soon backport the
manuals to 6.5 and 6.4. Other examples in the manuals, and the manuals
themselves, may need a closer look too...

With regard to GRASS 7 vector format, I want to change it further and
would therefore postpone a GRASS 7 sample dataset a bit.
Forward/backward compatibility will be limited: rebuilding topology
will be required when switching from 6.x to 7 or back (currently GRASS
6.x can read GRASS 7 vectors as they are). For the existing sample
datasets spearfish and nc_spm, v.build.all takes less than a minute on
my laptop (6.x and 7), so this is not really an obstacle.

For external file formats, we could also provide more links to free
datasources, however, examples based on these data may not work when
these datasources are updated.

Markus M

Feel free to post it on wiki (or suggest where it would fit and I will post it).
After I get some feedback I will prepare the first version of the new mapsets for networks and timeseries
to test whether they are suitable,

thanks, Helena

------------------------------------------------------------------------------------------------

Prepare new data sets (grass7 vector format?):
- provided as separate MAPSETS that can be inserted into nc_spm_08 or gisdemo_ncspm (or their grass7 versions)
- packaged all into single location (nc_spm_11) which could reach 1GB.
- each MAPSET will have an example of application

proposed new MAPSETS:

Road and street networks (nconemap, ncdot)
http://www.ncdot.org/it/gis/DataDistribution/DOTData/default.html
LRS road network, Integrated Statewide Road network, Bike Paths
- streets for wake http://www.wakegov.com/gis/services/data.htm\`
-add boundaries and POI for reference?
-include emergency centers ? hurricane evacuation routes? potential emergency shelters?
-include stream network?

Time_series_coast
coastal lidar point clouds, 1m res DEMs, orthos, 15 years in 10-15 snapshots
extended version of this
http://courses.ncsu.edu/mea582/common/media/01/NagsHead_series.zip
also used for coastal analysis toolbox add-ons

Landuse_img
new 4 channel 1m res 2009 imagery NAID
new NLCD 2006 from USGS and NLCD2001
0.15m res ortho?
include any post storm, post hurricane, post flood imagery for training purposes?
see example here:
http://www.ncdot.org/it/gis/DataDistribution/SpecialData/NCStormDamageAndCleanup.html

Time_series_census
this may take more time to prepare,
data are available for 1970,80,90,2000,2010
combine with election districts?

LOCATIONS:
- cleanup and provide nc_ll, nc_utm, nc_spf for testing projections

EXTERNAL data sets for testing external data formats
- point cloud asci, las
- shape
- geodatabase?
- MRsid, geotiff?

Anything else from here?
http://courses.ncsu.edu/mea582/common/GIS_anal_lecture/GIS_Anal_webdata.html
http://www.nconemap.com/Default.aspx?tabid=286
http://www.wakegov.com/gis/services/data.htm
(we provide link to this so only data to be used for GRASS tutorials and testing should be in the data set)

On Jun 20, 2011, at 11:15 AM, Markus Metz wrote:

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M

I have just finished a beta version of elevation and aerial photography time series mapset
to be used with nc_spm_08 or gisdemo location (it was much more work than expected).

Here are the links (I added them to the related wiki page:
http://grass.osgeo.org/wiki/Sample_datasets)

Reference location for NC with projection information
http://courses.ncsu.edu/mea792/common/media/gisdemo.zip

Time series mapset and data description with instructions
http://courses.ncsu.edu/mea792/common/media/nc_coast_demseries.zip
http://courses.ncsu.edu/mea792/common/media/readme_nc_coast_demseries.txt

To keep the mapset at reasonable size I have packaged the raw point clouds
separately, they can be imported with v.in.ascii or r.in.xyz
http://courses.ncsu.edu/mea792/common/media/nc_coast_pointseries.zip

The data set is rather complex with variable spatial coverage and time step,
(see the description) but should be useful for testing and teaching
more complex analyses.

Here is a "getting started" with this data set assignment
http://courses.ncsu.edu/mea792/common/Assign_GISamodel/a_timeseries.html
(I will ad a space-time cube practice soon).

We have just received 2009 and 2010 lidar data, as well as 2011 imagery
and Post hurricane Irene lidar survey was just flown, so I will try to include those into the final version.
I may use the las format for the newer lidar data

Finally a note - given that a mapset does not have coordinate system information
associated with it I am wondering whether packaging this mapset with its own
location would be a better idea.

Please let me know any suggestions, comments for the data set - I will try to
do as much as time allows before the final release.

Helena

P.S. Soeren, I see your submission of t.* commands - we will give it a try
with these data when it is ready.

P.S.S I am also working on the new roads and streams networks mapset
(there are several versions of these data sets so I need to find out which one to include)

On Jun 20, 2011, at 11:15 AM, Markus Metz wrote:

Hi all,

we may want to think about an update of the sample datasets. I have
recently updated the documentation of the v.net.* modules including
examples and noticed that roads digitized as multilanes have wrong
line directions, i.e. not matching drive directions. This makes
assignment of different forward/backward costs impossible. The vector
map roads in nc_spm_08 should also get a new layer with unique
category values. The updated, correct examples are anything but
intuitive and would be much shorter and simpler with already existing
unique categories in roads@PERMANENT.

Helena also mentioned the LiDAR-based DEM in nc_spm_08 which needs to
be expanded. With the development of a test suite for G7, I am sure
that there will be more requests for updates of the sample datasets.

Time for a TODO list somewhere?

Markus M