[GRASS-dev] lidar tools update in grass7

GRASS GRASS developer
1

details on r40831:

Performance has been increased for all modules by reducing the default segment size, interpolation settings are not affected. Previously, the modules seemed to be optimized for settings where only one segment was needed. As soon as several segments where needed, processing time increased drastically. Fixed for all modules, now optimized for larger datasets.

Sometimes interpolation settings were modified for the last row and column segments, i.e. for these segments different interpolation settings were used than for the other segments. The same spline step values are now used for all segments.

All modules have a new flag to estimate point density and mean distance between points. This mean distance is the minimum spline step, according to the documentation recommended spline steps are 2 to 4 times the mean distance, depending on the module.

All modules now create a proper name for the auxiliar table, sometimes a name was created but a different one was used or a fully qualified name with @ was used: illegal table name.
Several modules had memory leaks when accessing the auxiliar table, all are fixed.

v.lidar.edgedetection assigned category 0 instead of 3 to unknown points (neither edge nor terrain), fixed. There was a bad memory leak in v.lidar.edgedetection, fixed.

v.outlier could not properly detect outliers if several segments were needed, the points falling into the overlapping zones were not properly updated (uninitialized variable used).

v.surf.bspline no longer produces artifacts along region boundaries and is now optimized for larger datasets. AFAICT it did not properly work if the region had to be subdivided into several segments, it was restricted to one segment.

test environment was nc_spm_08
g.region vect=elev_lid792_bepts@PERMANENT res=1
north: 221230
south: 219580
west: 637740
east: 639530
nsres: 1
ewres: 1
rows: 1650
cols: 1790
cells: 2953500

e.g. for grass64
v.outlier --o --v input=elev_lid792_bepts@PERMANENT output=elev_lid792_bepts_nooutliers64 outlier=elev_lid792_bepts_outliers64 soe=5 son=5 thres_o=0.1
and grass7
v.outlier --o --v input=elev_lid792_bepts@PERMANENT output=elev_lid792_bepts_nooutliers outlier=elev_lid792_bepts_outliers soe=5 son=5 thres_o=0.1
these settings will detect more outliers than are probably there, they are for illustration of the artefacts.

obvious artefacts in elev_lid792_bepts_outliers64

or in grass64
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT raster=elev_lid792_rast64 sie=5 sin=5 type=bicubic lambda_i=0.1
vs. grass7
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT raster=elev_lid792_rast sie=5 sin=5 type=bicubic lambda_i=0.1

no more artefacts along the region boundaries and faster

2

Horay! Good job!
Still could You, please, take a look at this issue:
http://trac.osgeo.org/grass/ticket/96
Also while You understand that code, please, could You add some
commets to more tricky places in code to make code maintenance more
easy for not-so-die-hard coders? Some things in GRASS lidar tools are
really cool and advanced and thus not so easy to understand without
carefull code review for casual bugfixers.

Thanks!
Maris.

2010/2/5, Markus Metz <markus.metz.giswork@googlemail.com>:

details on r40831:

Performance has been increased for all modules by reducing the default
segment size, interpolation settings are not affected. Previously, the
modules seemed to be optimized for settings where only one segment was
needed. As soon as several segments where needed, processing time
increased drastically. Fixed for all modules, now optimized for larger
datasets.

Sometimes interpolation settings were modified for the last row and
column segments, i.e. for these segments different interpolation
settings were used than for the other segments. The same spline step
values are now used for all segments.

All modules have a new flag to estimate point density and mean distance
between points. This mean distance is the minimum spline step, according
to the documentation recommended spline steps are 2 to 4 times the mean
distance, depending on the module.

All modules now create a proper name for the auxiliar table, sometimes a
name was created but a different one was used or a fully qualified name
with @ was used: illegal table name.
Several modules had memory leaks when accessing the auxiliar table, all
are fixed.

v.lidar.edgedetection assigned category 0 instead of 3 to unknown points
(neither edge nor terrain), fixed. There was a bad memory leak in
v.lidar.edgedetection, fixed.

v.outlier could not properly detect outliers if several segments were
needed, the points falling into the overlapping zones were not properly
updated (uninitialized variable used).

v.surf.bspline no longer produces artifacts along region boundaries and
is now optimized for larger datasets. AFAICT it did not properly work if
the region had to be subdivided into several segments, it was restricted
to one segment.

test environment was nc_spm_08
g.region vect=elev_lid792_bepts@PERMANENT res=1
north: 221230
south: 219580
west: 637740
east: 639530
nsres: 1
ewres: 1
rows: 1650
cols: 1790
cells: 2953500

e.g. for grass64
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers64
outlier=elev_lid792_bepts_outliers64 soe=5 son=5 thres_o=0.1
and grass7
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers outlier=elev_lid792_bepts_outliers
soe=5 son=5 thres_o=0.1
these settings will detect more outliers than are probably there, they
are for illustration of the artefacts.

obvious artefacts in elev_lid792_bepts_outliers64

or in grass64
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast64 sie=5 sin=5 type=bicubic lambda_i=0.1
vs. grass7
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast sie=5 sin=5 type=bicubic lambda_i=0.1

no more artefacts along the region boundaries and faster

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

Maris Nartiss wrote:

Horay! Good job!
Still could You, please, take a look at this issue:
http://trac.osgeo.org/grass/ticket/96
  

The column option works now and the module no longer sits there and does nothing, it reports now progress, and is faster, particularly for bicubic interpolation. I would recommend backporting all changes after some more testing because some of the lidar tools are not working properly in grass 6.x.

In ticket #96, the command

v.surf.bspline input=precip_30ynormals rast=precip_30ynormals_surf column=annual layer=1 sin=1000 sie=1000

is ok for testing, nicer results are obtained with

v.surf.bspline input=precip_30ynormals rast=precip_30ynormals_surf type=bicubic column=annual layer=1 sin=4.0e+4 sie=4.0e+4

Also while You understand that code, please, could You add some
commets to more tricky places in code to make code maintenance more
easy for not-so-die-hard coders?

I'll try my best. I think I understand the code, but I don't really understand that Tcholetsky thing (which I did not touch), so I can't add comments there.

Some things in GRASS lidar tools are
really cool and advanced and thus not so easy to understand without
carefull code review for casual bugfixers.
  

The core of the lidar tools is the lidarlib, that is AFAICT robust and working. Bugs are most likely in modules, so I'll try to add comments there. I have reorganized the code for v.surf.bspline in the hope that it is now easier to read.

I forgot to mention, there is a problem with sqlite, it is much slower than dbf, no idea if this is a problem of my system or of the grass sqlite driver or of the way auxiliary tables are accessed by the lidar tools.

Markus M

Thanks!
Maris.

2010/2/5, Markus Metz <markus.metz.giswork@googlemail.com>:
  

details on r40831:

Performance has been increased for all modules by reducing the default
segment size, interpolation settings are not affected. Previously, the
modules seemed to be optimized for settings where only one segment was
needed. As soon as several segments where needed, processing time
increased drastically. Fixed for all modules, now optimized for larger
datasets.

Sometimes interpolation settings were modified for the last row and
column segments, i.e. for these segments different interpolation
settings were used than for the other segments. The same spline step
values are now used for all segments.

All modules have a new flag to estimate point density and mean distance
between points. This mean distance is the minimum spline step, according
to the documentation recommended spline steps are 2 to 4 times the mean
distance, depending on the module.

All modules now create a proper name for the auxiliar table, sometimes a
name was created but a different one was used or a fully qualified name
with @ was used: illegal table name.
Several modules had memory leaks when accessing the auxiliar table, all
are fixed.

v.lidar.edgedetection assigned category 0 instead of 3 to unknown points
(neither edge nor terrain), fixed. There was a bad memory leak in
v.lidar.edgedetection, fixed.

v.outlier could not properly detect outliers if several segments were
needed, the points falling into the overlapping zones were not properly
updated (uninitialized variable used).

v.surf.bspline no longer produces artifacts along region boundaries and
is now optimized for larger datasets. AFAICT it did not properly work if
the region had to be subdivided into several segments, it was restricted
to one segment.

test environment was nc_spm_08
g.region vect=elev_lid792_bepts@PERMANENT res=1
north: 221230
south: 219580
west: 637740
east: 639530
nsres: 1
ewres: 1
rows: 1650
cols: 1790
cells: 2953500

e.g. for grass64
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers64
outlier=elev_lid792_bepts_outliers64 soe=5 son=5 thres_o=0.1
and grass7
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers outlier=elev_lid792_bepts_outliers
soe=5 son=5 thres_o=0.1
these settings will detect more outliers than are probably there, they
are for illustration of the artefacts.

obvious artefacts in elev_lid792_bepts_outliers64

or in grass64
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast64 sie=5 sin=5 type=bicubic lambda_i=0.1
vs. grass7
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast sie=5 sin=5 type=bicubic lambda_i=0.1

no more artefacts along the region boundaries and faster

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

4

On Feb 6, 2010, at 3:46 AM, Markus Metz wrote:

Maris Nartiss wrote:

Horay! Good job!
Still could You, please, take a look at this issue:
http://trac.osgeo.org/grass/ticket/96

The column option works now and the module no longer sits there and does nothing, it reports now progress, and is faster, particularly for bicubic interpolation. I would recommend backporting all changes after some more testing because some of the lidar tools are not working properly in grass 6.x.

I vote for getting the fixes backported at least to grass65 and in fact they should go into the 64 release because the modules that are there currently have numerous bugs are not generally usable. (e.g. when running v.outlier the data got cleaned up correctly but only about half of the segments made it into the output file - we haven't tried to run it on smaller sections, but overall it was rather erratic).
Here is a good raw lidar data set for testing (it can be imported into nc_spm, it is the same coord. system),
but I can have it tested here if the fixes make it to mswidnows daily binaries.
http://skagit.meas.ncsu.edu/~helena/measwork/jockeys/data/JR07_allraw_spm.txt.gz

thanks a lot - that sounds like a great progress,

Helena

In ticket #96, the command

v.surf.bspline input=precip_30ynormals rast=precip_30ynormals_surf column=annual layer=1 sin=1000 sie=1000

is ok for testing, nicer results are obtained with

v.surf.bspline input=precip_30ynormals rast=precip_30ynormals_surf type=bicubic column=annual layer=1 sin=4.0e+4 sie=4.0e+4

Also while You understand that code, please, could You add some
commets to more tricky places in code to make code maintenance more
easy for not-so-die-hard coders?

I'll try my best. I think I understand the code, but I don't really understand that Tcholetsky thing (which I did not touch), so I can't add comments there.

Some things in GRASS lidar tools are
really cool and advanced and thus not so easy to understand without
carefull code review for casual bugfixers.

The core of the lidar tools is the lidarlib, that is AFAICT robust and working. Bugs are most likely in modules, so I'll try to add comments there. I have reorganized the code for v.surf.bspline in the hope that it is now easier to read.

I forgot to mention, there is a problem with sqlite, it is much slower than dbf, no idea if this is a problem of my system or of the grass sqlite driver or of the way auxiliary tables are accessed by the lidar tools.

Markus M

Thanks!
Maris.

2010/2/5, Markus Metz <markus.metz.giswork@googlemail.com>:

details on r40831:

Performance has been increased for all modules by reducing the default
segment size, interpolation settings are not affected. Previously, the
modules seemed to be optimized for settings where only one segment was
needed. As soon as several segments where needed, processing time
increased drastically. Fixed for all modules, now optimized for larger
datasets.

Sometimes interpolation settings were modified for the last row and
column segments, i.e. for these segments different interpolation
settings were used than for the other segments. The same spline step
values are now used for all segments.

All modules have a new flag to estimate point density and mean distance
between points. This mean distance is the minimum spline step, according
to the documentation recommended spline steps are 2 to 4 times the mean
distance, depending on the module.

All modules now create a proper name for the auxiliar table, sometimes a
name was created but a different one was used or a fully qualified name
with @ was used: illegal table name.
Several modules had memory leaks when accessing the auxiliar table, all
are fixed.

v.lidar.edgedetection assigned category 0 instead of 3 to unknown points
(neither edge nor terrain), fixed. There was a bad memory leak in
v.lidar.edgedetection, fixed.

v.outlier could not properly detect outliers if several segments were
needed, the points falling into the overlapping zones were not properly
updated (uninitialized variable used).

v.surf.bspline no longer produces artifacts along region boundaries and
is now optimized for larger datasets. AFAICT it did not properly work if
the region had to be subdivided into several segments, it was restricted
to one segment.

test environment was nc_spm_08
g.region vect=elev_lid792_bepts@PERMANENT res=1
north: 221230
south: 219580
west: 637740
east: 639530
nsres: 1
ewres: 1
rows: 1650
cols: 1790
cells: 2953500

e.g. for grass64
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers64
outlier=elev_lid792_bepts_outliers64 soe=5 son=5 thres_o=0.1
and grass7
v.outlier --o --v input=elev_lid792_bepts@PERMANENT
output=elev_lid792_bepts_nooutliers outlier=elev_lid792_bepts_outliers
soe=5 son=5 thres_o=0.1
these settings will detect more outliers than are probably there, they
are for illustration of the artefacts.

obvious artefacts in elev_lid792_bepts_outliers64

or in grass64
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast64 sie=5 sin=5 type=bicubic lambda_i=0.1
vs. grass7
v.surf.bspline --o --v input=elev_lid792_bepts@PERMANENT
raster=elev_lid792_rast sie=5 sin=5 type=bicubic lambda_i=0.1

no more artefacts along the region boundaries and faster

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev