#96: v.surf.bspline intermittent failure - maybe on large datasets
----------------------+-----------------------------------------------------
Reporter: cmbarton | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: major | Milestone: 6.3.0
Component: default | Version: unspecified
Keywords: |
----------------------+-----------------------------------------------------
v.surf.bspline is a potentially useful tool. But it seems to have bugs
that show up intermittently. On modest size datasets, it works very fast.
However, with many points, it seems to sit and do nothing. It's not clear
whether it is locked up or becomes extremely slow. I'm still not sure if
the column option works correctly.
#96: v.surf.bspline broken
-----------------------+----------------------------------------------------
Reporter: cmbarton | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: Raster | Version: unspecified
Resolution: | Keywords:
Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Changes (by cmbarton):
* platform: => All
* component: default => Raster
* cpu: => All
* summary: v.surf.bspline intermittent failure - maybe on large datasets
=> v.surf.bspline broken
Comment:
This has set broken for many months now. Should we remove it from the main
GRASS distribution until someone is able to work on it? It would be a nice
addition to GRASS interpolation tools, but it doesn't seem like a good
idea to ship a module that doesn't work and is not being fixed.
#96: v.surf.bspline broken
-----------------------+----------------------------------------------------
Reporter: cmbarton | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: Raster | Version: unspecified
Resolution: | Keywords:
Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Changes (by neteler):
* cc: rantolin (added)
Comment:
I got a bugfix from Roberto which needs to be tested (patch attached).
It includes G_percent() output and two message cosmetics from me.
Test case for North Carolina data set:
{{{
g.region res=500 vect=precip_30ynormals -ap
v.surf.bspline precip_30ynormals colum=annual rast=precip_30ynormals_surf
layer=1 sin=1000 sie=1000
}}}
v.univar precip_30ynormals column=annual type=point
number of features with non NULL attribute: 136
number of missing attributes: 0
number of NULL attributes: 0
minimum: 947.42
maximum: 2329.18
range: 1381.76
mean: 1289.31
mean of absolute values: 1289.31
population standard deviation: 198.572
population variance: 39431
population coefficient of variation: 0.154014
sample standard deviation: 199.306
sample variance: 39723.1
kurtosis: 8.48418
skewness: 2.48352
v.surf.bspline precip_30ynormals rast=precip_30ynormals_surf column=annual
layer=1 sin=1000 sie=1000
No vector map of sparse points to interpolate. Interpolation will be done
with <precip_30ynormals> vector map
[136] records selected from table
100%
v.surf.bspline complete.
r.univar precip_30ynormals_surf
100%
total null and non-null cells: 856388
total null cells: 0
Of the non-null cells:
----------------------
n: 856388
minimum: 0
maximum: 0
range: 0
mean: 0
mean of absolute values: 0
standard deviation: 0
variance: 0
variation coefficient: nan %
sum: 0
}}}
I found suspicious code which was introduced in the last fix round.
Commenting that out attributes are taken again. I don't understand the
purpose of the part which I have commented in attached patch
v.surf.bspline_fix_attribs.diff. AFAIK "type" is defined but not set.
There is another "type" in main() - perhaps a different variable was
supposed to be used to test against GV_POINTS?
Replying to [comment:7 neteler]:
> I found suspicious code which was introduced in the last fix round.
Commenting that out attributes are taken again. I don't understand the
purpose of the part which I have commented in attached patch
v.surf.bspline_fix_attribs.diff. AFAIK "type" is defined but not set.
There is another "type" in main() - perhaps a different variable was
supposed to be used to test against GV_POINTS?
>
> Markus
Still I have no idea how to fix this module (I'm lazy) (fix == does it
needs that check or not). Also I looked at P_Read_Vector_Region_Map
(lidarlib/zones.c) and it seemed somehow fishy - it reads vector line and
then uses only first coordinate of that line, still as code lacks any
comments about it's design specifics (and it has them i.e. as it crunches
over all vector lines on every function call instad of using standart V_*
functions) I can only suggest to somebody who understands that code to add
comments and, probably, line type check.
I was just able to test this with an sqlite attribute database. The column
option is still broken in all versions. If this is non-functional and no
way to fix it, perhaps we should just take it out of the interface and
manual.
this is an issue that may be related to the v.surf.bspline #96 bug.
When running v.outlier on 3D vector file with topology built but no DBF table
we get the following error:
The output files are actually written but when trying to display them we get
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outlierfree siz=1 co=red
WARNING: Coor files of vector map <JR2007_outlierfree@helena> is larger
than it should be (94411739 bytes excess)
WARNING: Unable to display areas, topology not available
Segmentation fault
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outliers siz=1 co=red
WARNING: Coor files of vector map <JR2007_outliers@helena> is larger than
it should be (839 bytes excess)
WARNING: Unable to display areas, topology not available
Looking at the manual it seems that the table is needed only for QGIS output where
the elevation is stored as category. So I am wondering whether this is a bug
that needs to be fixed or whether the v.* lidar modules require points
with elevation stored in DBF (which is a problem for large datasets) - if that is
the case it should be probably mentioned in the man page.
These modules would be really useful if we could get them to work,
Hello Helena,
no. v.outlier issues are not related to v.surf.bspline DBF problems.
v.surf.bspline has some hairy code and thus I was not trying to fix it
- to not broke something by accident.
v.outlier is different. Open a trac request to remove database
dependency for it. Current code allover manipulates with database and
it will require some work to make database manipulation optional.
Also on error it should not leave unfinished vector files on disk.
Also having QGIS as an output option should be renamed to reveal this
options output content as it migh be usefull also for other GUIs not
only QGIS.
Maris.
2009/11/22, helena <hmitaso@unity.ncsu.edu>:
this is an issue that may be related to the v.surf.bspline #96 bug.
When running v.outlier on 3D vector file with topology built but no
DBF table
we get the following error:
The output files are actually written but when trying to display them
we get
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outlierfree
siz=1 co=red
WARNING: Coor files of vector map <JR2007_outlierfree@helena> is larger
than it should be (94411739 bytes excess)
WARNING: Unable to display areas, topology not available
Segmentation fault
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outliers siz=1
co=red
WARNING: Coor files of vector map <JR2007_outliers@helena> is larger
than
it should be (839 bytes excess)
WARNING: Unable to display areas, topology not available
Looking at the manual it seems that the table is needed only for QGIS
output where
the elevation is stored as category. So I am wondering whether this
is a bug
that needs to be fixed or whether the v.* lidar modules require points
with elevation stored in DBF (which is a problem for large datasets) -
if that is
the case it should be probably mentioned in the man page.
These modules would be really useful if we could get them to work,
According to the comments in the code, the auxiliary table is used to store info about overlapping zones and is dropped at the end. The auxiliary table is always created and used, also when no output vector for display (QGIS output) is given.
Markus M
helena wrote:
this is an issue that may be related to the v.surf.bspline #96 bug.
When running v.outlier on 3D vector file with topology built but no DBF table
we get the following error:
The output files are actually written but when trying to display them we get
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outlierfree siz=1 co=red
WARNING: Coor files of vector map <JR2007_outlierfree@helena> is larger
than it should be (94411739 bytes excess)
WARNING: Unable to display areas, topology not available
Segmentation fault
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outliers siz=1 co=red
WARNING: Coor files of vector map <JR2007_outliers@helena> is larger than
it should be (839 bytes excess)
WARNING: Unable to display areas, topology not available
Looking at the manual it seems that the table is needed only for QGIS output where
the elevation is stored as category. So I am wondering whether this is a bug
that needs to be fixed or whether the v.* lidar modules require points
with elevation stored in DBF (which is a problem for large datasets) - if that is
the case it should be probably mentioned in the man page.
These modules would be really useful if we could get them to work,
Hello Markus,
if You understand taht code - how complex (usefull) would be getting
rid of DB at all? I mean - is it possible to store temporary data into
some faster data structure (file?) and thus eliminate all DB
dependencies for this module?
Maris.
2009/11/23, Markus Metz <markus.metz.giswork@googlemail.com>:
Please try trunk r39785.
According to the comments in the code, the auxiliary table is used to
store info about overlapping zones and is dropped at the end. The
auxiliary table is always created and used, also when no output vector
for display (QGIS output) is given.
Markus M
helena wrote:
this is an issue that may be related to the v.surf.bspline #96 bug.
When running v.outlier on 3D vector file with topology built but no
DBF table
we get the following error:
The output files are actually written but when trying to display them
we get
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outlierfree
siz=1 co=red
WARNING: Coor files of vector map <JR2007_outlierfree@helena> is larger
than it should be (94411739 bytes excess)
WARNING: Unable to display areas, topology not available
Segmentation fault
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outliers siz=1
co=red
WARNING: Coor files of vector map <JR2007_outliers@helena> is larger than
it should be (839 bytes excess)
WARNING: Unable to display areas, topology not available
Looking at the manual it seems that the table is needed only for QGIS
output where
the elevation is stored as category. So I am wondering whether this is
a bug
that needs to be fixed or whether the v.* lidar modules require points
with elevation stored in DBF (which is a problem for large datasets) -
if that is
the case it should be probably mentioned in the man page.
These modules would be really useful if we could get them to work,
sorry, I don't really understand the purpose of that auxiliary table in P_outlier() in outlier.c. It is possible that all code referring to that table can be safely removed, but that needs some more testing. If the table is indeed used and not getting too large, it could be replaced with another faster data structure kept in memory. If the table is getting large, it could be difficult to come up with something file-based that's faster than a database. Anyway, it looks like the LiDAR tools could do with some maintenance.
Markus M
Maris Nartiss wrote:
Hello Markus,
if You understand taht code - how complex (usefull) would be getting
rid of DB at all? I mean - is it possible to store temporary data into
some faster data structure (file?) and thus eliminate all DB
dependencies for this module?
Maris.
2009/11/23, Markus Metz <markus.metz.giswork@googlemail.com>:
Please try trunk r39785.
According to the comments in the code, the auxiliary table is used to
store info about overlapping zones and is dropped at the end. The
auxiliary table is always created and used, also when no output vector
for display (QGIS output) is given.
Markus M
helena wrote:
this is an issue that may be related to the v.surf.bspline #96 bug.
When running v.outlier on 3D vector file with topology built but no
DBF table
we get the following error:
The output files are actually written but when trying to display them
we get
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outlierfree
siz=1 co=red
WARNING: Coor files of vector map <JR2007_outlierfree@helena> is larger
than it should be (94411739 bytes excess)
WARNING: Unable to display areas, topology not available
Segmentation fault
GRASS 6.4.0RC5 (nc_spm_08):~/tgrassdata > d.vect JR2007_outliers siz=1
co=red
WARNING: Coor files of vector map <JR2007_outliers@helena> is larger than
it should be (839 bytes excess)
WARNING: Unable to display areas, topology not available
Looking at the manual it seems that the table is needed only for QGIS
output where
the elevation is stored as category. So I am wondering whether this is
a bug
that needs to be fixed or whether the v.* lidar modules require points
with elevation stored in DBF (which is a problem for large datasets) -
if that is
the case it should be probably mentioned in the man page.
These modules would be really useful if we could get them to work,
I just tried out the updates in GRASS 7 trunk. The column option still
doesn't work right. v.surf.bspline runs without complaining if you specify
a column, but the output is a flat map that does not use the column values
for interpolation.
Also, the layer selector no longer gives a -1 option in the GUI. So you
cannot use a 3D map either.