[GRASS-dev] [GRASS GIS] #548: grass7: v.reclass segfaults on string column

#548: grass7: v.reclass segfaults on string column
-------------------------+--------------------------------------------------
Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: major | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------
With today's devel7 trunk, v.reclass segfaults when trying to use a string
column as reclass criterium. Using nc_spm_06 demo data:

v.reclass in=boundary_county out=test_recl column=DOT_GROUP_

segfaults, whereas

v.reclass in=boundary_county out=test_recl2 column=DOT_COUNTY

works as expected (DOT_COUNTY is integer).

In grass65svn, both commands work as expected.

This also obviously makes v.dissolve fail on such columns.

Backtrace attached.

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords:
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

For me both commands work (I did: svn up ; make distclean).

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords:
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by mlennert):

Replying to [comment:1 neteler]:
> For me both commands work

Using grass7 and the above example from the nc_spm_06 data set ?

>(I did: svn up ; make distclean).

Same here...

I'll have to see what's going on locally, then.

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: v.reclass
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Changes (by martinl):

* cc: martinl` (added)
  * keywords: => v.reclass
  * priority: major => normal

Comment:

Replying to [ticket:548 mlennert]:
> With today's devel7 trunk, v.reclass segfaults when trying to use a
string column as reclass criterium. Using nc_spm_06 demo data:
>
> v.reclass in=boundary_county out=test_recl column=DOT_GROUP_
>
> segfaults, whereas
>
> v.reclass in=boundary_county out=test_recl2 column=DOT_COUNTY
>
> works as expected (DOT_COUNTY is integer).
>
> In grass65svn, both commands work as expected.
>
> This also obviously makes v.dissolve fail on such columns.

Works fine with DBF driver. When reclassing data using string column you
get with SQLite driver know error.

{{{
DBMI-SQLite driver error:
Cannot step:
SQL logic error or missing database
}}}

No segfault here.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: v.reclass
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

For me even that works:
{{{
GRASS 7.0.svn (nc_spm_08):~/grassdata/nc_spm_08/neteler > v.reclass
in=boundary_county out=test_recl column=DOT_GROUP_
  100%
Building topology for vector map <test_recl>...
Registering primitives...
2836 primitives registered
268782 vertices registered
Building areas...
  100%
926 areas built
130 isles built
Attaching islands...
  100%
Attaching centroids...
  100%
Number of nodes: 2040
Number of primitives: 2836
Number of points: 0
Number of lines: 0
Number of boundaries: 1910
Number of centroids: 926
Number of areas: 926
Number of isles: 130
v.reclass complete. 926 features reclassed.

GRASS 7.0.svn (nc_spm_08):~/grassdata/nc_spm_08/neteler > db.connect -p
driver:sqlite
database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
schema:
group:
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: v.reclass
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by martinl):

Replying to [comment:4 neteler]:
> For me even that works:
> {{{
> GRASS 7.0.svn (nc_spm_08):~/grassdata/nc_spm_08/neteler > v.reclass
in=boundary_county out=test_recl column=DOT_GROUP_
> GRASS 7.0.svn (nc_spm_08):~/grassdata/nc_spm_08/neteler > db.connect -p
> driver:sqlite
> database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
> schema:
> group:
> }}}

{{{
v.db.connect boundary_county -g
}}}

?

I suppose DBF driver. Then it works... The key is the driver used for
input vector map.

Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

On Tue, Aug 4, 2009 at 8:54 AM, GRASS GIS<trac@osgeo.org> wrote:

#548: grass7: v.reclass segfaults on string column

...

{{{
v.db.connect boundary_county -g
}}}

?

I suppose DBF driver. Then it works... The key is the driver used for
input vector map.

Good point, indeed I was reading it with @neteler from a DBF mapset
(my bad, sorry). Copying it over into the SQLite DB fails however:

1. I made a copy of "boundary_county" in my own mapset "neteler" (DBF)
   from PERMANENT
2. Then I switched into the sqlite mapset and run:

g.copy vect=boundary_county@neteler,boundary_county
Program received signal SIGSEGV, Segmentation fault.

#gdb:
Program received signal SIGSEGV, Segmentation fault.
0x00007f979f9a3d69 in rtree_load_from_sidx (fp=0x7fffa8a8c5c8,
rootpos=179105, t=0xe67010, off_t_size=4)
    at spindex_rw.c:735
735 newnode->branch[j].child = NULL;

(gdb) bt
#0 0x00007f979f9a3d69 in rtree_load_from_sidx (fp=0x7fffa8a8c5c8,
rootpos=179105, t=0xe67010, off_t_size=4)
    at spindex_rw.c:735
#1 0x00007f979f9a4228 in dig_Rd_spidx (fp=0x7fffa8a8c5c8,
Plus=0x7fffa8a8c3b0) at spindex_rw.c:843
#2 0x00007f97a065dfff in Vect_open_sidx (Map=0x7fffa8a8c3a0, mode=1)
at open.c:891
#3 0x00007f97a065ca70 in Vect__open_old (Map=0x7fffa8a8c3a0,
name=0xe62bc0 "boundary_county",
    mapset=0xe5e650 "sqlite", update=1, head_only=1) at open.c:262
#4 0x00007f97a065d295 in Vect_open_update_head (Map=0x7fffa8a8c3a0,
name=0xe62bc0 "boundary_county",
    mapset=0xe5e650 "sqlite") at open.c:522
#5 0x00007f97a065760a in Vect_copy (in=0x7fffa8a891d0
"boundary_county", mapset=0xe635f0 "neteler",
    out=0xe62bc0 "boundary_county") at map.c:237
#6 0x0000000000401b5e in do_copy (n=2, old=0xe62ae0
"boundary_county@neteler", mapset=0xe62f20 "neteler",
    new=0xe62bc0 "boundary_county") at do_copy.c:27
#7 0x00000000004019a5 in main (argc=3, argv=0x7fffa8a8f428) at main.c:93
}}}

It is unrelated hence I don't clutter the trac ticket.
Once this works, I continue with the ticket.

Markus

I will look into it, it's the new spatial index.

Markus M

Markus Neteler wrote:

On Tue, Aug 4, 2009 at 8:54 AM, GRASS GIS<trac@osgeo.org> wrote:
  

#548: grass7: v.reclass segfaults on string column
    

...
  

{{{
v.db.connect boundary_county -g
}}}

?

I suppose DBF driver. Then it works... The key is the driver used for
input vector map.
    
Good point, indeed I was reading it with @neteler from a DBF mapset
(my bad, sorry). Copying it over into the SQLite DB fails however:

1. I made a copy of "boundary_county" in my own mapset "neteler" (DBF)
   from PERMANENT
2. Then I switched into the sqlite mapset and run:

g.copy vect=boundary_county@neteler,boundary_county
Program received signal SIGSEGV, Segmentation fault.

#gdb:
Program received signal SIGSEGV, Segmentation fault.
0x00007f979f9a3d69 in rtree_load_from_sidx (fp=0x7fffa8a8c5c8,
rootpos=179105, t=0xe67010, off_t_size=4)
    at spindex_rw.c:735
735 newnode->branch[j].child = NULL;

(gdb) bt
#0 0x00007f979f9a3d69 in rtree_load_from_sidx (fp=0x7fffa8a8c5c8,
rootpos=179105, t=0xe67010, off_t_size=4)
    at spindex_rw.c:735
#1 0x00007f979f9a4228 in dig_Rd_spidx (fp=0x7fffa8a8c5c8,
Plus=0x7fffa8a8c3b0) at spindex_rw.c:843
#2 0x00007f97a065dfff in Vect_open_sidx (Map=0x7fffa8a8c3a0, mode=1)
at open.c:891
#3 0x00007f97a065ca70 in Vect__open_old (Map=0x7fffa8a8c3a0,
name=0xe62bc0 "boundary_county",
    mapset=0xe5e650 "sqlite", update=1, head_only=1) at open.c:262
#4 0x00007f97a065d295 in Vect_open_update_head (Map=0x7fffa8a8c3a0,
name=0xe62bc0 "boundary_county",
    mapset=0xe5e650 "sqlite") at open.c:522
#5 0x00007f97a065760a in Vect_copy (in=0x7fffa8a891d0
"boundary_county", mapset=0xe635f0 "neteler",
    out=0xe62bc0 "boundary_county") at map.c:237
#6 0x0000000000401b5e in do_copy (n=2, old=0xe62ae0
"boundary_county@neteler", mapset=0xe62f20 "neteler",
    new=0xe62bc0 "boundary_county") at do_copy.c:27
#7 0x00000000004019a5 in main (argc=3, argv=0x7fffa8a8f428) at main.c:93
}}}

It is unrelated hence I don't clutter the trac ticket.
Once this works, I continue with the ticket.

Markus
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Markus Neteler wrote:

#gdb:
Program received signal SIGSEGV, Segmentation fault.
0x00007f979f9a3d69 in rtree_load_from_sidx (fp=0x7fffa8a8c5c8,
rootpos=179105, t=0xe67010, off_t_size=4)
    at spindex_rw.c:735
735 newnode->branch[j].child = NULL;
  

Fixed in r38615.

Markus M

#548: grass7: v.reclass segfaults on string column
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Resolution: | Keywords: v.reclass
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

After the fix in the spatial index I can reproduce the problem with
SQLite:

{{{
g.gisenv set=DEBUG=3
v.reclass in=boundary_county out=test_recl column=DOT_GROUP_ --o
...
db_driver_open_database()
D3/3: name = '/home/neteler/grassdata/nc_spm_08/sqlite/sqlite.db'
D2/3: name2 = '/home/neteler/grassdata/nc_spm_08/sqlite/sqlite.db'
D3/3: SQL: SELECT cat, DOT_GROUP_ FROM boundary_county ORDER BY
DOT_GROUP_
D3/3: Escaped SQL: SELECT cat, DOT_GROUP_ FROM boundary_county ORDER BY
DOT_GROUP_
D3/3: describe_table()
D3/3: ncols = 2
D3/3: sqltype = 3
D3/3: litetype = 1
D3/3: sqltype = 1
D3/3: litetype = 3
D3/3: nkcols = 2
D3/3: sqltype = 3
D3/3: litetype = 1
D2/3: col: cat, nkcols 0, litetype : 1, sqltype 3
D3/3: sqltype = 1
D3/3: litetype = 3
D2/3: col: DOT_GROUP_, nkcols 1, litetype : 3, sqltype 1
D3/3: 926 rows selected
D3/3: key type = 15
D3/3: execute: create table test_recl (cat integer, DOT_GROUP_
varchar(20))
D3/3: db__create_index()
D3/3: SQL: create unique index test_recl_cat on test_recl ( cat )
D3/3: fetch row = -1
DBMI-SQLite driver error:
Cannot step:
SQL logic error or missing database

ERROR: Unable to fetch data from table <boundary_county>
}}}

No mention on http://www.sqlite.org/omitted.html, so I don't know why this
statement is rejected.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Resolution: | Keywords: v.reclass sqlite
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Changes (by mlennert):

  * keywords: v.reclass => v.reclass sqlite
  * component: Vector => Database
  * summary: grass7: v.reclass segfaults on string column => grass7 -
              v.reclass with sqlite driver: "Cannot step: SQL
              logic error or missing database"

Comment:

Replying to [comment:6 neteler]:
> After the fix in the spatial index I can reproduce the problem with
SQLite:

I confirm: no more segfault, don't where that came from, but same error as
Markus and Martin. Changing title of bug

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Resolution: | Keywords: v.reclass sqlite
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by wilsonadam):

I can confirm this in GRASS 6.4.0RC5 (2009) with a string field (using my
data, but it does the same thing).

{{{
v.reclass in=pgrid out=pgrid2 type=centroid column=gridID

DBMI-SQLite driver error:
Cannot step:
SQL logic error or missing database

ERROR: Unable to fetch data from table <pgrid>
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
------------------------------+---------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Keywords: v.reclass sqlite | Platform: Unspecified
      Cpu: Unspecified |
------------------------------+---------------------------------------------
Changes (by neteler):

* cc: martinl` (removed)

Comment:

Still failing in current GRASS 7 SVN:

{{{
GRASS 7.0.svn (nc_spm_08@grass70):~ > v.reclass in=myboundary_county
out=test_recl column=DOT_GROUP_
DBMI-SQLite driver error:
Cannot step:
SQL logic error or missing database

ERROR: Unable to fetch data from table <myboundary_county>
GRASS 7.0.svn (nc_spm_08@grass70):~ > v.db.connect -p myboundary_county
Vector map <myboundary_county> is connected by:
layer <1/boundary_county> table <myboundary_county> in database
</home/neteler/grassdata/nc_spm_08/grass70/sqlite.db> through driver
<sqlite> with key <cat>
}}}

It likely affects #1418 (v.dissolve failure) which suppresses error
messages from v.reclass.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
------------------------------+---------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Keywords: v.reclass sqlite | Platform: Unspecified
      Cpu: Unspecified |
------------------------------+---------------------------------------------

Comment(by mmetz):

Replying to [comment:9 neteler]:
> Still failing in current GRASS 7 SVN:

and the two other branches.

>
> It likely affects #1418 (v.dissolve failure) which suppresses error
messages from v.reclass.

Debugging a bit, the problem of v.reclass with the sqlite driver seems to
be that the same database is opened twice (once by each driver instance),
which causes database locking, therefore it is not possible to use one
driver instance to fetch records from one table and use another driver
instance to insert records into another table, all in the same sqlite
database.

BTW, there is a test in lib/db if a database is already opened when
opening a database, this test is called quite often but it never
complains.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
------------------------------+---------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Keywords: v.reclass sqlite | Platform: Unspecified
      Cpu: Unspecified |
------------------------------+---------------------------------------------

Comment(by mmetz):

Replying to [comment:10 mmetz]:
>
> Debugging a bit, the problem of v.reclass with the sqlite driver seems
to be that the same database is opened twice (once by each driver
instance), which causes database locking, therefore it is not possible to
use one driver instance to fetch records from one table and use another
driver instance to insert records into another table, all in the same
sqlite database.

Now it's possible, the other driver just has to wait a little bit until
the first driver is finished.

Fixed in r45156, r48157, r48158 (7.0, 6.5, 6.4.2, respectively).

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
------------------------------+---------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Database | Version: svn-trunk
Keywords: v.reclass sqlite | Platform: Unspecified
      Cpu: Unspecified |
------------------------------+---------------------------------------------

Comment(by mmetz):

Replying to [comment:11 mmetz]:
> Replying to [comment:10 mmetz]:
> >
> > Debugging a bit, the problem of v.reclass with the sqlite driver seems
to be that the same database is opened twice (once by each driver
instance), which causes database locking, therefore it is not possible to
use one driver instance to fetch records from one table and use another
driver instance to insert records into another table, all in the same
sqlite database.
>
> Now it's possible, the other driver just has to wait a little bit until
the first driver is finished.
>
Correction, only now (trunk r48333) it's possible. There were more bugs,
both in v.reclass and in the sqlite driver.

A test, within the sqlite mapset of nc_spm_08:
{{{
g.copy vect=zipcodes_wake,my_zipcodes_wake
v.dissolve input=my_zipcodes_wake@sqlite layer=1 column=ZIPNAME
output=zip_names
}}}

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
------------------------------+---------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: Database | Version: svn-releasebranch64
Keywords: v.reclass sqlite | Platform: Unspecified
      Cpu: Unspecified |
------------------------------+---------------------------------------------
Changes (by neteler):

  * version: svn-trunk => svn-releasebranch64
  * milestone: 7.0.0 => 6.4.3

Comment:

Backport of trunk r48333 to GRASS 6.5 in r51261. Can I backport to 6.4 as
well?
The zipcodes_wake example in comment 11 succeeds.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 6.4.3
Component: Database | Version: svn-releasebranch64
Resolution: fixed | Keywords: v.reclass sqlite
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Changes (by neteler):

  * status: new => closed
  * resolution: => fixed

Comment:

Backported to 6.4 in r52912 by mmetz. Closing.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

#548: grass7 - v.reclass with sqlite driver: "Cannot step: SQL logic error or
missing database"
--------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 6.4.3
Component: Database | Version: svn-releasebranch64
Resolution: fixed | Keywords: v.reclass sqlite
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------

Comment(by mmetz):

Replying to [comment:14 neteler]:
> Backported to 6.4 in r52912 by mmetz. Closing.

That was only the sqlite driver. v.reclass itself is now fixed in r52955.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/548#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>