[GRASS-dev] possible error in WIKI could cause data corruption

GRASS GIS GRASS GIS developer
1

I don’t know if this is bad advice in the GRASS WIKI or a bug in a GRASS database module. I have some vector files that were somehow imported into a dbf DBMS last year. I wanted to switch them all to sqlite. The GRASS WIKI advises…

Convert automatically all GRASS 6 vector maps to GRASS 7 (including attribute transfer from DBF to SQLite)

To become readable in GRASS 7, all vector maps in a mapset need to be updated to the improved vector file format (i.e., rebuild the topology with v.build.all along with the change to the SQLite database backend (SQLite is the new default in GRASS 7 instead of DBF in GRASS 6, using db.connect and v.db.reconnect.all).

Run this in GRASS GIS 7:

first rebuild topology for all vector maps in current mapset

v.build.all

define new default DB connection (switch from DBF to SQLite)db.connect -d

transfer all attribute tables from DBF to SQLite and clean old DBF tablesv.db.reconnect.all -cd

The vectors already had GRASS 7 topology, but I ran v.rebuild.all just to be sure. No problem.

db.connect -d ran with no problem

But v.db.reconnect.all -cd did not create tables in the sqlite database and corrupted the existing connection of the first table it hit. Fortunately, I made a backup of the mapset before I started this.

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all -cd

Reconnecting vector map catastro@surveyboundaries (1 of 9)…

Copying table to target database…
Traceback (most recent call last):
File “/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable”, line 99, in
main()
File “/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable”, line 77, in main
used = grass.db.db_table_in_vector(table)
File “/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py”, line 184, in db_table_in_vector
from vector import vector_db
ImportError: No module named vector
ERROR: Unable to drop table
GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ >

Michael

C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)

www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

2

On Sat, Sep 5, 2015 at 7:51 PM, Michael Barton <Michael.Barton@asu.edu> wrote:

I don’t know if this is bad advice in the GRASS WIKI or a bug in a GRASS
database module. I have some vector files that were somehow imported into a
dbf DBMS last year. I wanted to switch them all to sqlite. The GRASS WIKI
advises…

Convert automatically all GRASS 6 vector maps to GRASS 7 (including
attribute transfer from DBF to SQLite)

https://grasswiki.osgeo.org/wiki/Convert_all_GRASS_6_vector_maps_to_GRASS_7

...

The vectors already had GRASS 7 topology, but I ran v.rebuild.all just to be
sure. No problem.

db.connect -d ran with no problem

But v.db.reconnect.all -cd did not create tables in the sqlite database

But the sqlite file is there?

and corrupted the existing connection of the first table it hit. Fortunately, I
made a backup of the mapset before I started this.

Perhaps a problem on Mac?
This is strange - I have converted so many mapsets like that ..., even
with lots of vector maps inside.
The wiki page is there for a long time, I didn't see such reports so far.

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all
-cd

--------------------------------------------------------------------------------
Reconnecting vector map <catastro@surveyboundaries> (1 of 9)...
--------------------------------------------------------------------------------
Copying table <catastro> to target database...
Traceback (most recent call last):
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 99, in <module>
    main()
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 77, in main
    used = grass.db.db_table_in_vector(table)
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py",
line 184, in db_table_in_vector
    from vector import vector_db
ImportError: No module named vector

Maybe something got broken in GRASS 7.1?

I would definitely use 7.0 for such operations.

Markus

ERROR: Unable to drop table <catastro>
GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ >

Michael
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

3

On Sep 5, 2015, at 11:26 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Sep 5, 2015 at 7:51 PM, Michael Barton <Michael.Barton@asu.edu> wrote:

I don’t know if this is bad advice in the GRASS WIKI or a bug in a GRASS
database module. I have some vector files that were somehow imported into a
dbf DBMS last year. I wanted to switch them all to sqlite. The GRASS WIKI
advises…

Convert automatically all GRASS 6 vector maps to GRASS 7 (including
attribute transfer from DBF to SQLite)

Convert all GRASS 6 vector maps to GRASS 7 - GRASS-Wiki

...

The vectors already had GRASS 7 topology, but I ran v.rebuild.all just to be
sure. No problem.

db.connect -d ran with no problem

But v.db.reconnect.all -cd did not create tables in the sqlite database

But the sqlite file is there?

yes

and corrupted the existing connection of the first table it hit. Fortunately, I
made a backup of the mapset before I started this.

Perhaps a problem on Mac?
This is strange - I have converted so many mapsets like that ..., even
with lots of vector maps inside.
The wiki page is there for a long time, I didn't see such reports so far.

I had to make a new mapset and then copy the files from the old mapset to the new one. All went into sqlite fine that way.

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all
-cd

--------------------------------------------------------------------------------
Reconnecting vector map <catastro@surveyboundaries> (1 of 9)...
--------------------------------------------------------------------------------
Copying table <catastro> to target database...
Traceback (most recent call last):
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 99, in <module>
   main()
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 77, in main
   used = grass.db.db_table_in_vector(table)
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py",
line 184, in db_table_in_vector
   from vector import vector_db
ImportError: No module named vector

Maybe something got broken in GRASS 7.1?

I would definitely use 7.0 for such operations.

I’ll check

Michael

Markus

ERROR: Unable to drop table <catastro>
GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ >

Michael
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

4

So I decided to try this with 7.0 and looked again at the mapset. It turns out that the folder is there but not the sqlite.db.

So perhaps we need to add to the WIKI that you might need to create the sqlite.db

However, when I tried to create the database, it doesn’t seem to work.

db.createdb driver=sqlite database=$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db
dbmi: db_create_database() not implemented
dbmi: db_create_database() not implemented

No error. Just no database.

Michael
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

On Sep 5, 2015, at 11:26 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Sep 5, 2015 at 7:51 PM, Michael Barton <Michael.Barton@asu.edu> wrote:

I don’t know if this is bad advice in the GRASS WIKI or a bug in a GRASS
database module. I have some vector files that were somehow imported into a
dbf DBMS last year. I wanted to switch them all to sqlite. The GRASS WIKI
advises…

Convert automatically all GRASS 6 vector maps to GRASS 7 (including
attribute transfer from DBF to SQLite)

Convert all GRASS 6 vector maps to GRASS 7 - GRASS-Wiki

...

The vectors already had GRASS 7 topology, but I ran v.rebuild.all just to be
sure. No problem.

db.connect -d ran with no problem

But v.db.reconnect.all -cd did not create tables in the sqlite database

But the sqlite file is there?

and corrupted the existing connection of the first table it hit. Fortunately, I
made a backup of the mapset before I started this.

Perhaps a problem on Mac?
This is strange - I have converted so many mapsets like that ..., even
with lots of vector maps inside.
The wiki page is there for a long time, I didn't see such reports so far.

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all
-cd

--------------------------------------------------------------------------------
Reconnecting vector map <catastro@surveyboundaries> (1 of 9)...
--------------------------------------------------------------------------------
Copying table <catastro> to target database...
Traceback (most recent call last):
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 99, in <module>
   main()
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 77, in main
   used = grass.db.db_table_in_vector(table)
File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py",
line 184, in db_table_in_vector
   from vector import vector_db
ImportError: No module named vector

Maybe something got broken in GRASS 7.1?

I would definitely use 7.0 for such operations.

Markus

ERROR: Unable to drop table <catastro>
GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ >

Michael
____________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

5

On Sep 6, 2015 1:10 AM, “Michael Barton” <Michael.Barton@asu.edu> wrote:

So I decided to try this with 7.0 and looked again at the mapset. It turns out that the folder is there but not the sqlite.db.

So perhaps we need to add to the WIKI that you might need to create the sqlite.db

To my knowledge GRASS is designed to generate the file automatically, so the absence in your car is a bug.

However, when I tried to create the database, it doesn’t seem to work.

db.createdb driver=sqlite database=$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db
dbmi: db_create_database() not implemented
dbmi: db_create_database() not implemented

No error. Just no database.

I never used that for above mentioned reason…

Markus

Michael

C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

On Sep 5, 2015, at 11:26 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Sep 5, 2015 at 7:51 PM, Michael Barton <Michael.Barton@asu.edu> wrote:

I don’t know if this is bad advice in the GRASS WIKI or a bug in a GRASS
database module. I have some vector files that were somehow imported into a
dbf DBMS last year. I wanted to switch them all to sqlite. The GRASS WIKI
advises…

Convert automatically all GRASS 6 vector maps to GRASS 7 (including
attribute transfer from DBF to SQLite)

https://grasswiki.osgeo.org/wiki/Convert_all_GRASS_6_vector_maps_to_GRASS_7

…

The vectors already had GRASS 7 topology, but I ran v.rebuild.all just to be
sure. No problem.

db.connect -d ran with no problem

But v.db.reconnect.all -cd did not create tables in the sqlite database

But the sqlite file is there?

and corrupted the existing connection of the first table it hit. Fortunately, I
made a backup of the mapset before I started this.

Perhaps a problem on Mac?
This is strange - I have converted so many mapsets like that …, even
with lots of vector maps inside.
The wiki page is there for a long time, I didn’t see such reports so far.

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all
-cd

Reconnecting vector map catastro@surveyboundaries (1 of 9)…

Copying table to target database…
Traceback (most recent call last):
File
“/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable”,
line 99, in
main()
File
“/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable”,
line 77, in main
used = grass.db.db_table_in_vector(table)
File
“/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py”,
line 184, in db_table_in_vector
from vector import vector_db
ImportError: No module named vector

Maybe something got broken in GRASS 7.1?

I would definitely use 7.0 for such operations.

Markus

ERROR: Unable to drop table
GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ >

Michael

C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Head, Graduate Faculty in Complex Adaptive Systems Science
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

6

Hi,

I just remembered a change in trunk (see below):

On Sat, Sep 5, 2015 at 7:51 PM, Michael Barton <Michael.Barton@asu.edu> wrote:
...

GRASS 7.1.svn (Spain_fieldwork_medlands_ERTS89_Z30):~ > v.db.reconnect.all -cd
--------------------------------------------------------------------------------
Reconnecting vector map <catastro@surveyboundaries> (1 of 9)...
--------------------------------------------------------------------------------
Copying table <catastro> to target database...
Traceback (most recent call last):
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 99, in <module>
    main()
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/scripts/db.droptable",
line 77, in main
    used = grass.db.db_table_in_vector(table)
  File
"/Applications/GRASS/GRASS-7.1.app/Contents/MacOS/etc/python/grass/script/db.py",
line 184, in db_table_in_vector
    from vector import vector_db
ImportError: No module named vector
ERROR: Unable to drop table <catastro>

I checked the changes to be backported (still I would appreciate that
the respective devs take care themselves):

On Mon, Aug 17, 2015 at 1:47 PM, <svn_grass@osgeo.org> wrote:

Author: zarch
Date: 2015-08-17 04:47:30 -0700 (Mon, 17 Aug 2015)
New Revision: 65947

Modified:
   grass/trunk/lib/python/script/db.py
Log:
script.db: Fix absolute import inside function

Modified: grass/trunk/lib/python/script/db.py

--- grass/trunk/lib/python/script/db.py 2015-08-17 08:29:14 UTC (rev 65946)
+++ grass/trunk/lib/python/script/db.py 2015-08-17 11:47:30 UTC (rev 65947)
@@ -181,7 +181,7 @@

     :param str table: name of table to query
     """
- from vector import vector_db
+ from .vector import vector_db
     nuldev = file(os.devnull, 'w')
     used =
     vects = list_strings('vect')

_______________________________________________
grass-commit mailing list
grass-commit@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-commit

If your local GRASS 7.trunk version is newer than 2015-08-17,
- maybe r65947 is causing the issue
else
- maybe r65947 is solving the issue
?

Markus