[GRASS5] sqlite driver - remarks and questions

Grass 6.1 CVS 2005-11-16 built with sqlite 3.0.8-3, Ubuntu Hoary.

Using the CONTOURL.shp from VMAP0 dataset:

maciek@sorbus:~$ ogrinfo -al -so CONTOURL.shp
INFO: Open of 'CONTOURL.shp' using driver `ESRI Shapefile' successful.

Layer name: CONTOURL
Geometry: Line String
Feature Count: 173863
Extent: (-31.252195, 30.000000) - (179.999420, 81.845779)
Layer SRS WKT:
(unknown)
F_CODEDESC: String (50.0)
ID: Real (11.0)
HQC: Integer (6.0)
F_CODE: String (5.0)
TILE_ID: Integer (6.0)
EDG_ID: Real (11.0)
HQCDESC: String (50.0)
ZV2DESC: String (50.0)
ZV2: Integer (6.0)

NOTES AND QUESTIONS:

1. Importing with v.in.ogr to sqlite took over 37 minutes compared to
about 41 seconds when using dbf driver with the same file. Could be
faster?

2. d.what.vect is significantly faster on sqlite than on dbf driver.
Tried on the same dataset, the response time is great.

3. But, the initial "Building spatial index" is as long for sqlite as
for dbf, and occures each time d.what.vect is invoked. Is it
unavoidable?

3. Exporting from sqlite back to shapefile with v.out.ogr is extremely
fast when comapared to dbf - about 60 (sixty) times faster, wow!

4. But, it fails to complete! At about 80% all my RAM (1GB) and swap
(1GB) was eaten by sqlite process, which freezed my machine completely,
had to power-off. Memory leak? Can be fixed?

5. The sqlitebrowser, which has been mentioned on grass lists several
times, is an _almost_ decent tool for viewing sqlite database, seems
stable and fairly fast, but most of the needed editing tables
functionality is lacking and there's no functionality for sorting data.
I wouldn't depend on it for serious database handling. Does anybody know
of something similar (sql GUI mixed with a kind of a spreadsheet) but
more functional?

Info about the used CONTOURL.shp, after imported into Grass layer as
"contourl" with sqlite database:

GRASS 6.1.cvs (sqlite_ll):~ > v.db.connect -p contourl
Vector map <contourl> is connected by:
layer <1> table <contourl> in database
</home/grassdata/sqlite_ll/test/sqlite.db> through driver <sqlite> with
key <cat>

GRASS 6.1.cvs (sqlite_ll):~ > v.info contourl
+------------------------------------------------
| Layer: contourl Organization:
| Mapset: test Source Date:
| Location: sqlite_ll Name of creator:
| Database: /home/grassdata
| Title:
| Map Scale: 1:1
| Map format: native
|------------------------------------------------
| Type of Map: Vector (level: 2)
|
| Number of points: 0 Number of areas: 0
| Number of lines: 173863 Number of islands: 0
| Number of boundaries: 0 Number of faces: 0
| Number of centroids: 0 Number of kernels: 0
|
| Map is 3D: 0
| Number of dblinks: 1
|
| Projection: Latitude-Longitude (zone 0)
| N: 81.846 S: 30.000
| E: 179.999 W: -31.252
| B: 0.000 T: 0.000
|
| Digitize threshold: 0.00000
| Comments: |
|
+------------------------------------------------

GRASS 6.1.cvs (sqlite_ll):~ > v.info -c contourl
Displaying column types/names for database connection of layer 1:
INTEGER|cat
TEXT|F_CODEDESC
DOUBLE PRECISION|ID
INTEGER|HQC
TEXT|F_CODE
INTEGER|TILE_ID
DOUBLE PRECISION|EDG_ID
TEXT|HQCDESC
TEXT|ZV2DESC
INTEGER|ZV2

Maciek

P.S.
Crossposting. I thought both users and developers might be interested.
Sorry for double mail.

--------------------
W polskim Internecie s± setki milionów stron. My przekazujemy Tobie tylko najlepsze z nich!
http://katalog.epf.pl/

Hi,

An update regarding this post I sent few weeks ago.

On Tue, 22 Nov 2005 09:38:36 +0100
Maciek Sieczka <werchowyna@epf.pl> wrote:

Grass 6.1 CVS 2005-11-16 built with sqlite 3.0.8-3, Ubuntu Hoary.

Using the CONTOURL.shp from VMAP0 dataset:

maciek@sorbus:~$ ogrinfo -al -so CONTOURL.shp
INFO: Open of 'CONTOURL.shp' using driver `ESRI Shapefile' successful.

Layer name: CONTOURL
Geometry: Line String
Feature Count: 173863
Extent: (-31.252195, 30.000000) - (179.999420, 81.845779)
Layer SRS WKT:
(unknown)
F_CODEDESC: String (50.0)
ID: Real (11.0)
HQC: Integer (6.0)
F_CODE: String (5.0)
TILE_ID: Integer (6.0)
EDG_ID: Real (11.0)
HQCDESC: String (50.0)
ZV2DESC: String (50.0)
ZV2: Integer (6.0)

<snip>

3. Exporting from sqlite back to shapefile with v.out.ogr is extremely
fast when comapared to dbf - about 60 (sixty) times faster, wow!

4. But, it fails to complete! At about 80% all my RAM (1GB) and swap
(1GB) was eaten by sqlite process, which freezed my machine
completely, had to power-off. Memory leak? Can be fixed?

Today I re-tried this with newer Grass 6.1 CVS, newer system (Ubuntu
Breezy) and sqlite 3.3.2, on the same machine. This time v.out.ogr
using sqlite performed much better. I don't know what is the reason
that now it works. Couple of things changed in my setup. Here's a
summary of what I recall that changed between the two scenarios:

SCENARIO 1, 22 Nov 2005:

Ubuntu 5.04 Hoary
Grass 6.1 CVS 2005-11-16
Sqlite 3.0.8

Symptoms: v.out.ogr failed to export a 173863 vector lines layer. At
'80%' system became was completely non responsive. There wasn't any
progress. Had to reboot.

SCENARIO 2, 05 Feb 2006:

Ubuntu 5.10 Breezy
Grass 6.1 CVS 2006-02-01
Sqlite 3.3.2

Symptoms: v.out.ogr managed to export a 173863 vector lines
layer. The 100% was achieved within about 5 minutes, memory use
remained fairly small until then. However, before the whole export was
accomplished it took 90 minutes more, while memory and swap were used
100% all that time. What's nice though, is that the system remained
responsive even then, only it was slow (considering 100% swap and
memory use it was a rocket though ;)).

Maciek

--------------------
W polskim Internecie s? setki milion?w stron. My przekazujemy Tobie tylko najlepsze z nich!
http://katalog.panoramainternetu.pl/