[GRASS-dev] Import PostGIS vector layer into SQLite database

Hi all,

I am working with vector layers (points) composed of more than 100 000 geometries. These layers are originally in a PostGis database and I import them with v.in.ogr, using a formulation like:

v.in.ogr \
    input="PG:host=127.0.0.1 dbname=gisdb user=user password=secret" \
    layer=schema.points \
    output=points \
    -o --overwrite

This command is successful, however, operations with the resulting points layer are remarkably slow. For instance, this layer must be later loaded in R and that operation alone is taking over 3 hours.

After some back-and-forth testing, I identified the problem: v.in.ogr is not saving its output in the internal GRASS SQLite database. Knowing this, I can create a copy of the original output, that is automatically inserted into the SQLite database, e.g.:

g.copy vector=points,points_mirror

Importing the points_mirror layer to R takes 3 minutes instead of 3 hours, for instance.

While mirroring vectors imported with v.in.ogr solves the issue, it is an inelegant solution that also takes up unnecessary space. I would prefer to instruct v.in.ogr somehow to force it to use the SQLite database. I could not find any reference in the manual, is it possible to do?

Thank you.

···

Luís Moreira de Sousa

Email: luis.de.sousa@protonmail.ch

RingID: ring:7ca91d83f4f9dec82fec9f1144b8e5c1ef2a110c

URL: https://ldesousa.github.io

Sent with ProtonMail Secure Email.

On Wed, Mar 27, 2019 at 5:42 PM Luí­s Moreira de Sousa <luis.de.sousa@protonmail.ch> wrote:

Hi all,

I am working with vector layers (points) composed of more than 100 000 geometries. These layers are originally in a PostGis database and I import them with v.in.ogr, using a formulation like:

v.in.ogr
input=“PG:host=127.0.0.1 dbname=gisdb user=user password=secret”
layer=schema.points
output=points
-o --overwrite

This command is successful, however, operations with the resulting points layer are remarkably slow. For instance, this layer must be later loaded in R and that operation alone is taking over 3 hours.

After some back-and-forth testing, I identified the problem: v.in.ogr is not saving its output in the internal GRASS SQLite database. Knowing this, I can create a copy of the original output, that is automatically inserted into the SQLite database, e.g.:

g.copy vector=points,points_mirror

Importing the points_mirror layer to R takes 3 minutes instead of 3 hours, for instance.

Which GRASS version are you using? This should not happen with recent versions of GRASS 7.4 and 7.6.

Markus M

While mirroring vectors imported with v.in.ogr solves the issue, it is an inelegant solution that also takes up unnecessary space. I would prefer to instruct v.in.ogr somehow to force it to use the SQLite database. I could not find any reference in the manual, is it possible to do?

Thank you.


Luís Moreira de Sousa
Email: luis.de.sousa@protonmail.ch
RingID: ring:7ca91d83f4f9dec82fec9f1144b8e5c1ef2a110c
URL: https://ldesousa.github.io

Sent with ProtonMail Secure Email.


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Hi Markus,

I am working with GRASS 7.6 installed from the ubuntugis PPA.

Cheers.

···

On Wed, Mar 27, 2019 at 5:42 PM Luí­s Moreira de Sousa <luis.de.sousa@protonmail.ch> wrote:

Hi all,

I am working with vector layers (points) composed of more than 100 000 geometries. These layers are originally in a PostGis database and I import them with v.in.ogr, using a formulation like:

v.in.ogr
input=“PG:host=127.0.0.1 dbname=gisdb user=user password=secret”
layer=schema.points
output=points
-o --overwrite

This command is successful, however, operations with the resulting points layer are remarkably slow. For instance, this layer must be later loaded in R and that operation alone is taking over 3 hours.

After some back-and-forth testing, I identified the problem: v.in.ogr is not saving its output in the internal GRASS SQLite database. Knowing this, I can create a copy of the original output, that is automatically inserted into the SQLite database, e.g.:

g.copy vector=points,points_mirror

Importing the points_mirror layer to R takes 3 minutes instead of 3 hours, for instance.

Which GRASS version are you using? This should not happen with recent versions of GRASS 7.4 and 7.6.

Markus M

While mirroring vectors imported with v.in.ogr solves the issue, it is an inelegant solution that also takes up unnecessary space. I would prefer to instruct v.in.ogr somehow to force it to use the SQLite database. I could not find any reference in the manual, is it possible to do?

Thank you.


Luís Moreira de Sousa
Email: luis.de.sousa@protonmail.ch
RingID: ring:7ca91d83f4f9dec82fec9f1144b8e5c1ef2a110c
URL: https://ldesousa.github.io

Sent with ProtonMail Secure Email.


grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev