[GRASS-user] partial vector copy to smaller territory

GRASS GRASS user
1

Hello,

What is the best and fast way to make a copy of a part of vector map (national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

2

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

3

The following partial script may be helpful (having the source map in shape format):

assume folder in home directory containing source map - in this case contours.shp# v_w,v_s,v_e,v_n are string variables with destination region# the commands create a subset defined by the destination region: contours.shp# note that the identical naming of input and output is perhaps confusing but harmless# as long as the home directory does not contain files named contours.*# —# importing required modules

import os
import sys
import subprocess
from array import array
import string
import math
from grass.script import core as grass
from grass.script import db as grass

excerpt

cmd = 'ogr2ogr -clipsrc' + ' ' + v_w + ' ' + v_s + ' ' + v_e + ' ' + v_n
cmd = cmd + ' ' + '~/contours.shp' + ' ~/folder_name/contours.shp'
subprocess.call(cmd, shell = True)
grass.run_command('v.in.ogr', input='contours.shp', layer='contours', output='contours')

Jon

On 31 Oct 2018, at 10:12, Frank David wrote:

Hello,

What is the best and fast way to make a copy of a part of vector map (national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

4

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

Cheers,

Frank

···

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

5

Hi Jon,

Thanks for your reply. It’s very interesting to make a python script, even if the input data doe not come from grass database as I would like.

Cheers,

Frank

···

Le 31/10/2018 à 12:35, Jón Eiríksson a écrit :

The following partial script may be helpful (having the source map in shape format):

assume folder in home directory containing source map - in this case contours.shp# v_w,v_s,v_e,v_n are string variables with destination region# the commands create a subset defined by the destination region: contours.shp# note that the identical naming of input and output is perhaps confusing but harmless# as long as the home directory does not contain files named contours.*# —# importing required modules

import os
import sys
import subprocess
from array import array
import string
import math
from grass.script import core as grass
from grass.script import db as grass

excerpt

cmd = 'ogr2ogr -clipsrc' + ' ' + v_w + ' ' + v_s + ' ' + v_e + ' ' + v_n
cmd = cmd + ' ' + '~/contours.shp' + ' ~/folder_name/contours.shp'
subprocess.call(cmd, shell = True)
grass.run_command('v.in.ogr', input='contours.shp', layer='contours', output='contours')

Jon

On 31 Oct 2018, at 10:12, Frank David wrote:

Hello,

What is the best and fast way to make a copy of a part of vector map (national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

6

Hei Frank,

Did you try the “spatial” option in v.in.ogr (or the r-flag) (if your data is in a shape-file to start with):

https://grass.osgeo.org/grass74/manuals/v.in.ogr.html ?

And is v.clip slower than v.overlay (that could be used as well)?

Cheers

Stefan

···

From: grass-user grass-user-bounces@lists.osgeo.org On Behalf Of Frank David
Sent: onsdag 31. oktober 2018 13:14
To: Veronica Andreo veroandreo@gmail.com
Cc: grass-user grass-user@lists.osgeo.org
Subject: Re: [GRASS-user] partial vector copy to smaller territory

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,

Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

7

Hi Stefan,

Thank you. I did not know. That’s great.

Cheers,
Frank

···

Le 31/10/2018 à 13:53, Stefan Blumentrath a écrit :

Hei Frank,

Did you try the “spatial” option in v.in.ogr (or the r-flag) (if your data is in a shape-file to start with):

https://grass.osgeo.org/grass74/manuals/v.in.ogr.html ?

And is v.clip slower than v.overlay (that could be used as well)?

Cheers

Stefan

From: grass-user grass-user-bounces@lists.osgeo.org On Behalf Of Frank David
Sent: onsdag 31. oktober 2018 13:14
To: Veronica Andreo veroandreo@gmail.com
Cc: grass-user grass-user@lists.osgeo.org
Subject: Re: [GRASS-user] partial vector copy to smaller territory

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,

Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

8

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Markus M

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

9

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
>
> Hello Vero,
>
> Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)... it's may be a bad idea to have a so heavy vector map in Grass...

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Hello Markus,

I'm using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?

Cheers,
Frank

Markus M
>
> Cheers,
>
> Frank
>
> Le 31/10/2018 à 11:26, Veronica Andreo a écrit :
>
> Check v.extract or v.clip
>
> Cheers,
> Vero
>
> El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> escribió:
>>
>> Hello,
>>
>> What is the best and fast way to make a copy of a part of vector map
>> (national) to get a smaller territory ? (partial copy).
>> Thank you for your advice !
>>
>> Frank
>> _______________________________________________
>> grass-user mailing list
>> grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
>> https://lists.osgeo.org/mailman/listinfo/grass-user
>
> _______________________________________________
> grass-user mailing list
> grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/grass-user

10

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min vs 1 hour

Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.

Markus M

Cheers,
Frank

Markus M

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

11

Hi Frank,

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min vs 1 hour

Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here > https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

Cheers,
Frank

12

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr> wrote:

Le 02/11/2018 à 14:43, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min vs 1 hour

Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here > https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

“here” means a laptop with Intel Core i7 4th generation and standard SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing attributes, depending on the database connection. I am using the default sqlite database.

You could use top to check if v.extract is running at 100%.

Markus M

Cheers,
Frank

Markus M

Cheers,
Frank

Markus M

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

13

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
>
> Le 02/11/2018 à 14:43, Markus Metz a écrit :
>
> On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
> >
> > Le 31/10/2018 à 20:57, Markus Metz a écrit :
> >
> > Hi Frank,
> >
> > On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
> > >
> > > Hello Vero,
> > >
> > > Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)... it's may be a bad idea to have a so heavy vector map in Grass...
> >
> > this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?
> >
> > Hello Markus,
> >
> > I'm using 7.4.1 on debian9
> >
> > But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?
>
> This is possible, but it should not slow it down that much: 10 min vs 1 hour
>
> Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.
>
> Markus,
>
> The data is CorineLandCover for france. You can download 400mo here > https://we.tl/t-tvldlmkVw3
>
> First I have imported the shape in PERMANENT
>
> and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12
>
> Hope it helps you...

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

"here" means a laptop with Intel Core i7 4th generation and standard SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing attributes, depending on the database connection. I am using the default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I'm trying to do again the operation on local machine. I want fisrt, create a vector with only selected data (on full France area). But I've met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area where=CODE_12 IN (111,112,121,311,312,313,324) output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>...
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
'IN') in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
'IN') in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized "IN" sql condition. If I force layer=-1, it runs bur the output vector is the same as input one.

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Markus M
>
> Cheers,
> Frank
>
> Markus M
> >
> > Cheers,
> > Frank
> >
> > Markus M
> > >
> > > Cheers,
> > >
> > > Frank
> > >
> > > Le 31/10/2018 à 11:26, Veronica Andreo a écrit :
> > >
> > > Check v.extract or v.clip
> > >
> > > Cheers,
> > > Vero
> > >
> > > El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> escribió:
> > >>
> > >> Hello,
> > >>
> > >> What is the best and fast way to make a copy of a part of vector map
> > >> (national) to get a smaller territory ? (partial copy).
> > >> Thank you for your advice !
> > >>
> > >> Frank
> > >> _______________________________________________
> > >> grass-user mailing list
> > >> grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
> > >> https://lists.osgeo.org/mailman/listinfo/grass-user
> > >
> > > _______________________________________________
> > > grass-user mailing list
> > > grass-user@lists.osgeo.org <mailto:grass-user@lists.osgeo.org>
> > > https://lists.osgeo.org/mailman/listinfo/grass-user

14

On Mon, Nov 5, 2018 at 2:23 PM Frank David <frank.david@geophom.fr> wrote:

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr> wrote:

Le 02/11/2018 à 14:43, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)… it’s may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min vs 1 hour

Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here > https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

“here” means a laptop with Intel Core i7 4th generation and standard SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing attributes, depending on the database connection. I am using the default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I’m trying to do again the operation on local machine. I want fisrt, create a vector with only selected data (on full France area). But I’ve met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area where=CODE_12 IN (111,112,121,311,312,313,324) output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>…
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized “IN” sql condition. If I force layer=-1, it runs bur the output vector is the same as input one.

DBF does not recognize the “IN”. It is highly recommended to leave the GRASS database connection at the default sqlite. You can reset the GRASS database connection to the default with
db.connect -d
After that verify with
db.connect -p

Markus M

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Markus M

Cheers,
Frank

Markus M

Cheers,
Frank

Markus M

Cheers,

Frank

Le 31/10/2018 à 11:26, Veronica Andreo a écrit :

Check v.extract or v.clip

Cheers,
Vero

El mié., 31 oct. 2018 7:20, Frank David <frank.david@geophom.fr> escribió:

Hello,

What is the best and fast way to make a copy of a part of vector map
(national) to get a smaller territory ? (partial copy).
Thank you for your advice !

Frank

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

15

On Mon, Nov 5, 2018 at 2:32 PM Frank David <frank.david@geophom.fr> wrote:

Le 02/11/2018 à 22:27, Markus Metz a écrit :
On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr> wrote:
> Le 02/11/2018 à 14:43, Markus Metz a écrit :
> On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr> wrote:
> > Le 31/10/2018 à 20:57, Markus Metz a écrit :
> > On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr> wrote:

...

> > > Thanks for your reply. It took almost one hour to extract my area (I guess because of topology).

...

I'm trying to do again the operation on local machine.

...

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area where=CODE_12 IN (111,112,121,311,312,313,324) output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>...
DBMI-DBF

...

--> the use of the DBF driver is the reason why it is so slow at your end.
As soon as you switch to the SQLite backend (see Markus Metz'
suggestion from today), it will be much faster.

Just for clarification,

markusN

16

Le 05/11/2018 à 14:23, Frank David a écrit :

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
>
> Le 02/11/2018 à 14:43, Markus Metz a écrit :
>
> On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
> >
> > Le 31/10/2018 à 20:57, Markus Metz a écrit :
> >
> > Hi Frank,
> >
> > On Wed, Oct 31, 2018 at 1:39 PM Frank David <frank.david@geophom.fr <mailto:frank.david@geophom.fr>> wrote:
> > >
> > > Hello Vero,
> > >
> > > Thanks for your reply. It took almost one hour to extract my area (I guess because of topology). I do the same with Qgis from a shape in less than 10 minutes (but with several operations)... it's may be a bad idea to have a so heavy vector map in Grass...
> >
> > this kind of operation could be much faster in latest GRASS 7.4 than in any previous versions. Which GRASS version are you using?
> >
> > Hello Markus,
> >
> > I'm using 7.4.1 on debian9
> >
> > But my grass db folder is on a small server (Synology NAS), so may be the network makes slow down the operation ?
>
> This is possible, but it should not slow it down that much: 10 min vs 1 hour
>
> Can you provide a link to the vector from which you want to extract a small area, and the command you used to extract a small area? Maybe there is scope for optimization.
>
> Markus,
>
> The data is CorineLandCover for france. You can download 400mo here > https://we.tl/t-tvldlmkVw3
>
> First I have imported the shape in PERMANENT
>
> and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12
>
> Hope it helps you...

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

"here" means a laptop with Intel Core i7 4th generation and standard SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing attributes, depending on the database connection. I am using the default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I'm trying to do again the operation on local machine. I want fisrt, create a vector with only selected data (on full France area). But I've met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area where=CODE_12 IN (111,112,121,311,312,313,324) output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>...
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
'IN') in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
'IN') in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized "IN" sql condition. If I force layer=-1, it runs bur the output vector is the same as input one.

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Hi,

I've repeated the operation v.in.ogr and v.extract with success on different grass mapset, but I get always the same error (on v.extract : see above) only with one of my local grass mapset (PERMANENT). I do not understand what happens...

Regards,
Frank

17

On Tue, Nov 6, 2018 at 4:03 PM Frank David <frank.david@geophom.fr> wrote:

Le 05/11/2018 à 14:23, Frank David a écrit :

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 02/11/2018 à 14:43, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David
<frank.david@geophom.fr mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my
area (I guess because of topology). I do the same with Qgis from a
shape in less than 10 minutes (but with several operations)… it’s
may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4
than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so
may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min
vs 1 hour

Can you provide a link to the vector from which you want to extract
a small area, and the command you used to extract a small area? Maybe
there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here
https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT
where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

“here” means a laptop with Intel Core i7 4th generation and standard
SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing
attributes, depending on the database connection. I am using the
default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I’m trying to do again the operation on local machine. I want fisrt,
create a vector with only selected data (on full France area). But
I’ve met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area
where=CODE_12 IN (111,112,121,311,312,313,324)
output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>…
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized “IN” sql condition. If I force
layer=-1, it runs bur the output vector is the same as input one.

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Hi,

I’ve repeated the operation v.in.ogr and v.extract with success on
different grass mapset, but I get always the same error (on v.extract :
see above) only with one of my local grass mapset (PERMANENT). I do not
understand what happens…

In the current mapset, the default database connection must not be DBF. You can reset the default database connection to sqlite with

db.connect -d
After that verify with
db.connect -p
that sqlite is indeed the default database connection.

Now you can repeat with v.in.ogr + v.extract

Markus M

18

Le 06/11/2018 à 21:36, Markus Metz a écrit :

On Tue, Nov 6, 2018 at 4:03 PM Frank David <frank.david@geophom.fr> wrote:

Le 05/11/2018 à 14:23, Frank David a écrit :

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 02/11/2018 à 14:43, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David
<frank.david@geophom.fr mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my
area (I guess because of topology). I do the same with Qgis from a
shape in less than 10 minutes (but with several operations)… it’s
may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4
than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so
may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min
vs 1 hour

Can you provide a link to the vector from which you want to extract
a small area, and the command you used to extract a small area? Maybe
there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here
https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT
where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

“here” means a laptop with Intel Core i7 4th generation and standard
SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing
attributes, depending on the database connection. I am using the
default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I’m trying to do again the operation on local machine. I want fisrt,
create a vector with only selected data (on full France area). But
I’ve met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area
where=CODE_12 IN (111,112,121,311,312,313,324)
output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>…
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized “IN” sql condition. If I force
layer=-1, it runs bur the output vector is the same as input one.

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Hi,

I’ve repeated the operation v.in.ogr and v.extract with success on
different grass mapset, but I get always the same error (on v.extract :
see above) only with one of my local grass mapset (PERMANENT). I do not
understand what happens…

In the current mapset, the default database connection must not be DBF. You can reset the default database connection to sqlite with

db.connect -d
After that verify with
db.connect -p
that sqlite is indeed the default database connection.

Now you can repeat with v.in.ogr + v.extract

Markus M

I’m afraid it’s not the problem…

db.connect -d
Pilote par défaut / base de données définie à :
pilote : sqlite
base : /media/hdd1/grassdata/France_L93/PERMANENT/sqlite/sqlite.db
(Wed Nov 7 10:46:01 2018) La commande s’est terminée (0 sec)
(Wed Nov 7 10:46:29 2018)
db.connect -p
driver: sqlite
database: /media/hdd1/grassdata/France_L93/PERMANENT/sqlite/sqlite.db
schema:
group:

and when execute v.extract I get the same error…

I have tested again the same operation in other mapset than PERMANENT, and it works. I don’t understand… if any idea ?..

Cheers,
Frank

19

On Wed, Nov 7, 2018 at 11:18 AM Frank David <frank.david@geophom.fr> wrote:

Le 06/11/2018 à 21:36, Markus Metz a écrit :

On Tue, Nov 6, 2018 at 4:03 PM Frank David <frank.david@geophom.fr> wrote:

Le 05/11/2018 à 14:23, Frank David a écrit :

Le 02/11/2018 à 22:27, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 6:32 PM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 02/11/2018 à 14:43, Markus Metz a écrit :

On Fri, Nov 2, 2018 at 11:58 AM Frank David <frank.david@geophom.fr
mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Le 31/10/2018 à 20:57, Markus Metz a écrit :

Hi Frank,

On Wed, Oct 31, 2018 at 1:39 PM Frank David
<frank.david@geophom.fr mailto:[frank.david@geophom.fr](mailto:frank.david@geophom.fr)> wrote:

Hello Vero,

Thanks for your reply. It took almost one hour to extract my
area (I guess because of topology). I do the same with Qgis from a
shape in less than 10 minutes (but with several operations)… it’s
may be a bad idea to have a so heavy vector map in Grass…

this kind of operation could be much faster in latest GRASS 7.4
than in any previous versions. Which GRASS version are you using?

Hello Markus,

I’m using 7.4.1 on debian9

But my grass db folder is on a small server (Synology NAS), so
may be the network makes slow down the operation ?

This is possible, but it should not slow it down that much: 10 min
vs 1 hour

Can you provide a link to the vector from which you want to extract
a small area, and the command you used to extract a small area? Maybe
there is scope for optimization.

Markus,

The data is CorineLandCover for france. You can download 400mo here
https://we.tl/t-tvldlmkVw3

First I have imported the shape in PERMANENT

and I do : v.extract --verbose input=CLC12_FR_RGF@PERMANENT
where=CODE_12 IN (111,112,131,311,312,313,324) output=clc12

Hope it helps you…

import of the shapefile with v.in.ogr takes here 5 minutes
v.extract takes here 27 seconds instead of 1 hour

“here” means a laptop with Intel Core i7 4th generation and standard
SATA SSD, i.e. no fancy new high-speed system at all.

I am afraid this does not help you. What could take long is writing
attributes, depending on the database connection. I am using the
default sqlite database.

You could use top to check if v.extract is running at 100%.

Hello Markus,

Thank you for your help.

I’m trying to do again the operation on local machine. I want fisrt,
create a vector with only selected data (on full France area). But
I’ve met a problem with v.extract.

v.extract --overwrite --verbose input=CLC12_FR_RGF@PERMANENT type=area
where=CODE_12 IN (111,112,121,311,312,313,324)
output=CLC12_FR_RGF_BOISBATI@PERMANENT
Loading categories from table <CLC12_FR_RGF>…
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
DBMI-DBF erreur de pilote :
SQL parser error (syntax error, unexpected NAME processing
‘IN’) in statement:
SELECT cat FROM CLC12_FR_RGF WHERE CODE_12 IN
(111,112,121,311,312,313,324)
Unable to open cursor.
ERROR: Unable select records from table <CLC12_FR_RGF>

It looks that grass does not recognized “IN” sql condition. If I force
layer=-1, it runs bur the output vector is the same as input one.

Am I wrong somewhere ? I use Grass 7.4.2 on debian9.

Thank you for your help.

Frank

Hi,

I’ve repeated the operation v.in.ogr and v.extract with success on
different grass mapset, but I get always the same error (on v.extract :
see above) only with one of my local grass mapset (PERMANENT). I do not
understand what happens…

In the current mapset, the default database connection must not be DBF. You can reset the default database connection to sqlite with

db.connect -d
After that verify with
db.connect -p
that sqlite is indeed the default database connection.

Now you can repeat with v.in.ogr + v.extract

Markus M

I’m afraid it’s not the problem…

db.connect -d
Pilote par défaut / base de données définie à :
pilote : sqlite
base : /media/hdd1/grassdata/France_L93/PERMANENT/sqlite/sqlite.db
(Wed Nov 7 10:46:01 2018) La commande s’est terminée (0 sec)
(Wed Nov 7 10:46:29 2018)
db.connect -p
driver: sqlite
database: /media/hdd1/grassdata/France_L93/PERMANENT/sqlite/sqlite.db
schema:
group:

and when execute v.extract I get the same error…

Have you re-imported the vector?
What is the output of v.db.connect map=CLC12_FR_RGF -p
The “IN” SQL syntax works only if the vector CLC12_FR_RGF is not using dbf. You must make sure with “v.db.connect map=CLC12_FR_RGF -p” that CLC12_FR_RGF is really using sqlite as database driver.

HTH,

Markus M

I have tested again the same operation in other mapset than PERMANENT, and it works. I don’t understand… if any idea ?..

Cheers,
Frank