[GRASS-user] Help with defining DB Drivers for input and outputs

Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver?

I’ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I’ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail.

I want the outputs written to sqlite, because v.out.ogr and v.out.postgis when the attributes are stored in PostgreSQL seems unworkably slow (which may be a pg_conf issue but my DB is performing well at scale for non-GRASS parallelised workflows)?

I want to keep the input attributes still in PostgreSQL because its the road table, and the fwdcost and bwdcost columns get updated by data from non-GRASS schema.

I need the fastest pipeline performance possible for when I scale the solution out (which will be done using GNU Parallel).

Right now I’m stuck between a rock and a hard place.

Really hoping someone can help me out… :slight_smile:

Thanks
Mark

#CREATE EMPTY MAP TO HOLD THE OUTPUTS

g.remove -f type=vector pattern=“temp**”
layername="temp_8
v.edit --overwrite tool=create map=$layername
v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’

v.info $layername

#RUN THE ANALYSIS, WHERE THE INPUT LAYER ATTRIBUTES RESIDE IN POSTGRESQL
v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100 out=$layername afcolumn=fwdcost abcolumn=bwdcost

NO GOOD - THE OUTPUT ATTRIBUTE TABLE STILL APPEARS IN PGL

##########################

GRASS 7.0.0svn (nodeclean):~ > v.edit --overwrite tool=create map=$layername
Building topology for vector map temp_8@PERMANENT
Registering primitives…
0 primitives registered
0 vertices registered
Building areas…
0 areas built
0 isles built
Attaching islands…
Attaching centroids…
Number of nodes: 0
Number of primitives: 0
Number of points: 0
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
v.edit complete.
GRASS 7.0.0svn (nodeclean):~ > v.db.connect map=$layername table=$layername driver=sqlite database=‘$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’
WARNING: Table <temp_8> does not exist in database
<$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db>
The table <temp_8> is now part of vector map <temp_8> and may be deleted or
overwritten by GRASS modules
DBMI-SQLite driver error:
Unable to create index:
create unique index if not exists temp_8_cat on temp_8 ( cat )
no such table: main.temp_8

DBMI-SQLite driver error:
Unable to create index:
create unique index if not exists temp_8_cat on temp_8 ( cat )
no such table: main.temp_8

WARNING: Cannot create index
Select privileges were granted on the table
GRASS 7.0.0svn (nodeclean):~ > v.db.connect -p map=$layername
Vector map <temp_8> is connected by:
layer <1/temp_8> table <temp_8> in database </var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db> through driver with key

GRASS 7.0.0svn (nodeclean):~ > v.info $layername

±---------------------------------------------------------------------------+
| Name: temp_8 |
| Mapset: PERMANENT |
| Location: nodeclean |
| Database: /var/tmp |
| Title: |
| Map scale: 1:1 |
| Name of creator: root |
| Organization: |
| Source date: Wed Feb 11 22:49:13 2015 |

Timestamp (first layer): none
Map format: native
----------------------------------------------------------------------------
Type of map: vector (level: 2)
Number of points: 0 Number of centroids: 0
Number of lines: 0 Number of boundaries: 0
Number of areas: 0 Number of islands: 0
Map is 3D: No
Number of dblinks: 1
Projection: Albers Equal Area
N: 0 S: 0
E: 0 W: 0
Digitization threshold: 0
Comment:
±---------------------------------------------------------------------------+

GRASS 7.0.0svn (nodeclean):~ > v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100 out=$layername afcolumn=fwdcost abcolumn=bwdcost
WARNING: Vector map <temp_8> already exists and will be overwritten
WARNING: Table <temp_8> linked to vector map <temp_8> does not exist
Building graph…

Outputs of temp_8 visible in PostgreSQL….#########

On 12/02/15 00:12, Mark Wynter wrote:

Within the same mapset, is it possible to specify SQLITE as the database
for the output layer, whilst using a map as an input that has a PG driver?

I’ve read the manual and tried a few approaches, but I found the outputs
still get written to PG. May be I’ve missed a critical step? See
messages below. I also experimented with v.add.table and v.add.columns
to the empty map, to no avail.

You should use db.connect to set your default settings to SQLite, then run your command. By overwriting your map in the v.net.allpairs call you overwrite everything, including any db connections set before and you use the default mapset connection settings which seem to be set to PG.

Moritz

I want the outputs written to sqlite, because v.out.ogr and
v.out.postgis when the attributes are stored in PostgreSQL seems
unworkably slow (which may be a pg_conf issue but my DB is performing
well at scale for non-GRASS parallelised workflows)?

I want to keep the input attributes still in PostgreSQL because its the
road table, and the fwdcost and bwdcost columns get updated by data from
non-GRASS schema.

I need the fastest pipeline performance possible for when I scale the
solution out (which will be done using GNU Parallel).

Right now I’m stuck between a rock and a hard place.

Really hoping someone can help me out… :slight_smile:

Thanks
Mark

#CREATE EMPTY MAP TO HOLD THE OUTPUTS
g.remove -f type=vector pattern="temp**"
layername="temp_8
v.edit --overwrite tool=create map=$layername
v.db.connect map=$layername table=$layername driver=sqlite
  database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’
v.info $layername

#RUN THE ANALYSIS, WHERE THE INPUT LAYER ATTRIBUTES RESIDE IN POSTGRESQL
v.net.allpairs --overwrite in=nsw_road_network_final_connected
cats=1-100out=$layername afcolumn=fwdcost abcolumn=bwdcost

# NO GOOD - THE OUTPUT ATTRIBUTE TABLE STILL APPEARS IN PGL

##########################
GRASS 7.0.0svn (nodeclean):~ > v.edit --overwrite tool=create map=$layername
Building topology for vector map <temp_8@PERMANENT>...
Registering primitives...
0 primitives registered
0 vertices registered
Building areas...
0 areas built
0 isles built
Attaching islands...
Attaching centroids...
Number of nodes: 0
Number of primitives: 0
Number of points: 0
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
v.edit complete.
GRASS 7.0.0svn (nodeclean):~ > v.db.connect map=$layername
table=$layername driver=sqlite
database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db'
WARNING: Table <temp_8> does not exist in database
          <$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db>
The table <temp_8> is now part of vector map <temp_8> and may be deleted or
overwritten by GRASS modules
DBMI-SQLite driver error:
Unable to create index:
create unique index if not exists temp_8_cat on temp_8 ( cat )
no such table: main.temp_8

DBMI-SQLite driver error:
Unable to create index:
create unique index if not exists temp_8_cat on temp_8 ( cat )
no such table: main.temp_8

WARNING: Cannot create index
Select privileges were granted on the table
GRASS 7.0.0svn (nodeclean):~ > v.db.connect -p map=$layername
Vector map <temp_8> is connected by:
layer <1/temp_8> table <temp_8> in database
</var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db> through driver <sqlite>
with key <cat>
GRASS 7.0.0svn (nodeclean):~ > v.info $layername

  +----------------------------------------------------------------------------+
  | Name: temp_8
       |
  | Mapset: PERMANENT
       |
  | Location: nodeclean
       |
  | Database: /var/tmp
       |
  | Title:
       |
  | Map scale: 1:1
       |
  | Name of creator: root
       |
  | Organization:
       |
  | Source date: Wed Feb 11 22:49:13 2015
       |
  | Timestamp (first layer): none
       |
  |----------------------------------------------------------------------------|
  | Map format: native
       |
  |----------------------------------------------------------------------------|
  | Type of map: vector (level: 2)
       |
  |
       |
  | Number of points: 0 Number of centroids: 0
       |
  | Number of lines: 0 Number of boundaries: 0
       |
  | Number of areas: 0 Number of islands: 0
       |
  |
       |
  | Map is 3D: No
       |
  | Number of dblinks: 1
       |
  |
       |
  | Projection: Albers Equal Area
       |
  |
       |
  | N: 0 S: 0
       |
  | E: 0 W: 0
       |
  |
       |
  | Digitization threshold: 0
       |
  | Comment:
       |
  |
       |
  +----------------------------------------------------------------------------+

GRASS 7.0.0svn (nodeclean):~ > v.net.allpairs --overwrite
in=nsw_road_network_final_connected cats=1-100 out=$layername
afcolumn=fwdcost abcolumn=bwdcost
WARNING: Vector map <temp_8> already exists and will be overwritten
WARNING: Table <temp_8> linked to vector map <temp_8> does not exist
Building graph…

#### Outputs of temp_8 visible in PostgreSQL….#########

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user