[GRASS-user] error importing postgresql db

Hi,

I tried importing a PostgreSQL table (a view) using v.in.ogr. However,
the module failed with:

---<--------------------cut here---------------start------------------->---
v.in.ogr dsn="PG:host=localhost port=5433 dbname=mydb" layer=mylayer out=test type=point
Projection of input dataset and current location appear to match
Layer: mylayer
WARNING: Width for column sex set to 255 (was not specified by OGR), some
         strings may be truncated!
aDBMI-DBF driver error:
SQL parser error: syntax error, unexpected TIME, expecting NAME processing 'time'
in statement:
create table test (cat integer, a integer, b varchar ( 3 ), c integer, d integer, e integer, f integer, g integer, h integer, i integer, j integer, k varchar ( 10 ), l varchar ( 500 ), m varchar ( 12 ), sex varchar ( 255 ), n varchar ( 300 ), time datetime, time_b datetime)
Error in db_execute_immediate()

ERROR: Unable to create table: 'create table test (cat integer, a
       integer, b varchar ( 3 ), c integer, d integer,
       e integer, f integer, g integer, h integer, i
       integer, j integer, k varchar ( 10 ), l varchar ( 500
       ), m varchar ( 12 ), sex varchar ( 255 ), n varchar (
       300 ), time datetime, time_b datetime)'
---<--------------------cut here---------------end--------------------->---

The warning about the sex column is harmless since it is just a custom
type and the 255 limit is fine here. The parsing error with time column
seems to have prevented the vector from building properly though. The
same error occurs when doing it with v.in.db. The view and the
underlying tables show no problems in Postgresql.

Cheers,

--
Seb

Seb:

I tried importing a PostgreSQL table (a view) using v.in.ogr. However,
the module failed

...

The warning about the sex column is harmless since it is just a custom
type and the 255 limit is fine here. The parsing error with time column
seems to have prevented the vector from building properly though. The
same error occurs when doing it with v.in.db. The view and the
underlying tables show no problems in Postgresql.

Not sure if this is a conflict with SQL commands, but you could try to
_feed_ new names for your columns by using the "cname=" paramater of
"v.in.ogr" and see what happens?

Nikos

On Mon, 01 Jun 2009 00:54:05 +0200,
Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

Seb:

I tried importing a PostgreSQL table (a view) using v.in.ogr.
However, the module failed

...

The warning about the sex column is harmless since it is just a
custom type and the 255 limit is fine here. The parsing error with
time column seems to have prevented the vector from building properly
though. The same error occurs when doing it with v.in.db. The view
and the underlying tables show no problems in Postgresql.

Not sure if this is a conflict with SQL commands, but you could try to
_feed_ new names for your columns by using the "cname=" paramater of
"v.in.ogr" and see what happens?

I tried that but the problem remains. It seems as if there's problems
importing such datetime columns. Thanks.

--
Seb

On Mon, Jun 1, 2009 at 1:50 AM, Seb <spluque@gmail.com> wrote:

On Mon, 01 Jun 2009 00:54:05 +0200,
Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

Seb:

...

Not sure if this is a conflict with SQL commands, but you could try to
_feed_ new names for your columns by using the "cname=" paramater of
"v.in.ogr" and see what happens?

I tried that but the problem remains. It seems as if there's problems
importing such datetime columns. Thanks.

Can you please post cmd line and error?

Markus

On Mon, 1 Jun 2009 07:40:28 +0200,
Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jun 1, 2009 at 1:50 AM, Seb <spluque@gmail.com> wrote:

On Mon, 01 Jun 2009 00:54:05 +0200,
Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

Seb:

...

Not sure if this is a conflict with SQL commands, but you could try
to _feed_ new names for your columns by using the "cname=" paramater
of "v.in.ogr" and see what happens?

I tried that but the problem remains. It seems as if there's
problems importing such datetime columns. Thanks.

Can you please post cmd line and error?

Ok, I set this up differently because I found yesterday that it's a
better approach in my case to import just the vector geometry and then
connect the vector to the database (it seems more efficient not to
duplicate the information by having it in the database and GRASS). The
table I'd be importing is actually a view, so it was very simple to
modify the view to give names that don't collide with SQL if unquoted
(although it would be nice to avoid this inconvenience). However, I'm
still running into problems:

---<--------------------cut here---------------start------------------->---
# We're in an AEA projection, data in lon/lat so extract first to reproject
$ db.select table='locs' database='dbname=locations,port=5433' \
    driver='pg' > locs_ll; head locs_ll
sample_id|project|species|sex|flipper_tag|date_time_deployed|claw_marks|mass|body_length|girth|fat_depth|locid|prognum|date_time|class|lon|lat
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27867|01142|2008-8-12 8:28:08|1|-63.076|58.448
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27868|01142|2008-8-12 9:41:28|A|-63.101|58.431
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27869|01142|2008-8-12 11:22:01|A|-63.086|58.407
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27870|01142|2008-8-12 11:48:25|2|-63.086|58.413
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27871|01142|2008-8-12 12:58:22|B|-63.119|58.463
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27872|01142|2008-8-12 13:39:50|0|-63.106|58.428
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27873|01142|2008-8-12 13:51:36|B|-63.067|58.398
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27874|01142|2008-8-12 14:54:49|A|-63.061|58.442
SAGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27875|01142|2008-8-12 15:20:01|0|-63.134|58.463

# Reproject to AEA
$ awk -F"|" 'NR > 1 {print $16, $17}' locs_ll | \
    cs2cs +init=epsg:4326 +to $(g.proj -jf) | \
    awk -v OFS="|" 'BEGIN {print "lon_aea", "lat_aea"}; {print $1, $2}' \
    > locs_aea; head locs_aea
lon_aea|lat_aea
-4.44|22.06
-5.90|20.17
-5.03|17.49
-5.03|18.16
-6.94|23.74
-6.19|19.84
-3.92|16.49
-3.56|21.39
-7.82|23.74

# Paste back unique ID (locid, for category) and pass to v.in.ascii
$ cut -d'|' -f12 locs_ll | paste -d"|" - locs_aea | \
    v.in.ascii out=locs_pts_import skip=1 x=2 y=3 cat=1
Scanning input for column types...
Maximum input row length: 22
Maximum number of columns: 3
Minimum number of columns: 3
Importing points...

Building topology for vector map <locs_pts_import>...
Registering primitives...
1311 primitives registered
1311 vertices registered
Building areas...

0 areas built
0 isles built
Attaching islands...
Attaching centroids...

Topology was built
Number of nodes: 976
Number of primitives: 1311
Number of points: 1311
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
v.in.ascii complete.

# So far so good (data even plots ok). Connect to postgresql (no DBF
# created by default, so no need to remove any previous connection)
$ v.db.connect map=locs_pts_import driver='pg' database='dbname=locations,port=5433' \
    table='locs' key='locid'

The table <locs> is now part of vector map

<locs_pts_import> and may be deleted or overwritten by GRASS modules
DBMI-Postgres driver error:
Cannot create index:
create unique index locs_locid on locs ( locid )
ERROR: "locs" is not a table

WARNING: Cannot create index
aSelect privileges were granted on the table

# Try to remove the vector -- OMG it tries to drop the view in
# Postgresql (how to avoid this even if it worked?!)
$ g.remove -f vect=locs_pts_import
Removing vector <locs_pts_import>
DBMI-Postgres driver error:
Cannot execute:
drop table locs
ERROR: "locs" is not a table
HINT: Use DROP VIEW to remove a view.

WARNING: Unable to drop table: 'drop table locs'
aSegmentation fault

$ g.version -b
GRASS 6.4.0RC4 (2009)

# ./configure --host=x86_64-linux-gnu --build=x86_64-linux-gnu --prefix=/usr/lib --sysconfdir=/etc --sharedstatedir=/var --enable-socket --enable-shared --with-postgres --with-mysql --with-cxx --with-x --with-gdal --with-freetype --with-motif --with-readline --with-nls --with-odbc --with-sqlite --enable-largefile --with-freetype-includes=/usr/include/freetype2 --with-tcltk-includes=/usr/include/tcl --with-postgres-includes=/usr/include/postgresql --with-mysql-includes=/usr/include/mysql --with-proj-share=/usr/share/proj --with-wxwidgets=/usr/lib/wx/config/gtk2-unicode-release-2.8 --with-python=/usr/bin/python-config --with-cairo
---<--------------------cut here---------------end--------------------->---

So it seems we cannot work with views because an index cannot to be
created. I also learned that attempting to remove a vector from GRASS
also drops its linked table from an external database, which fortunately
failed in this case. At this point, I don't know what the best way to
proceed is. Thanks in advance for any further feedback.

--
Seb

On Mon, 1 Jun 2009 07:40:28 +0200,
Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jun 1, 2009 at 1:50 AM, Seb <spluque@gmail.com> wrote:

On Mon, 01 Jun 2009 00:54:05 +0200,
Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

Seb:

...

Not sure if this is a conflict with SQL commands, but you could try
to _feed_ new names for your columns by using the "cname=" paramater
of "v.in.ogr" and see what happens?

I tried that but the problem remains. It seems as if there's
problems importing such datetime columns. Thanks.

Can you please post cmd line and error?

Here's what I get from v.in.db with a view that has a Postgis geometry
column where the reprojection to the local GRASS location was done
directly. This gives the same error message as what I saw with
v.in.ogr, even after changing the column names with the 'cnames'
argument:

---<--------------------cut here---------------start------------------->---
v.in.db driver=pg database="dbname=locations,port=5433" \
        table=locs x='x(pts_aea)' y='y(pts_aea)' key=locid out=test --o

WARNING: Vector map <test> already exists and will be overwritten

aWARNING: pg driver: PostGIS column 'pts_ll', type 'geometry' will not be
         converted
aWARNING: pg driver: PostGIS column 'pts_aea', type 'geometry' will not be
         converted
aWriting features...

Copying attributes...
WARNING: pg driver: PostGIS column 'pts_ll', type 'geometry' will not be
         converted
aWARNING: pg driver: PostGIS column 'pts_aea', type 'geometry' will not be
         converted
aWARNING: Type TEXT converted to 'VARCHAR(250)'
aDBMI-DBF driver error:
SQL parser error: syntax error, unexpected NAME processing 'datetime'
in statement:
create table test ( sample_id varchar(12), project varchar(500), species varchar(300), sex varchar(250), flipper_tag integer, date_time_deployed datetime, claw_marks integer, mass double precision, body_length double precision, girth double precision, fat_depth double precision, locid integer, prognum varchar(10), date_time datetime, class varchar(3), lon double precision, lat double precision )
Cannot create table

WARNING: Unable to create table <Ðg¹>
aWARNING: Unable to copy table
aBuilding topology for vector map <test>...
Registering primitives...
1311 primitives registered
1311 vertices registered
Building areas...

0 areas built
0 isles built
Attaching islands...
Attaching centroids...

Number of nodes: 976
Number of primitives: 1311
Number of points: 1311
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
v.in.db complete. 1311 points written to vector map.
---<--------------------cut here---------------end--------------------->---

--
Seb

Hi again,

I traced one of the problems with my attempts to import vector and
associated attribute tables. I found that if some rows contain null
values for at least one column, then v.in.ascii cannot parse the table
correctly. With an input file as this:

---<--------------------cut here---------------start------------------->---
$ cat /tmp/gmt.XQTuDE/test.dat
sample_id|project|species|sex|flipper_tag|date_time_deployed|claw_marks|mass|body_length|girth|fat_depth|locid|prognum|date_time|class|lon|lat|lon_aea|lat_aea
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27867|01142|2008-8-12 8:28:08|1|-63.076|58.448|-4.44|22.06
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27868|01142|2008-8-12 9:41:28|A|-63.101|58.431|-5.90|20.17
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27869|01142|2008-8-12 11:22:01|A|-63.086|58.407|-5.03|17.49
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27870|01142|2008-8-12 11:48:25|2|-63.086|58.413|-5.03|18.16
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27871|01142|2008-8-12 12:58:22|B|-63.119|58.463|-6.94|23.74
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27872|01142|2008-8-12 13:39:50|0|-63.106|58.428|-6.19|19.84
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27873|01142|2008-8-12 13:51:36|B|-63.067|58.398|-3.92|16.49
AGB-08-0001|Fjord|Ph|male|1316|2008-8-11 18:30:00||25|90|82.5||27874|01142|2008-8-12 14:54:49|A|-63.061|58.442|-3.56|21.39
---<--------------------cut here---------------end--------------------->---

Note the empty columns 7 and 11 for all rows. v.in.ascii produces this
error:

---<--------------------cut here---------------start------------------->---
v.in.ascii in=/tmp/gmt.XQTuDE/test.dat out=test skip=1 x=18 y=19 cat=12 --o --v
Scanning input for column types...
Maximum input row length: 159
Maximum number of columns: 19
Minimum number of columns: 19
Column: 1 type: string length: 11
Column: 2 type: string length: 5
Column: 3 type: string length: 2
Column: 4 type: string length: 4
Column: 5 type: integer
Column: 6 type: string length: 18
Column: 7 type: string length: 0
Column: 8 type: integer
Column: 9 type: integer
Column: 10 type: double
Column: 11 type: string length: 0
Column: 12 type: integer
Column: 13 type: integer
Column: 14 type: string length: 18
Column: 15 type: string length: 1
Column: 16 type: double
Column: 17 type: double
Column: 18 type: double
Column: 19 type: double
Importing points...

Populating table...
dbmi: Protocol error
Building topology for vector map <test>...
Registering primitives...
8 primitives registered
8 vertices registered
Building areas...

0 areas built
0 isles built
Attaching islands...
Attaching centroids...

Topology was built
Number of nodes: 8
Number of primitives: 8
Number of points: 8
Number of lines: 0
Number of boundaries: 0
Number of centroids: 0
Number of areas: 0
Number of isles: 0
v.in.ascii complete.
---<--------------------cut here---------------end--------------------->---

The error only occurs if the null values are present.

Cheers,

--
Seb

On Tue, Jun 2, 2009 at 10:09 PM, Seb <spluque@gmail.com> wrote:

Hi again,

I traced one of the problems with my attempts to import vector and
associated attribute tables.

Which backend do you use? DBF?

I found that if some rows contain null
values for at least one column, then v.in.ascii cannot parse the table
correctly.

It is not v.in.ascii but the DBMI driver having troubles.

...

Note the empty columns 7 and 11 for all rows. v.in.ascii produces this
error:

---<--------------------cut here---------------start------------------->---
v.in.ascii in=/tmp/gmt.XQTuDE/test.dat out=test skip=1 x=18 y=19 cat=12 --o --v
Scanning input for column types...
Maximum input row length: 159
Maximum number of columns: 19
Minimum number of columns: 19
Column: 1 type: string length: 11
Column: 2 type: string length: 5
Column: 3 type: string length: 2
Column: 4 type: string length: 4
Column: 5 type: integer
Column: 6 type: string length: 18
Column: 7 type: string length: 0
Column: 8 type: integer
Column: 9 type: integer
Column: 10 type: double
Column: 11 type: string length: 0
Column: 12 type: integer
Column: 13 type: integer
Column: 14 type: string length: 18
Column: 15 type: string length: 1
Column: 16 type: double
Column: 17 type: double
Column: 18 type: double
Column: 19 type: double
Importing points...

Populating table...
dbmi: Protocol error

I can reproduce it here.

Homework for developers:

db_close_database_shutdown_driver() seems to be oversimplified.
It calls db_close_database(driver) and db_shutdown_driver(driver)
but without testing their return status. Subsequently v.in.ascii cannot
issue a G_fatal_error() in case of "dbmi: Protocol error" as it should.

db_close_database(driver) looks ok:
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/c_closedb.c#L26

but less sure about db_shutdown_driver(driver):
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/shutdown.c#L39

and not at all about db_close_database_shutdown_driver():
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/db.c#L62
which returns brute-force DB_OK.

If db_shutdown_driver(driver) is acceptable, then put if conditions
into db_close_database_shutdown_driver() for DB_FAILED on failure?

Markus

On Tue, 2 Jun 2009 22:34:59 +0200,
Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Jun 2, 2009 at 10:09 PM, Seb <spluque@gmail.com> wrote:

Hi again,

I traced one of the problems with my attempts to import vector and
associated attribute tables.

Which backend do you use? DBF?

Yes, the default, in 6.4.0RC4 (Debian sid). Thanks for having a look!

Cheers,

--
Seb

On Tue, Jun 2, 2009 at 10:44 PM, Seb <spluque@gmail.com> wrote:

On Tue, 2 Jun 2009 22:34:59 +0200,
Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Jun 2, 2009 at 10:09 PM, Seb <spluque@gmail.com> wrote:

Hi again,

I traced one of the problems with my attempts to import vector and
associated attribute tables.

Which backend do you use? DBF?

Yes, the default, in 6.4.0RC4 (Debian sid). Thanks for having a look!

I had the same problem today and didn't find a solution
for the DBF driver (didn't try too hard, though).

Hopefully other list members have an idea (maybe write NULL
or NA or so?).

Markus

Cheers,

--
Seb

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Markus Neteler wrote:

Homework for developers:

db_close_database_shutdown_driver() seems to be oversimplified.
It calls db_close_database(driver) and db_shutdown_driver(driver)
but without testing their return status. Subsequently v.in.ascii cannot
issue a G_fatal_error() in case of "dbmi: Protocol error" as it should.

db_close_database(driver) looks ok:
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/c_closedb.c#L26

but less sure about db_shutdown_driver(driver):
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/shutdown.c#L39

and not at all about db_close_database_shutdown_driver():
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/db.c#L62
which returns brute-force DB_OK.

If db_shutdown_driver(driver) is acceptable, then put if conditions
into db_close_database_shutdown_driver() for DB_FAILED on failure?

Something like:

int db_close_database_shutdown_driver(dbDriver * driver)
{
    int status = db_close_database(driver);
    if (db_shutdown_driver(driver) != 0)
  status = DB_FAILED;

    return status;
}

Except that db_shutdown_driver() should return a useful (and portable)
result, e.g. "WIFEXITED(status) ? WEXITSTATUS(status) : -1" rather
than the raw status.

Also, I'm unclear as to why db_shutdown_driver() only uses
DB_PROC_SHUTDOWN_DRIVER on Windows.

--
Glynn Clements <glynn@gclements.plus.com>

On Wed, Jun 3, 2009 at 3:58 AM, Glynn Clements <glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

Homework for developers:

db_close_database_shutdown_driver() seems to be oversimplified.
It calls db_close_database(driver) and db_shutdown_driver(driver)
but without testing their return status. Subsequently v.in.ascii cannot
issue a G_fatal_error() in case of "dbmi: Protocol error" as it should.

db_close_database(driver) looks ok:
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/c_closedb.c#L26

but less sure about db_shutdown_driver(driver):
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/shutdown.c#L39

and not at all about db_close_database_shutdown_driver():
http://trac.osgeo.org/grass/browser/grass/trunk/lib/db/dbmi_client/db.c#L62
which returns brute-force DB_OK.

If db_shutdown_driver(driver) is acceptable, then put if conditions
into db_close_database_shutdown_driver() for DB_FAILED on failure?

Something like:

int db_close_database_shutdown_driver(dbDriver * driver)
{
int status = db_close_database(driver);
if (db_shutdown_driver(driver) != 0)
status = DB_FAILED;

return status;
}

Thanks, tested and submitted.

Except that db_shutdown_driver() should return a useful (and portable)
result, e.g. "WIFEXITED(status) ? WEXITSTATUS(status) : -1" rather
than the raw status.

... not sure how to implement this...

Also, I'm unclear as to why db_shutdown_driver() only uses
DB_PROC_SHUTDOWN_DRIVER on Windows.

... also no idea...

Markus

On Tue, Jun 2, 2009 at 10:09 PM, Seb <spluque@gmail.com> wrote:

Hi again,

I traced one of the problems with my attempts to import vector and
associated attribute tables. I found that if some rows contain null
values for at least one column, then v.in.ascii cannot parse the table
correctly.

Indeed, the DBF driver has troubles with empty columns.

...

Column: 15 type: string length: 1
Column: 16 type: double
Column: 17 type: double
Column: 18 type: double
Column: 19 type: double
Importing points...

Populating table...
dbmi: Protocol error

Ok, I have fixed that with Glynn's help. Now, at this point
v.in.ascii fails with

  "ERROR: Could not close attribute table. The DBMI driver did not
accept all attributes"

rather than continuing with a broken map/table.

In your example, when filling the empty columns, it works.
I suggest to use SQLite instead of the limited DBF driver.

Markus