[GRASS-dev] DBMI parse error question

Hi,

I try to import a map from PostgreSQL into GRASS/DBF driver
with v.in.db, but get

DBMI-DBF driver error:
SQL parser error in statement:
create table ticks_feltre_2002_2006_sites ( ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer )
Cannot create table

but I don't see where the evil is:

echo " ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer" | tr -s ',' '\n' | sort
adf integer
adm integer
adtot integer
comune varchar(250)
data date
east double precision
ID integer
larve integer
localita varchar(250)
mapsheet varchar(250)
mgrs varchar(250)
microsito varchar(250)
ninfe integer
north double precision
note varchar(250)
ora time
periodo_prelievo varchar(250)
siteid integer
stazione varchar(250)
tcelsius double precision
umidrel double precision

Any ideas? Of course it would be fancy to have a more reasonable
error message. This error uses to pop up if preserved words are used
as column names, but I don't see any (even, it should tell me in an
ideal world).

Markus

On Tue, Feb 27, 2007 at 11:38:01PM +0100, Markus Neteler wrote:

Hi,

I try to import a map from PostgreSQL into GRASS/DBF driver
with v.in.db, but get
[..]
adf integer
adm integer
adtot integer
comune varchar(250)
data date

From:
http://www.postgresql.org/docs/8.2/interactive/sql-keywords-appendix.html#KEYWORDS-TABLE

data is a reserved word in SQL99.

Maybe this?
--
Thierry Laronde (Alceste) <tlaronde +AT+ polynum +dot+ com>
                 http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C

Hi Markus,

I had a similar error once when I used 'data' as a field name.

Kirk

On Feb 27, 2007, at 4:38 PM, Markus Neteler wrote:

Hi,

I try to import a map from PostgreSQL into GRASS/DBF driver
with v.in.db, but get

DBMI-DBF driver error:
SQL parser error in statement:
create table ticks_feltre_2002_2006_sites ( ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer )
Cannot create table

but I don't see where the evil is:

echo " ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer" | tr -s ',' '\n' | sort
adf integer
adm integer
adtot integer
comune varchar(250)
data date
east double precision
ID integer
larve integer
localita varchar(250)
mapsheet varchar(250)
mgrs varchar(250)
microsito varchar(250)
ninfe integer
north double precision
note varchar(250)
ora time
periodo_prelievo varchar(250)
siteid integer
stazione varchar(250)
tcelsius double precision
umidrel double precision

Any ideas? Of course it would be fancy to have a more reasonable
error message. This error uses to pop up if preserved words are used
as column names, but I don't see any (even, it should tell me in an
ideal world).

Markus

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

On Wed, Feb 28, 2007 at 12:08:23AM +0100, tlaronde@polynum.com wrote:

On Tue, Feb 27, 2007 at 11:38:01PM +0100, Markus Neteler wrote:
> Hi,
>
> I try to import a map from PostgreSQL into GRASS/DBF driver
> with v.in.db, but get
>[..]
> adf integer
> adm integer
> adtot integer
> comune varchar(250)
> data date

From:
http://www.postgresql.org/docs/8.2/interactive/sql-keywords-appendix.html#KEYWORDS-TABLE

data is a reserved word in SQL99.

Maybe this?

Thanks for the useful list!

But I don't think that this is the problem because
- I am using "data" often (but not "date")
- this is the DBF backend

Markus

Hi Kirk,

I renamed the column to "datum" and still it fails.

Markus

On Tue, Feb 27, 2007 at 05:12:00PM -0600, Kirk Wythers wrote:

Hi Markus,

I had a similar error once when I used 'data' as a field name.

Kirk

On Feb 27, 2007, at 4:38 PM, Markus Neteler wrote:

>Hi,
>
>I try to import a map from PostgreSQL into GRASS/DBF driver
>with v.in.db, but get
>
>DBMI-DBF driver error:
>SQL parser error in statement:
>create table ticks_feltre_2002_2006_sites ( ID integer, comune
>varchar(250), localita varchar(250), microsito varchar(250), data
>date, ora time, larve integer, ninfe integer, adm integer, adf
>integer, adtot integer, tcelsius double precision, umidrel double
>precision, note varchar(250), mapsheet varchar(250), mgrs varchar
>(250), stazione varchar(250), periodo_prelievo varchar(250), north
>double precision, east double precision, siteid integer )
>Cannot create table
>
>but I don't see where the evil is:
>
>echo " ID integer, comune varchar(250), localita varchar(250),
>microsito varchar(250), data date, ora time, larve integer, ninfe
>integer, adm integer, adf integer, adtot integer, tcelsius double
>precision, umidrel double precision, note varchar(250), mapsheet
>varchar(250), mgrs varchar(250), stazione varchar(250),
>periodo_prelievo varchar(250), north double precision, east double
>precision, siteid integer" | tr -s ',' '\n' | sort
> adf integer
> adm integer
> adtot integer
> comune varchar(250)
> data date
> east double precision
> ID integer
> larve integer
> localita varchar(250)
> mapsheet varchar(250)
> mgrs varchar(250)
> microsito varchar(250)
> ninfe integer
> north double precision
> note varchar(250)
> ora time
> periodo_prelievo varchar(250)
> siteid integer
> stazione varchar(250)
> tcelsius double precision
> umidrel double precision
>
>Any ideas? Of course it would be fancy to have a more reasonable
>error message. This error uses to pop up if preserved words are used
>as column names, but I don't see any (even, it should tell me in an
>ideal world).
>
>Markus
>
>_______________________________________________
>grass-dev mailing list
>grass-dev@grass.itc.it
>http://grass.itc.it/mailman/listinfo/grass-dev

Ok, another try:

On Tue, Feb 27, 2007 at 11:38:01PM +0100, Markus Neteler wrote:

periodo_prelievo varchar(250)

DBF has a limit of 10 characters for a field (column) name. If the DBF
back-end does not silently truncate and fails, that's perhaps this.

2nd round :slight_smile:
--
Thierry Laronde (Alceste) <tlaronde +AT+ polynum +dot+ com>
                 http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C

Markus Neteler wrote:

> >I try to import a map from PostgreSQL into GRASS/DBF driver
> >with v.in.db, but get
> >
> >DBMI-DBF driver error:
> >SQL parser error in statement:
> >create table ticks_feltre_2002_2006_sites ( ID integer, comune
> >varchar(250), localita varchar(250), microsito varchar(250), data
> >date, ora time, larve integer, ninfe integer, adm integer, adf
> >integer, adtot integer, tcelsius double precision, umidrel double
> >precision, note varchar(250), mapsheet varchar(250), mgrs varchar
> >(250), stazione varchar(250), periodo_prelievo varchar(250), north
> >
> >double precision, east double precision, siteid integer )
> >Cannot create table
> >
> >but I don't see where the evil is:
> >
> >echo " ID integer, comune varchar(250), localita varchar(250),
> >microsito varchar(250), data date, ora time, larve integer, ninfe
> >integer, adm integer, adf integer, adtot integer, tcelsius double
> >precision, umidrel double precision, note varchar(250), mapsheet
> >varchar(250), mgrs varchar(250), stazione varchar(250),
> >periodo_prelievo varchar(250), north double precision, east double
> >
> >precision, siteid integer" | tr -s ',' '\n' | sort
> > adf integer
> > adm integer
> > adtot integer
> > comune varchar(250)
> > data date
> > east double precision
> > ID integer
> > larve integer
> > localita varchar(250)
> > mapsheet varchar(250)
> > mgrs varchar(250)
> > microsito varchar(250)
> > ninfe integer
> > north double precision
> > note varchar(250)
> > ora time
> > periodo_prelievo varchar(250)
> > siteid integer
> > stazione varchar(250)
> > tcelsius double precision
> > umidrel double precision
> >
> >Any ideas? Of course it would be fancy to have a more reasonable
> >error message. This error uses to pop up if preserved words are
> >used as column names, but I don't see any (even, it should tell me
> >in an ideal world).

cutting the list in half, trialing that, cutting in half again, etc.
shows that "ora time" is the offending entry.

Hamish

Markus Neteler wrote:

Hi,

I try to import a map from PostgreSQL into GRASS/DBF driver
with v.in.db, but get

DBMI-DBF driver error:
SQL parser error in statement:
create table ticks_feltre_2002_2006_sites ( ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer )
Cannot create table

but I don't see where the evil is:

echo " ID integer, comune varchar(250), localita varchar(250), microsito varchar(250), data date, ora time, larve integer, ninfe integer, adm integer, adf integer, adtot integer, tcelsius double precision, umidrel double precision, note varchar(250), mapsheet varchar(250), mgrs varchar(250), stazione varchar(250), periodo_prelievo varchar(250), north double precision, east double precision, siteid integer" | tr -s ',' '\n' | sort
adf integer
adm integer
adtot integer
comune varchar(250)
data date
east double precision
ID integer
larve integer
localita varchar(250)
mapsheet varchar(250)
mgrs varchar(250)
microsito varchar(250)
ninfe integer
north double precision
note varchar(250)
ora time
periodo_prelievo varchar(250)
siteid integer
stazione varchar(250)
tcelsius double precision
umidrel double precision

Any ideas?

As Hamish points out, it's "ora time" ("time" isn't a valid type for the
DBF driver).

Of course it would be fancy to have a more reasonable
error message. This error uses to pop up if preserved words are used
as column names, but I don't see any (even, it should tell me in an
ideal world).

Try defining YYERROR_VERBOSE in yac.y, i.e.:

--- lib/db/sqlp/yac.y 5 Feb 2007 10:45:38 -0000 1.25
+++ lib/db/sqlp/yac.y 28 Feb 2007 06:46:04 -0000
@@ -25,6 +25,7 @@
#include <grass/sqlp.h>

#define YYDEBUG 1
+#define YYERROR_VERBOSE 1

%}

This should give error messages of the form "unexpected X, expecting Y",
where X and Y are the terminal (token) names used in the grammar.

[Except that db.execute completely ignores the error message returned by
the parser in sqlpStmt->errmsg. Actually, AFAICT, it's the DBF driver
which completely ignores the error message ("grep errmsg *" in
db/drivers/dbf comes up blank).]

That's still not ideal, as the error messages won't identify the
specific value; e.g. for the above case, you just get "unexpected NAME".

If you want more detailed error messages, you have to add explicit rules
which match each erroneous case and use YYERROR to flag an error, e.g.:

y_columndef:
    NAME VARCHAR '(' INTNUM ')' { sqpColumnDef( $1, SQLP_VARCHAR, $4, 0 ); }
  | NAME INT { sqpColumnDef( $1, SQLP_INTEGER, 0, 0 ); }
  | NAME INTEGER { sqpColumnDef( $1, SQLP_INTEGER, 0, 0 ); }
  | NAME DOUBLE { sqpColumnDef( $1, SQLP_DOUBLE, 0, 0 ); }
  | NAME DOUBLE PRECISION { sqpColumnDef( $1, SQLP_DOUBLE, 0, 0 ); }
  | NAME DATE { sqpColumnDef( $1, SQLP_DATE, 0, 0 ); }
+ | NAME NAME {
+ char buff[1000];
+ sprintf(buf, "%s: invalid type: %s", $1, $2);
+ yyerror(buf);
+ YYERROR;
+ }
  ;

Adding the above to yac.y (and hacking yyerror() to actually print the
error rather than just return it so that the driver can ignore it)
results in "ora: invalid type: time".

The problem here is that you essentially need to enumerate all of the
different syntax errors which you want to catch. You also need to ensure
that you don't make a rule too broad and end up treating valid
statements as syntax errors.

--
Glynn Clements <glynn@gclements.plus.com>

On Wed, Feb 28, 2007 at 02:01:29PM +1300, Hamish wrote:

Markus Neteler wrote:
> > >I try to import a map from PostgreSQL into GRASS/DBF driver
> > >with v.in.db, but get
> > >
> > >DBMI-DBF driver error:
> > >SQL parser error in statement:
> > >create table ticks_feltre_2002_2006_sites ( ID integer, comune
> > >varchar(250), localita varchar(250), microsito varchar(250), data
> > >date, ora time, larve integer, ninfe integer, adm integer, adf
> > >integer, adtot integer, tcelsius double precision, umidrel double
> > >precision, note varchar(250), mapsheet varchar(250), mgrs varchar
> > >(250), stazione varchar(250), periodo_prelievo varchar(250), north
> > >
> > >double precision, east double precision, siteid integer )
> > >Cannot create table
> > >
> > >but I don't see where the evil is:
> > >
> > >echo " ID integer, comune varchar(250), localita varchar(250),
> > >microsito varchar(250), data date, ora time, larve integer, ninfe
> > >integer, adm integer, adf integer, adtot integer, tcelsius double
> > >precision, umidrel double precision, note varchar(250), mapsheet
> > >varchar(250), mgrs varchar(250), stazione varchar(250),
> > >periodo_prelievo varchar(250), north double precision, east double
> > >
> > >precision, siteid integer" | tr -s ',' '\n' | sort
> > > adf integer
> > > adm integer
> > > adtot integer
> > > comune varchar(250)
> > > data date
> > > east double precision
> > > ID integer
> > > larve integer
> > > localita varchar(250)
> > > mapsheet varchar(250)
> > > mgrs varchar(250)
> > > microsito varchar(250)
> > > ninfe integer
> > > north double precision
> > > note varchar(250)
> > > ora time
> > > periodo_prelievo varchar(250)
> > > siteid integer
> > > stazione varchar(250)
> > > tcelsius double precision
> > > umidrel double precision
> > >
> > >Any ideas? Of course it would be fancy to have a more reasonable
> > >error message. This error uses to pop up if preserved words are
> > >used as column names, but I don't see any (even, it should tell me
> > >in an ideal world).

cutting the list in half, trialing that, cutting in half again, etc.
shows that "ora time" is the offending entry.

Hamish - thanks for finding this one - I was confused by "ora"
but in fact "time" wasn't supported!

I have now added "time" type support to the parser and
voila' it works.

Thanks,
Markus

On Wed, Feb 28, 2007 at 07:15:47AM +0000, Glynn Clements wrote:

Markus Neteler wrote:

...

> Any ideas?

As Hamish points out, it's "ora time" ("time" isn't a valid type for the
DBF driver).

I have added support for "time" now.

> Of course it would be fancy to have a more reasonable
> error message. This error uses to pop up if preserved words are used
> as column names, but I don't see any (even, it should tell me in an
> ideal world).

Try defining YYERROR_VERBOSE in yac.y, i.e.:

--- lib/db/sqlp/yac.y 5 Feb 2007 10:45:38 -0000 1.25
+++ lib/db/sqlp/yac.y 28 Feb 2007 06:46:04 -0000
@@ -25,6 +25,7 @@
#include <grass/sqlp.h>

#define YYDEBUG 1
+#define YYERROR_VERBOSE 1

%}

This should give error messages of the form "unexpected X, expecting Y",
where X and Y are the terminal (token) names used in the grammar.

[Except that db.execute completely ignores the error message returned by
the parser in sqlpStmt->errmsg. Actually, AFAICT, it's the DBF driver
which completely ignores the error message ("grep errmsg *" in
db/drivers/dbf comes up blank).]

I see - but I have no clue how to implement that.

That's still not ideal, as the error messages won't identify the
specific value; e.g. for the above case, you just get "unexpected NAME".

If you want more detailed error messages, you have to add explicit rules
which match each erroneous case and use YYERROR to flag an error, e.g.:

y_columndef:
    NAME VARCHAR '(' INTNUM ')' { sqpColumnDef( $1, SQLP_VARCHAR, $4, 0 ); }
  | NAME INT { sqpColumnDef( $1, SQLP_INTEGER, 0, 0 ); }
  | NAME INTEGER { sqpColumnDef( $1, SQLP_INTEGER, 0, 0 ); }
  | NAME DOUBLE { sqpColumnDef( $1, SQLP_DOUBLE, 0, 0 ); }
  | NAME DOUBLE PRECISION { sqpColumnDef( $1, SQLP_DOUBLE, 0, 0 ); }
  | NAME DATE { sqpColumnDef( $1, SQLP_DATE, 0, 0 ); }
+ | NAME NAME {
+ char buff[1000];
+ sprintf(buf, "%s: invalid type: %s", $1, $2);
+ yyerror(buf);
+ YYERROR;
+ }
  ;

Adding the above to yac.y (and hacking yyerror() to actually print the
error rather than just return it so that the driver can ignore it)
results in "ora: invalid type: time".

This would be perfect. I implemented locally you suggestion but the DBF
driver remains silent 8due to above indicated problem which I don't know
how to solve).

The problem here is that you essentially need to enumerate all of the
different syntax errors which you want to catch. You also need to ensure
that you don't make a rule too broad and end up treating valid
statements as syntax errors.

This is not fully clear to me.

I better to try to add better error testing to not break
it completely.

Markus

On Wed, Feb 28, 2007 at 08:32:23AM +0100, Markus Neteler wrote:
...

I better to try to add better error testing to not break
it completely.

?

Wanted to say:
I better leave this to someone else to not break it completely.

Markus

Markus Neteler wrote:

> [Except that db.execute completely ignores the error message returned by
> the parser in sqlpStmt->errmsg. Actually, AFAICT, it's the DBF driver
> which completely ignores the error message ("grep errmsg *" in
> db/drivers/dbf comes up blank).]

I see - but I have no clue how to implement that.

The two quick options are:

1. Modify yyerror() (lib/db/sqlp/lex.l, line 246) to actually print
the error message, e.g.:

  void yyerror( char *s )
  {
    fprintf(stderr, "%s\n", s);
    ..
  }

2. Modify the DBF driver (db/drivers/dbf/dbfexe.c, line 65):

    if (yyparse() != 0) {
  sqpFreeStmt(st);
  G_free ( tmpsql) ;
  append_error("SQL parser error in statement:\n%s\n", sql);
  return DB_FAILED;
    }

to include st->errmsg somewhere in the output.

However: the notion of returning an error message back to the caller
is flawed. If the parser performed error recovery (rather than
aborting at the first error), you could get multiple error messages
for a single SQL statement (e.g. one for each invalid column
definition). In that case, making yyerror() simply print the message
to stderr (as r.mapcalc's parser does) would be preferred.

OTOH, if lib/db/sqlp is supposed to be a general-purpose SQL parser
library, a hard-coded error handling which prints to stderr might not
be desirable. In that case, you would probably want to use a callback.

As is often the case, deciding what to do is harder than actually
doing it.

--
Glynn Clements <glynn@gclements.plus.com>