[GRASS-dev] adding 'desc' to dbf sql driver

Hi,

Reworking on d.vect.chart I need to be able to sort the reults of a select in descending order. As the dbf driver doesn't allow this, I have been trying to see how to extend the driver.

IIUC, it's "just" a question of conditionalising the qsort call on line 566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use

qsort(set, nset, sizeof(int), -cmp_row);

if the desc keyword is present.

However, I have some trouble understanding the parsing of the sql statement.

If I understand correctly, we would need to extent the SQLPSTMT structure in include/sqlp.h to include a flag for 'desc' (and possible 'asc') and the parser to identify and set that flag.

But this is as far as I get. Could someone help me with this ?

Thanks,

Moritz

Hi Moritz. See below

On 9/22/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:

Hi,

Reworking on d.vect.chart I need to be able to sort the reults of a
select in descending order. As the dbf driver doesn't allow this, I have
been trying to see how to extend the driver.

IIUC, it's "just" a question of conditionalising the qsort call on line
566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use

qsort(set, nset, sizeof(int), -cmp_row);

That might work. Although it would be much nicer to use an expression
as sorting order.

if the desc keyword is present.

However, I have some trouble understanding the parsing of the sql
statement.

If I understand correctly, we would need to extent the SQLPSTMT
structure in include/sqlp.h to include a flag for 'desc' (and possible
'asc') and the parser to identify and set that flag.

But this is as far as I get. Could someone help me with this ?

Quick shot: you need to

- change SQLPSTMT to have an "order"/"direction"/"sort_order" element

- add ASC and DESC in lex.l as tokens

- change yac.y to stg like:

y_order: y_order_asc | y_order_desc ;
y_order_asc:
       NAME ASC { sqpOrderColumn( $1 ); sqpOrderDirection( 1 ); }
;
y_order_desc:
      NAME DESC { sqpOrderColumn( $1 );sqpOrderDirection( 2 );}
;

- add in sql.c a sqpOrderDirection() function that sets the element
you added to SQLPSTMT

(Alternatively, you can change sqpOrderColumn() to receive two
arguments, one being the sorting direction. I'd also rather #define
SORT_ASC and SORT_DESC instead of using magic numbers as above.)

- use that value in dbfexe.c to test the sorting direction.

HTH,

Daniel.

Thanks,

Moritz

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

--
-- Daniel Calvelo Aros

--
-- Daniel Calvelo Aros

On Thu, September 28, 2006 03:31, Daniel Calvelo wrote:

Hi Moritz. See below

Thanks Daniel. As soon as I find the time, I'll try to translate your
recipe into code...

Moritz

On 9/22/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:

Hi,

Reworking on d.vect.chart I need to be able to sort the reults of a
select in descending order. As the dbf driver doesn't allow this, I have
been trying to see how to extend the driver.

IIUC, it's "just" a question of conditionalising the qsort call on line
566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use

qsort(set, nset, sizeof(int), -cmp_row);

That might work. Although it would be much nicer to use an expression
as sorting order.

if the desc keyword is present.

However, I have some trouble understanding the parsing of the sql
statement.

If I understand correctly, we would need to extent the SQLPSTMT
structure in include/sqlp.h to include a flag for 'desc' (and possible
'asc') and the parser to identify and set that flag.

But this is as far as I get. Could someone help me with this ?

Quick shot: you need to

- change SQLPSTMT to have an "order"/"direction"/"sort_order" element

- add ASC and DESC in lex.l as tokens

- change yac.y to stg like:

y_order: y_order_asc | y_order_desc ;
y_order_asc:
       NAME ASC { sqpOrderColumn( $1 ); sqpOrderDirection( 1 ); }
;
y_order_desc:
      NAME DESC { sqpOrderColumn( $1 );sqpOrderDirection( 2 );}
;

- add in sql.c a sqpOrderDirection() function that sets the element
you added to SQLPSTMT

(Alternatively, you can change sqpOrderColumn() to receive two
arguments, one being the sorting direction. I'd also rather #define
SORT_ASC and SORT_DESC instead of using magic numbers as above.)

- use that value in dbfexe.c to test the sorting direction.

HTH,

Daniel.

Thanks,

Moritz

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

--
-- Daniel Calvelo Aros

Hi Daniel,

On Thu, September 28, 2006 03:31, Daniel Calvelo wrote:

Hi Moritz. See below

On 9/22/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:

Hi,

Reworking on d.vect.chart I need to be able to sort the reults of a
select in descending order. As the dbf driver doesn't allow this, I have
been trying to see how to extend the driver.

IIUC, it's "just" a question of conditionalising the qsort call on line
566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use

qsort(set, nset, sizeof(int), -cmp_row);

That might work. Although it would be much nicer to use an expression
as sorting order.

if the desc keyword is present.

However, I have some trouble understanding the parsing of the sql
statement.

If I understand correctly, we would need to extent the SQLPSTMT
structure in include/sqlp.h to include a flag for 'desc' (and possible
'asc') and the parser to identify and set that flag.

But this is as far as I get. Could someone help me with this ?

Quick shot: you need to

- change SQLPSTMT to have an "order"/"direction"/"sort_order" element

- add ASC and DESC in lex.l as tokens

- change yac.y to stg like:

y_order: y_order_asc | y_order_desc ;
y_order_asc:
       NAME ASC { sqpOrderColumn( $1 ); sqpOrderDirection( 1 ); }
;
y_order_desc:
      NAME DESC { sqpOrderColumn( $1 );sqpOrderDirection( 2 );}
;

- add in sql.c a sqpOrderDirection() function that sets the element
you added to SQLPSTMT

(Alternatively, you can change sqpOrderColumn() to receive two
arguments, one being the sorting direction. I'd also rather #define
SORT_ASC and SORT_DESC instead of using magic numbers as above.)

- use that value in dbfexe.c to test the sorting direction.

Here's an attempt at translating your very clear instructions. Could you
or someone else just have a quick look to see if I didn't do anything bad.
One question for me was how to handle the different comparison functions
for ASC and DESC. At this stage I just duplicated the function
(cmp_row_asc and cpm_row_desc) and inverted the signs. Maybe there is a
more elegant way ?

Thanks !

Moritz

(attachments)

dbf.diff.tgz (1.87 KB)

Hi Moritz,

Looks good to me. In the yac.y patch, you don't need to duplicate the
code for ORDER BY NAME and ORDER BY NAME ASC. For cmp_row_desc, you
could just return( - cmp_row_asc) with the same arguments. Glynn, a
cursory look?

FYI, the text file in lib/db/sqlp/test/test allows for some testing of
the parser. That may help you find any hidden bugs in the sqlp part of
your patch. You might also wish to update print.c to accomodate for
asc/desc (it's a one-line change near the end).

And of course, after testing, this has to be documented :slight_smile:

Daniel.

On 10/7/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:

Hi Daniel,

On Thu, September 28, 2006 03:31, Daniel Calvelo wrote:
> Hi Moritz. See below
>
> On 9/22/06, Moritz Lennert <mlennert@club.worldonline.be> wrote:
>> Hi,
>>
>> Reworking on d.vect.chart I need to be able to sort the reults of a
>> select in descending order. As the dbf driver doesn't allow this, I have
>> been trying to see how to extend the driver.
>>
>> IIUC, it's "just" a question of conditionalising the qsort call on line
>> 566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use
>>
>> qsort(set, nset, sizeof(int), -cmp_row);
>
> That might work. Although it would be much nicer to use an expression
> as sorting order.
>
>> if the desc keyword is present.
>>
>> However, I have some trouble understanding the parsing of the sql
>> statement.
>>
>> If I understand correctly, we would need to extent the SQLPSTMT
>> structure in include/sqlp.h to include a flag for 'desc' (and possible
>> 'asc') and the parser to identify and set that flag.
>>
>> But this is as far as I get. Could someone help me with this ?
>
> Quick shot: you need to
>
> - change SQLPSTMT to have an "order"/"direction"/"sort_order" element
>
> - add ASC and DESC in lex.l as tokens
>
> - change yac.y to stg like:
>
> y_order: y_order_asc | y_order_desc ;
> y_order_asc:
> NAME ASC { sqpOrderColumn( $1 ); sqpOrderDirection( 1 ); }
> ;
> y_order_desc:
> NAME DESC { sqpOrderColumn( $1 );sqpOrderDirection( 2 );}
> ;
>
> - add in sql.c a sqpOrderDirection() function that sets the element
> you added to SQLPSTMT
>
> (Alternatively, you can change sqpOrderColumn() to receive two
> arguments, one being the sorting direction. I'd also rather #define
> SORT_ASC and SORT_DESC instead of using magic numbers as above.)
>
> - use that value in dbfexe.c to test the sorting direction.

Here's an attempt at translating your very clear instructions. Could you
or someone else just have a quick look to see if I didn't do anything bad.
One question for me was how to handle the different comparison functions
for ASC and DESC. At this stage I just duplicated the function
(cmp_row_asc and cpm_row_desc) and inverted the signs. Maybe there is a
more elegant way ?

Thanks !

Moritz

--
-- Daniel Calvelo Aros

Moritz Lennert wrote:

Hi,

Reworking on d.vect.chart I need to be able to sort the reults of a
select in descending order. As the dbf driver doesn't allow this, I have
been trying to see how to extend the driver.

IIUC, it's "just" a question of conditionalising the qsort call on line
566 of db/drivers/dbf/dbfexe.c, and (would this be enough ?) use

qsort(set, nset, sizeof(int), -cmp_row);

That won't work. It might compile, but you're essentially converting
the address of the cmp_row() function to an integer, negating it, then
converting it back to a pointer.

However, you can do:

  static int cmp_row_desc(const void *pa, const void *pb)
  {
    return -cmp_row_asc(pa, pb);
  }

--
Glynn Clements <glynn@gclements.plus.com>

Daniel Calvelo wrote:

Hi Moritz,

Looks good to me. In the yac.y patch, you don't need to duplicate the
code for ORDER BY NAME and ORDER BY NAME ASC.

How do I take care of the case where neither asc nor desc are given in the sql ? This should default to asc, but this:

+y_order_asc:
+ NAME | NAME ASC { sqpOrderColumn( $1, SORT_ASC ); }

does not work, i.e. results of 'order by' without asc are not sorted.

So how does this need to be formulated ?

For cmp_row_desc, you
could just return( - cmp_row_asc) with the same arguments. Glynn, a
cursory look?

Did that and it seems to work.

FYI, the text file in lib/db/sqlp/test/test allows for some testing of
the parser. That may help you find any hidden bugs in the sqlp part of
your patch.

I get

Input row: -->>INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', +32, +56.7 );
<<--
Input statement: -->>INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', +32, +56.7 )<<--
Error: statement was not parsed successfully.

Don't know how this would be related to my changes.

(BTW

INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', -32, -56.7 );

does not cause any problems.)

If I remove the above insert statement and add:

SELECT c1,c3 FROM pok order by c1;

it runs perfectly.

However, when I insert

SELECT c1,c3 FROM pok order by c1 ASC;
or
SELECT c1,c3 FROM pok order by c1 DESC;

I get a segmentation fault:

Input row: -->>SELECT c1,c3 FROM pok order by c1 ASC;
<<--
Input statement: -->>SELECT c1,c3 FROM pok order by c1 ASC<<--
********** SQL PARSER RESULT **********
INPUT: SELECT c1,c3 FROM pok order by c1 ASC
COMMAND: SELECT
TABLE: pok
COLUMN 1: c1
COLUMN 2: c3
./test2: line 15: 32474 Done echo "SELECT c1,c3 FROM pok where c3 <> 34 and c5 = 2.3;
SELECT c1,c3 FROM pok order by c1 ASC;
INSERT INTO pok VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', -32, -56.7 );
upDAte pok SET c2 = '5', c3 = 1 WHERE c1 = '1' AND c5 = 6;
delete FROM pok WHERE c1 = '1' and c2=3 AND c5 <= 4.35;
CREATE TABLE pok ( c1 INT, c2 VARCHAR (5), c3 DOUBLE );
DROP TABLE pok;
ALTER TABLE pok ADD COLUMN id int;
ALTER TABLE pok ADD popis varchar(10);
UPDATE pok SET c2 = NULL, c1=c2, c3=(-c3+12)/c1 where c1 NOT NULL;
update pok set c1=c2,c2=c1,c3=NULL where c1+2>1/cat+0.5 and not (c1=1 or c2=2);" 32475 Segmentation fault | ./sqlptest

I'm a bit lost here. What is print.c used for ?

You might also wish to update print.c to accomodate for
asc/desc (it's a one-line change near the end).

Did that, but not sure if correctly. Should I check for the existance of orderDir before using it, i.e. should it rather be

if ( sqlpStmt->command == SQLP_SELECT ) {
     if ( sqlpStmt->orderDir )
        {
         fprintf( stderr, "ORDER BY: %s %s\n", sqlpStmt->orderCol, sqlpStmt->orderDir );
        } else {
         fprintf( stderr, "ORDER BY: %s\n", sqlpStmt->orderCol );
        }

or is this not necessary ?

And of course, after testing, this has to be documented :slight_smile:

Obviously... :slight_smile:

Moritz

Forgot to attach the latest patch. Here it is.

Moritz

Moritz Lennert wrote:

Daniel Calvelo wrote:

Hi Moritz,

Looks good to me. In the yac.y patch, you don't need to duplicate the
code for ORDER BY NAME and ORDER BY NAME ASC.

How do I take care of the case where neither asc nor desc are given in the sql ? This should default to asc, but this:

+y_order_asc:
+ NAME | NAME ASC { sqpOrderColumn( $1, SORT_ASC ); }

does not work, i.e. results of 'order by' without asc are not sorted.

So how does this need to be formulated ?

For cmp_row_desc, you
could just return( - cmp_row_asc) with the same arguments. Glynn, a
cursory look?

Did that and it seems to work.

FYI, the text file in lib/db/sqlp/test/test allows for some testing of
the parser. That may help you find any hidden bugs in the sqlp part of
your patch.

I get

Input row: -->>INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', +32, +56.7 );
<<--
Input statement: -->>INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', +32, +56.7 )<<--
Error: statement was not parsed successfully.

Don't know how this would be related to my changes.

(BTW

INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', -32, -56.7 );

does not cause any problems.)

If I remove the above insert statement and add:

SELECT c1,c3 FROM pok order by c1;

it runs perfectly.

However, when I insert

SELECT c1,c3 FROM pok order by c1 ASC;
or
SELECT c1,c3 FROM pok order by c1 DESC;

I get a segmentation fault:

Input row: -->>SELECT c1,c3 FROM pok order by c1 ASC;
<<--
Input statement: -->>SELECT c1,c3 FROM pok order by c1 ASC<<--
********** SQL PARSER RESULT **********
INPUT: SELECT c1,c3 FROM pok order by c1 ASC
COMMAND: SELECT
TABLE: pok
COLUMN 1: c1
COLUMN 2: c3
./test2: line 15: 32474 Done echo "SELECT c1,c3 FROM pok where c3 <> 34 and c5 = 2.3;
SELECT c1,c3 FROM pok order by c1 ASC;
INSERT INTO pok VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', -32, -56.7 );
upDAte pok SET c2 = '5', c3 = 1 WHERE c1 = '1' AND c5 = 6;
delete FROM pok WHERE c1 = '1' and c2=3 AND c5 <= 4.35;
CREATE TABLE pok ( c1 INT, c2 VARCHAR (5), c3 DOUBLE );
DROP TABLE pok;
ALTER TABLE pok ADD COLUMN id int;
ALTER TABLE pok ADD popis varchar(10);
UPDATE pok SET c2 = NULL, c1=c2, c3=(-c3+12)/c1 where c1 NOT NULL;
update pok set c1=c2,c2=c1,c3=NULL where c1+2>1/cat+0.5 and not (c1=1 or c2=2);" 32475 Segmentation fault | ./sqlptest

I'm a bit lost here. What is print.c used for ?

You might also wish to update print.c to accomodate for
asc/desc (it's a one-line change near the end).

Did that, but not sure if correctly. Should I check for the existance of orderDir before using it, i.e. should it rather be

if ( sqlpStmt->command == SQLP_SELECT ) {
    if ( sqlpStmt->orderDir )
       {
        fprintf( stderr, "ORDER BY: %s %s\n", sqlpStmt->orderCol, sqlpStmt->orderDir );
       } else {
        fprintf( stderr, "ORDER BY: %s\n", sqlpStmt->orderCol );
       }

or is this not necessary ?

And of course, after testing, this has to be documented :slight_smile:

Obviously... :slight_smile:

Moritz

(attachments)

dbf.diff.tgz (1.81 KB)

Moritz Lennert wrote:

> Looks good to me. In the yac.y patch, you don't need to duplicate the
> code for ORDER BY NAME and ORDER BY NAME ASC.

How do I take care of the case where neither asc nor desc are given in
the sql ? This should default to asc, but this:

+y_order_asc:
+ NAME | NAME ASC {
sqpOrderColumn( $1, SORT_ASC ); }

does not work, i.e. results of 'order by' without asc are not sorted.

So how does this need to be formulated ?

y_order_asc:
    NAME { sqpOrderColumn( $1, SORT_ASC ); }
  | NAME ASC { sqpOrderColumn( $1, SORT_ASC ); }
  ;

The | splits a definition into separate rules. Each rule has its own
action; if none is given, the default action is "{ $$ = $1; }". So, in
your original version, the specified action only applied to the
"NAME ASC" rule, while "NAME" got the default action.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

Moritz Lennert wrote:

Looks good to me. In the yac.y patch, you don't need to duplicate the
code for ORDER BY NAME and ORDER BY NAME ASC.

How do I take care of the case where neither asc nor desc are given in the sql ? This should default to asc, but this:

+y_order_asc:
+ NAME | NAME ASC { sqpOrderColumn( $1, SORT_ASC ); }

does not work, i.e. results of 'order by' without asc are not sorted.

So how does this need to be formulated ?

y_order_asc:
    NAME { sqpOrderColumn( $1, SORT_ASC ); }
  | NAME ASC { sqpOrderColumn( $1, SORT_ASC ); }
  ;

The | splits a definition into separate rules. Each rule has its own
action; if none is given, the default action is "{ $$ = $1; }". So, in
your original version, the specified action only applied to the
"NAME ASC" rule, while "NAME" got the default action.

This is how I did it in fact. Thanks.

I have just committed the series of changes necessary for the support of the ASC/DESC keywords in the ORDER BY clause in the dbf driver.

For me it works, so I thought the best would be to commit it so that others could test it.

The only issue I still have is with the test case in lib/db/sqlp/test/, but I don't know if that is a problem with my code, or with the testing...

Input row: -->>SELECT c1,c3 FROM pok order by c1 ASC;
<<--
Input statement: -->>SELECT c1,c3 FROM pok order by c1 ASC<<--
********** SQL PARSER RESULT **********
INPUT: SELECT c1,c3 FROM pok order by c1 ASC
COMMAND: SELECT
TABLE: pok
COLUMN 1: c1
COLUMN 2: c3
./test2: line 15: 32474 Done echo "SELECT c1,c3 FROM pok where c3 <> 34 and c5 = 2.3;
SELECT c1,c3 FROM pok order by c1 ASC;
INSERT INTO pok VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', 32, 56.7 );
INSERT INTO pok ( c1, c2,c3) VALUES ( 'abc', -32, -56.7 );
upDAte pok SET c2 = '5', c3 = 1 WHERE c1 = '1' AND c5 = 6;
delete FROM pok WHERE c1 = '1' and c2=3 AND c5 <= 4.35;
CREATE TABLE pok ( c1 INT, c2 VARCHAR (5), c3 DOUBLE );
DROP TABLE pok;
ALTER TABLE pok ADD COLUMN id int;
ALTER TABLE pok ADD popis varchar(10);
UPDATE pok SET c2 = NULL, c1=c2, c3=(-c3+12)/c1 where c1 NOT NULL;
update pok set c1=c2,c2=c1,c3=NULL where c1+2>1/cat+0.5 and not (c1=1 or c2=2);" 32475 Segmentation fault | ./sqlptest

Moritz