[GRASS-dev] Re: [GRASS-stats] Re: [R-sig-Geo] writing shapefiles / DBF files when input data contains NA

On Tue, 7 Oct 2008, Dylan Beaudette wrote:

On Tue, Oct 7, 2008 at 6:26 PM, Hamish <hamish_b@yahoo.com> wrote:

Dylan:

It looks like the limiting factor in this equation is the
code used in v.out.ogr.

Following Dylan's posting on the GDAL list, and Frank's response, I suggest the following simple patch to vector/v.out.ogr/main.c (here 6.3):

$ diff -u main.c_old main.c
--- main.c_old 2008-10-09 10:54:19.000000000 +0200
+++ main.c 2008-10-09 11:30:34.000000000 +0200
@@ -625,7 +625,9 @@
                         colsqltype = db_get_column_sqltype(Column);
                         colctype = db_sqltype_to_Ctype ( colsqltype );
                         G_debug (2, " colctype = %d", colctype );
- switch ( colctype ) {
+/* RSB 081009 emit unset fields */
+ if (!db_test_value_isnull(Value)) {
+ switch ( colctype ) {
                              case DB_C_TYPE_INT:
                                 OGR_F_SetFieldInteger( Ogr_feature, j, db_get_va
lue_int(Value) );
                                 break;
@@ -639,7 +641,8 @@
                                 db_convert_column_value_to_string (Column, &dbst
ring);
                                 OGR_F_SetFieldString( Ogr_feature, j, db_get_str
ing (&dbstring) );
                                 break;
- }
+ }
+ } /* RSB */
                     }
                 }
             }

In 6.4 this is after line 717. Essentially it just uses db_test_value_isnull() not to set values in OGR fields if the DB field value is NULL, and follows Frank's suggestion.

This matches code near line 939 in OGRGRASSLayer::SetAttributes()
gdal/ogr/ogr_frmts/grass/ogrgrasslayer.cpp in the vector plugin, which uses:

if ( !db_test_value_isnull(value) )

I'm sure the patch needs checking, but with changes in the R rgdal package to support vector null data correctly, it ought to improve the interface.

Best wishes,

Roger

maybe a silly question, but is a 3rd party format even needed here?

$ ogrinfo --formats | grep -i grass
-> "GRASS" (readonly)

at least in the one direction.

Excellent question. I had also wondered about this. It looks like
there is a new argument in readVECT():

# read in directly with GDAL/OGR -- no intermediate file:
x <- readVECT6('xxx', plugin=TRUE)

This is quite fast and depends on the GDAL-GRASS plugin... However,
NULL data in a GRASS table is not imported correctly-- character
fields are imported as '', and numeric fields as 0.

So... Is the error in GDAL itself?

I tried inspecting a vector from GRASS with NULL data in some of the
columns from the table, using ogrinfo -al
location/mapset/vector/xxx/head

OGRFeature(1):23
cat (Integer) = 24
cat_ (Integer) = 24
str1 (String) = (null)
xyz (Real) = (null)
abc (Integer) = (null)
POINT (591583 4925280 0)

... which seems to correctly report the NULL values.

This leads me to suspect that something in readOGR() and writeOGR are
at fault in handling of NULL values.

Unfortunately looking at the rgdal source code wasn't very productive
(my fault).

Cheers,

Dylan

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand@nhh.no

On Thursday 09 October 2008, Roger Bivand wrote:

On Tue, 7 Oct 2008, Dylan Beaudette wrote:
> On Tue, Oct 7, 2008 at 6:26 PM, Hamish <hamish_b@yahoo.com> wrote:
>> Dylan:
>>> It looks like the limiting factor in this equation is the
>>> code used in v.out.ogr.

Following Dylan's posting on the GDAL list, and Frank's response, I
suggest the following simple patch to vector/v.out.ogr/main.c (here 6.3):

$ diff -u main.c_old main.c
--- main.c_old 2008-10-09 10:54:19.000000000 +0200
+++ main.c 2008-10-09 11:30:34.000000000 +0200
@@ -625,7 +625,9 @@
                         colsqltype = db_get_column_sqltype(Column);
                         colctype = db_sqltype_to_Ctype ( colsqltype );
                         G_debug (2, " colctype = %d", colctype );
- switch ( colctype ) {
+/* RSB 081009 emit unset fields */
+ if (!db_test_value_isnull(Value)) {
+ switch ( colctype ) {
                              case DB_C_TYPE_INT:
                                 OGR_F_SetFieldInteger( Ogr_feature, j,
db_get_va lue_int(Value) );
                                 break;
@@ -639,7 +641,8 @@
                                 db_convert_column_value_to_string (Column,
&dbst
ring);
                                 OGR_F_SetFieldString( Ogr_feature, j,
db_get_str ing (&dbstring) );
                                 break;
- }
+ }
+ } /* RSB */
                     }
                 }
             }

In 6.4 this is after line 717. Essentially it just uses
db_test_value_isnull() not to set values in OGR fields if the DB field
value is NULL, and follows Frank's suggestion.

OK-- I am using GRASS 6.4 SVN. Since this is the current (stable) developer
release it might be good to stick with it. I am using r33792 .

Before making the suggested changes, I have tried exporting some point data
with v.out.ogr:

# known working GRASS file with categories
v.out.ogr -e -c in=a_temp dsn=v.out.ogr_example layer=a_temp type=point

Exporting 25 points/lines...
100%
0 features written
WARNING: 25 features found without category skip
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This looks like a new problem (?)... I know that these points have categories,
why is v.out.ogr ignoring them?

Do these lines have anything to do with this:

v.out.ogr:434
Vect_cat_get(Cats, field, &cat);
      if (cat < 0 && !donocat) { /* Do not export not labeled */
    nocatskip++;
    continue;
      }

.... NO!
There is some kind of logic-related bug:

# does not export any features,
# complaining that they don't have categories
v.out.ogr -c in=xxx dsn=some_folder layer=new_shpfile

# exported expected features
v.out.ogr -c in=xxx dsn=new_shpfile

OK- posted here:
http://trac.osgeo.org/grass/ticket/327

The suggested fix appears to result in the creation of a DBF file (in the case
of shapefile) with properly encoded NULL.

Attached is a patch that applies this change to the develbranch-64 tree.

Should I make a ticket for this patch?

Cheers,

Dylan

This matches code near line 939 in OGRGRASSLayer::SetAttributes()
gdal/ogr/ogr_frmts/grass/ogrgrasslayer.cpp in the vector plugin, which
uses:

if ( !db_test_value_isnull(value) )

I'm sure the patch needs checking, but with changes in the R rgdal package
to support vector null data correctly, it ought to improve the interface.

Best wishes,

Roger

>> maybe a silly question, but is a 3rd party format even needed here?
>>
>> $ ogrinfo --formats | grep -i grass
>> -> "GRASS" (readonly)
>>
>> at least in the one direction.
>
> Excellent question. I had also wondered about this. It looks like
> there is a new argument in readVECT():
>
> # read in directly with GDAL/OGR -- no intermediate file:
> x <- readVECT6('xxx', plugin=TRUE)
>
> This is quite fast and depends on the GDAL-GRASS plugin... However,
> NULL data in a GRASS table is not imported correctly-- character
> fields are imported as '', and numeric fields as 0.
>
> So... Is the error in GDAL itself?
>
> I tried inspecting a vector from GRASS with NULL data in some of the
> columns from the table, using ogrinfo -al
> location/mapset/vector/xxx/head
>
> OGRFeature(1):23
> cat (Integer) = 24
> cat_ (Integer) = 24
> str1 (String) = (null)
> xyz (Real) = (null)
> abc (Integer) = (null)
> POINT (591583 4925280 0)
>
> ... which seems to correctly report the NULL values.
>
> This leads me to suspect that something in readOGR() and writeOGR are
> at fault in handling of NULL values.
>
> Unfortunately looking at the rgdal source code wasn't very productive
> (my fault).
>
> Cheers,
>
> Dylan

--
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341

(attachments)

v.out.ogr-null_value_support.patch (1.62 KB)

On Mon, 13 Oct 2008, Dylan Beaudette wrote:

On Thursday 09 October 2008, Roger Bivand wrote:

On Tue, 7 Oct 2008, Dylan Beaudette wrote:

On Tue, Oct 7, 2008 at 6:26 PM, Hamish <hamish_b@yahoo.com> wrote:

Dylan:

It looks like the limiting factor in this equation is the
code used in v.out.ogr.

Following Dylan's posting on the GDAL list, and Frank's response, I
suggest the following simple patch to vector/v.out.ogr/main.c (here 6.3):

$ diff -u main.c_old main.c
--- main.c_old 2008-10-09 10:54:19.000000000 +0200
+++ main.c 2008-10-09 11:30:34.000000000 +0200
@@ -625,7 +625,9 @@
                         colsqltype = db_get_column_sqltype(Column);
                         colctype = db_sqltype_to_Ctype ( colsqltype );
                         G_debug (2, " colctype = %d", colctype );
- switch ( colctype ) {
+/* RSB 081009 emit unset fields */
+ if (!db_test_value_isnull(Value)) {
+ switch ( colctype ) {
                              case DB_C_TYPE_INT:
                                 OGR_F_SetFieldInteger( Ogr_feature, j,
db_get_va lue_int(Value) );
                                 break;
@@ -639,7 +641,8 @@
                                 db_convert_column_value_to_string (Column,
&dbst
ring);
                                 OGR_F_SetFieldString( Ogr_feature, j,
db_get_str ing (&dbstring) );
                                 break;
- }
+ }
+ } /* RSB */
                     }
                 }
             }

In 6.4 this is after line 717. Essentially it just uses
db_test_value_isnull() not to set values in OGR fields if the DB field
value is NULL, and follows Frank's suggestion.

This matches code near line 939 in OGRGRASSLayer::SetAttributes()
gdal/ogr/ogr_frmts/grass/ogrgrasslayer.cpp in the vector plugin, which
uses:

if ( !db_test_value_isnull(value) )

I'm sure the patch needs checking, but with changes in the R rgdal package
to support vector null data correctly, it ought to improve the interface.

Best wishes,

Roger

Hi Roger:

Have the changes in the rgdal package made it into the mainstream code? I have
submitted tickets to the GRASS trac site for the v.out.ogr patch. The changes
you suggested appear to work well.

No sign of any change in the 6.3 or 6.4 trunks, unfortunately. 7.0 has the same bug. No response to the posting or your tickets. The only relevant thread was Markus' suggesting a name change in 7.0. Hamish: could you please prod the person(s) needed to move on ticket 333, and get them to do the same on 6.4 and 7.0 (the patch is for 6.3)?

Best wishes,

Roger

Cheers,

Dylan

maybe a silly question, but is a 3rd party format even needed here?

$ ogrinfo --formats | grep -i grass
-> "GRASS" (readonly)

at least in the one direction.

Excellent question. I had also wondered about this. It looks like
there is a new argument in readVECT():

# read in directly with GDAL/OGR -- no intermediate file:
x <- readVECT6('xxx', plugin=TRUE)

This is quite fast and depends on the GDAL-GRASS plugin... However,
NULL data in a GRASS table is not imported correctly-- character
fields are imported as '', and numeric fields as 0.

So... Is the error in GDAL itself?

I tried inspecting a vector from GRASS with NULL data in some of the
columns from the table, using ogrinfo -al
location/mapset/vector/xxx/head

OGRFeature(1):23
cat (Integer) = 24
cat_ (Integer) = 24
str1 (String) = (null)
xyz (Real) = (null)
abc (Integer) = (null)
POINT (591583 4925280 0)

... which seems to correctly report the NULL values.

This leads me to suspect that something in readOGR() and writeOGR are
at fault in handling of NULL values.

Unfortunately looking at the rgdal source code wasn't very productive
(my fault).

Cheers,

Dylan

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand@nhh.no

On Tue, Oct 14, 2008 at 8:59 AM, Roger Bivand <Roger.Bivand@nhh.no> wrote:

On Mon, 13 Oct 2008, Dylan Beaudette wrote:

Have the changes in the rgdal package made it into the mainstream code? I
have submitted tickets to the GRASS trac site for the v.out.ogr patch. The
changes you suggested appear to work well.

No sign of any change in the 6.3 or 6.4 trunks, unfortunately. 7.0 has the
same bug. No response to the posting or your tickets. The only relevant
thread was Markus' suggesting a name change in 7.0. Hamish: could you please
prod the person(s) needed to move on ticket 333, and get them to do the same
on 6.4 and 7.0 (the patch is for 6.3)?

I have fixed trax #333 now with minor modification (C++ style comments
are no good).
Applied to 6.3.svn, 6.4.svn, 7.svn.

Thanks for the fix,
Markus

On Tue, Oct 14, 2008 at 3:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Oct 14, 2008 at 8:59 AM, Roger Bivand <Roger.Bivand@nhh.no> wrote:

On Mon, 13 Oct 2008, Dylan Beaudette wrote:

Have the changes in the rgdal package made it into the mainstream code? I
have submitted tickets to the GRASS trac site for the v.out.ogr patch. The
changes you suggested appear to work well.

No sign of any change in the 6.3 or 6.4 trunks, unfortunately. 7.0 has the
same bug. No response to the posting or your tickets. The only relevant
thread was Markus' suggesting a name change in 7.0. Hamish: could you please
prod the person(s) needed to move on ticket 333, and get them to do the same
on 6.4 and 7.0 (the patch is for 6.3)?

I have fixed trax #333 now with minor modification (C++ style comments
are no good).
Applied to 6.3.svn, 6.4.svn, 7.svn.

Thanks for the fix,
Markus

Thanks for applying the fix Markus, and sorry about the poor comment
style-- that was my fault.

Dylan

On Tue, Oct 14, 2008 at 11:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Oct 14, 2008 at 8:59 AM, Roger Bivand <Roger.Bivand@nhh.no> wrote:

On Mon, 13 Oct 2008, Dylan Beaudette wrote:

Have the changes in the rgdal package made it into the mainstream code? I
have submitted tickets to the GRASS trac site for the v.out.ogr patch. The
changes you suggested appear to work well.

No sign of any change in the 6.3 or 6.4 trunks, unfortunately. 7.0 has the
same bug. No response to the posting or your tickets. The only relevant
thread was Markus' suggesting a name change in 7.0. Hamish: could you please
prod the person(s) needed to move on ticket 333, and get them to do the same
on 6.4 and 7.0 (the patch is for 6.3)?

I have fixed trax #333 now with minor modification (C++ style comments
are no good).
Applied to 6.3.svn, 6.4.svn, 7.svn.

Sorry, discovered only now that the 6.4.svn fix was uncommitted on my
laptop. Submitted as revision 34049, unfortunately with wrong ticket
reference.

So, now this issue should be really solved.

Markus

On Wed, Oct 29, 2008 at 3:00 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Oct 14, 2008 at 11:13 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Tue, Oct 14, 2008 at 8:59 AM, Roger Bivand <Roger.Bivand@nhh.no> wrote:

On Mon, 13 Oct 2008, Dylan Beaudette wrote:

Have the changes in the rgdal package made it into the mainstream code? I
have submitted tickets to the GRASS trac site for the v.out.ogr patch. The
changes you suggested appear to work well.

No sign of any change in the 6.3 or 6.4 trunks, unfortunately. 7.0 has the
same bug. No response to the posting or your tickets. The only relevant
thread was Markus' suggesting a name change in 7.0. Hamish: could you please
prod the person(s) needed to move on ticket 333, and get them to do the same
on 6.4 and 7.0 (the patch is for 6.3)?

I have fixed trax #333 now with minor modification (C++ style comments
are no good).
Applied to 6.3.svn, 6.4.svn, 7.svn.

Sorry, discovered only now that the 6.4.svn fix was uncommitted on my
laptop. Submitted as revision 34049, unfortunately with wrong ticket
reference.

So, now this issue should be really solved.

Markus

Thanks Markus. I'll update my local copy and give it a try.

This should fix several long-standing issues with moving data from GRASS->R.

Cheers,

Dylan