[GRASS-dev] [GRASS GIS] #354: v.out.ogr: bad DB read for format=GPX

#354: v.out.ogr: bad DB read for format=GPX
---------------------------+------------------------------------------------
Reporter: hamish | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Keywords: v.out.ogr GPX | Platform: Linux
      Cpu: x86-32 |
---------------------------+------------------------------------------------
Hi,

this is moved from gdal bug # 2638
  http://trac.osgeo.org/gdal/ticket/2638

GPX format is not reading the DB table correctly, then filling in the
category number as the elevation.

It works ok using ogr2ogr:

{{{
# Spearfish dataset:

# export shapefile
v.out.ogr in=archsites type=point dsn=archsites.shp

# get PROJ.4 params of current location (Spearfish=UTM 13N)
IN_PROJ=`g.proj -jf`

ogr2ogr -f GPX -dsco GPX_USE_EXTENSIONS=YES \
   -a_srs "$IN_PROJ" -t_srs EPSG:4326 \
   output.gpx archsites.shp
}}}

result:
{{{
<?xml version="1.0"?>
<gpx version="1.1" creator="GDAL 1.5.0" xmlns:xsi="http://www.w3.org/2001/
XMLSchema-instan
ce" xmlns:ogr="http://osgeo.org/gdal&quot;
xmlns="http://www.topografix.com/GPX/1/1&quot; xsi:schema
Location="http://www.topografix.com/GPX/1/1
http://www.topografix.com/GPX/1/1/gpx.xsd&quot;&gt;
<wpt lat="44.381771718385657" lon="-103.826359511024279">
   <extensions>
     <ogr:cat> 1</ogr:cat>
     <ogr:str1>Signature Rock</ogr:str1>
   </extensions>
</wpt>
<wpt lat="44.456408157151380" lon="-103.844259647363543">
   <extensions>
     <ogr:cat> 2</ogr:cat>
     <ogr:str1>No Name</ogr:str1>
   </extensions>
</wpt>
<wpt lat="44.447669817618234" lon="-103.870697678751810">
   <extensions>
     <ogr:cat> 3</ogr:cat>
     <ogr:str1>Canyon Station</ogr:str1>
   </extensions>
</wpt>
...
}}}

slight issue: cat number (column type integers) get padded?

but using GRASS's 'v.out.ogr format=GPX' (via v.out.gpsbabel script) I
get:
{{{
v.out.gpsbabel -w in=archsites out=archsites.gpx where="cat < 4" --verbose
Extracting data ...
Reprojecting data ...
Exporting data ...
WARNING: No attribute table found -> using only category numbers as
          attributes
Cleaning up ...
Fast exit.
}}}

(but there is an attribute table found, otherwise SQL where= query
would fail in the v.extract step)

and output:

{{{
<?xml version="1.0"?>
<gpx version="1.1" creator="GDAL 1.5.0" xmlns:xsi="http://www.w3.org/2001/
XMLSchema-instan
ce" xmlns:ogr="http://osgeo.org/gdal&quot;
xmlns="http://www.topografix.com/GPX/1/1&quot; xsi:schema
Location="http://www.topografix.com/GPX/1/1
http://www.topografix.com/GPX/1/1/gpx.xsd&quot;&gt;
<wpt lat="44.381755779999999" lon="-103.826838609999996">
   <ele>1</ele>
   <extensions>
   </extensions>
</wpt>
<wpt lat="44.456392070000000" lon="-103.844739759999996">
   <ele>2</ele>
   <extensions>
   </extensions>
</wpt>
<wpt lat="44.447653719999998" lon="-103.871178959999995">
   <ele>3</ele>
   <extensions>
   </extensions>
</wpt>
</gpx>
}}}

So it's a v.out.ogr bug I guess. I tried copying the vector into the
current mapset from PERMANENT, but it didn't help.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

also to note diff in lat/lon in above two examples:
{{{
<wpt lat="44.381771718385657" lon="-103.826359511024279">
<wpt lat="44.381755779999999" lon="-103.826838609999996">
}}}

I don't really know why that is different. Both ogr2ogr and m.proj (cs2cs
frontend script) should be using +nadgrids conus and identical setup
method..? strange.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

Replying to [comment:1 hamish]:
> also to note diff in lat/lon in above two examples:
> {{{
> <wpt lat="44.381771718385657" lon="-103.826359511024279">
> <wpt lat="44.381755779999999" lon="-103.826838609999996">
> }}}
>
> I don't really know why that is different. Both ogr2ogr and
> m.proj (cs2cs frontend script) should be using +nadgrids conus
> and identical setup method..? strange.

v.out.gps[babel] -> m.proj -> cs2cs is called with "-f %.8f", which
explains the FP quant in the second set, but the top one (ogr2ogr) changes
at ~ %.4f or .5 (>1m)

??
Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

It appears that ogr2ogr isn't applying the given +nadgrid file.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

Replying to [comment:3 hamish]:
> It appears that ogr2ogr isn't applying the given +nadgrid file.

Correct, first google hit for ogr2ogr+nadgrids gave the answer: for
ogr2ogr you need to append +wktext to the input SRS. Then it works.
     http://lists.osgeo.org/pipermail/gdal-dev/2007-April/012561.html

Is this an issue for the R-GRASS interface?

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

back to original issue,

this is a followup to #2638,
  https://trac.osgeo.org/gdal/ticket/2638#comment:1
cross-filed as GDAL bug # 2653
  https://trac.osgeo.org/gdal/ticket/2653

data intended for <extensions> is filling standard GPX fields
instead.

Exporting a points feature map as GPX:
(GRASS's North Carolina sample dataset)
{{{
# export GRASS points map as GPX format
GRASS64> v.out.ogr in=hospitals dsn=hsp.gpx format=GPX type=point \
    dsco="GPX_USE_EXTENSIONS=YES"

GRASS64> v.info -c hospitals
Displaying column types/names for database connection of layer 1:
INTEGER|cat
INTEGER|OBJECTID
DOUBLE PRECISION|AREA
DOUBLE PRECISION|PERIMETER
INTEGER|HLS_
INTEGER|HLS_ID
CHARACTER|NAME
CHARACTER|ADDRESS
CHARACTER|CITY
CHARACTER|ZIP
CHARACTER|COUNTY
CHARACTER|PHONE
CHARACTER|CANCER
INTEGER|POLYGONID
DOUBLE PRECISION|SCALE
DOUBLE PRECISION|ANGLE

# SQL: select * FROM hospitals ...
GRASS64> v.db.select hospitals | head -n 4
cat|OBJECTID|AREA|PERIMETER|HLS_|HLS_ID|NAME|ADDRESS|CITY|ZIP|COUNTY|PHONE|CANCER|POLYGONID|SCALE|ANGLE
1|1|0|0|1|1|Cherry Hospital|400 Stevens Mill
Rd|Goldsboro|27533|Wayne|(919) 731-3200|yes|0|1|1
2|2|0|0|2|2|Wayne Memorial Hospital|2700 Wayne Memorial
Dr|Goldsboro|27534|Wayne|(919) 736-1110|yes|0|1|1
3|3|0|0|3|3|Watauga Medical Center|336 Deerfield
Rd|Boone|28607|Watauga|(828) 262-4100|yes|0|1|1

$ cat hsp.gpx
<?xml version="1.0"?>
<gpx version="1.1" creator="GDAL 1.5.0" xmlns:xsi="http://www.w3.org/2001/
XMLSchema-instance" xmlns:ogr="http://osgeo.org/gdal&quot; xmlns="http:
//www.topografix.com/GPX/1/1"
xsi:schemaLocation="http://www.topografix.com/GPX/1/1
http://www.topografix.com/GPX/1/1/gpx.xsd&quot;&gt;
<wpt lat="182012.655400562332943" lon="697237.563861500239000">
   <ele>1</ele>
   <magvar>0</magvar>
   <geoidheight>0</geoidheight>
   <name>1</name>
   <cmt>1</cmt>
   <desc>Cherry Hospital</desc>
   <src>400 Stevens Mill Rd</src>
   <link href="Goldsboro"><text>27533</text><type>Wayne</type></link>
   <link href="(919) 731-3200"><text>yes</text><type>0</type></link>
   <sym>1</sym>
   <type>1</type>
   <extensions>
   </extensions>
</wpt>
<wpt lat="183568.906400562496856" lon="704563.187861500075087">
   <ele>2</ele>
   <magvar>0</magvar>
   <geoidheight>0</geoidheight>
   <name>2</name>
   <cmt>2</cmt>
   <desc>Wayne Memorial Hospital</desc>
   <src>2700 Wayne Memorial Dr</src>
   <link href="Goldsboro"><text>27534</text><type>Wayne</type></link>
   <link href="(919) 736-1110"><text>yes</text><type>0</type></link>
   <sym>1</sym>
   <type>1</type>
   <extensions>
   </extensions>
</wpt>
<wpt lat="274897.219400562578812" lon="371094.156861500116065">
   <ele>3</ele>
   <magvar>0</magvar>
   <geoidheight>0</geoidheight>
   <name>3</name>
   <cmt>3</cmt>
   <desc>Watauga Medical Center</desc>
   <src>336 Deerfield Rd</src>
   <link href="Boone"><text>28607</text><type>Watauga</type></link>
   <link href="(828) 262-4100"><text>yes</text><type>0</type></link>
   <sym>1</sym>
   <type>1</type>
   <extensions>
   </extensions>
</wpt>
...
}}}

the "cat" field will always be present regardless of if the map has an
attached attribute DB or not. (i.e. a key INTEGER db field will always be
present, and by default this is named "cat")

AFAICT if there is no attribute DB attached to the map the key field (eg
"cat") should be used for <name> in the GPX file.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

----
rouault wrote:

Hamish,

I'm pretty sure that it's a GRASS issue and not a OGR one.

Disclaimer, I've not read GRASS source code, but the only way to explain
what happens is that GRASS proceeds that way :

    * first, it uses poLayer->CreateField(poField) where poField in your
example is something like ("cat", String), then ("OBJECTID", String),
etc... But for the GPX driver, when you add the first use field, namely
"cat", it's not put at index 0 of the field definition array, but at field
10 or so, after all the compulsory GPX fields for the <wpt> tag

    * then, GRASS uses poFeature->SetField(0,
"the_value_of_the_cat_record_on_that_feature"), which is wrong because
index 0 is in fact the <ele> field. Instead of using index 0, it should
use poFeature->GetFieldIndex("cat") to retrieve the good index of the cat
field when it is set in the GPX layer object.

I hope I'm clear and that will make sense for the GRASS developers.

----

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

yes, that would seem to be the case: "j" is used as Field, it assumes OGR
fields use the same index as grass column order:

https://trac.osgeo.org/grass/browser/grass/trunk/vector/v.out.ogr/main.c#L705

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Comment (by hamish):

patch to v.out.ogr added to ticket. It populates OGR table using correct
OGR field index, not GRASS's DB column order.

todo:
  - clear data columns (need to open Table/cursor or move that
functionality into top of population loop.
  - verify the removal of keyval doesn't hurt anything (e.g. case
insensitive column names)
  - test, test, test (breaking shapefile output would be bad..)

please test,
Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

#354: v.out.ogr: bad DB read for format=GPX
---------------------+------------------------------------------------------
  Reporter: hamish | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: closed
  Priority: major | Milestone: 6.4.0
Component: Vector | Version: svn-develbranch6
Resolution: fixed | Keywords: v.out.ogr GPX
  Platform: Linux | Cpu: x86-32
---------------------+------------------------------------------------------
Changes (by hamish):

  * status: new => closed
  * resolution: => fixed

Comment:

fix applied in SVN in r34356, r34357.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/354#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>