[GRASS-dev] [QGIS Application - Bug report #19733] v.rast.stats is broken due to parsing error of double underscore in column names

GRASS GRASS developer
,
1

Hi devs,

does anyone have an idea how to fix this issue in QGIS processing?

Thanks
Markus

---------- Forwarded message ---------
From: <redmine@qgis.org>
Date: Fr., 31. Aug. 2018, 22:38
Subject: [QGIS Application - Bug report #19733] v.rast.stats is broken due to parsing error of double underscore in column names
To:

Issue #19733 has been updated by Markus Neteler.

I suspect that the underscore parsing error happens in

plugins/processing/algs/grass7/Grass7Algorithm.py

around line 553.

Bug report #19733: v.rast.stats is broken due to parsing error of double underscore in column names
http://issues.qgis.org/issues/19733#change-92384

  • Author: Markus Neteler
  • Status: Open
  • Priority: Normal
  • Assignee:
  • Category: Processing/GRASS
  • Affected QGIS version: 3.2.2
  • Operating System: All
  • Pull Request or Patch supplied: No
  • Crashes QGIS or corrupts data: No
  • Regression?: No
  • Easy fix?: No
  • Resolution:

Processing > GRASS 7 > v.rast.stats is not functional (reported also in several forums [1]). The reason is apparently a parse error since column names have multiple “_”. When reducing the statistical measures to those without underscore (e.g.“sum”) it works. When having “percentile” is doesn’t since the percentile value is transferred via underscore, e.g. “percentile=95” becomes “percentile_95” which is not parsed properly in QGIS processing.

By default all measure are active, i.e.
number,minimum,maximum,range,average,stddev,variance,coeff_var,sum,first_quartile,median,third_quartile,percentile

Issue is in both QGIS 2.18.x as well as 3.2.x.

Log:

Algorithmus v.rast.stats - Calculates univariate statistics from a raster map based on vector polygons and uploads statistics to new attribute columns. startet…
g.proj -c proj4=“+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs”
v.in.ogr min_area=0.0001 snap=-1 input=“/home/user/geodata_dav/mundialis_daten/projekte/who_geodata/data/Districts” layer=Districts output=tmp15356101025715 --overwrite -o
r.external input=“/home/user/ES5_TemperatureCoolingModul/actualT_dailymean.tif” band=1 output=tmp15356101025716 --overwrite -o
g.region n=322441.565 s=312777.529 e=180172.152 w=172817.428 res=10.0064272109
v.rast.stats -c map=“tmp15356101025715” raster=“tmp15356101025716” column_prefix=“s” method=“third_quartile,percentile” percentile=“2” --overwrite
v.out.ogr --overwrite -s -e input=tmp15356101025715 output=“/tmp/processing5113522d9c174e7bb445f24ef38c5f5a/135e78c4ae1d430da568b502506b4ab0” format=ESRI_Shapefile output_layer=output
[…]
Preprocessing input data…
Processing input data (7 categories)…
0…3…6…9…12…15…18…21…24…27…30…33…36…39…42…45…48…51…54…57…60…63…66…69…72…75…78…81…84…87…90…93…96…99…100
Updating the database …
ERROR 6: Failed to add field named ‘s_third_quartile’
ERROR 6: Failed to add field named ‘s_percentile_2’
Exporting 13 areas (may take some time)…
7…15…23…30…38…46…53…61…69…76…84…92…100

Assumed solution: fix the parsing of the column names (allow for multiple underscores).

[1] Reported without solution here:

–
You have received this notification because you have either subscribed to it, or are involved in it.
To change your notification preferences, please click here: http://issues.qgis.org/my/account

2

On Sat, Sep 1, 2018 at 7:58 AM Markus Neteler <neteler@osgeo.org> wrote:

Hi devs,

does anyone have an idea how to fix this issue in QGIS processing?

this is a GDAL error message also happening in GRASS directly:

[…]

v.out.ogr --overwrite -s -e input=tmp15356101025715 output=“/tmp/processing5113522d9c174e7bb445f24ef38c5f5a/135e78c4ae1d430da568b502506b4ab0” format=ESRI_Shapefile output_layer=output
[…]

ERROR 6: Failed to add field named ‘s_third_quartile’
ERROR 6: Failed to add field named ‘s_percentile_2’

the underscores are not the problem because exporting columns like ‘s_t_q’ and ‘s_p_2’ works. The field names are too long for dbf (max 10 characters). The solution would be to allow that “the field may be created in a slightly different form depending on the limitations of the format driver” [0] in v.out.ogr. The ERROR 6 would then change to e.g.

Warning 6: Normalized/laundered field name: ‘s_third_quartile’ to ‘s_third_qu’

Markus M

[0] https://gdal.org/ogr__api_8h.html#aab585ef1166c61c4819f7fd46ee4a275

3

On Sat, Sep 1, 2018 at 6:07 PM Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Sat, Sep 1, 2018 at 7:58 AM Markus Neteler <neteler@osgeo.org> wrote:

Hi devs,

does anyone have an idea how to fix this issue in QGIS processing?

this is a GDAL error message also happening in GRASS directly:

[…]

v.out.ogr --overwrite -s -e input=tmp15356101025715 output=“/tmp/processing5113522d9c174e7bb445f24ef38c5f5a/135e78c4ae1d430da568b502506b4ab0” format=ESRI_Shapefile output_layer=output
[…]
ERROR 6: Failed to add field named ‘s_third_quartile’
ERROR 6: Failed to add field named ‘s_percentile_2’

the underscores are not the problem because exporting columns like ‘s_t_q’ and ‘s_p_2’ works. The field names are too long for dbf (max 10 characters). The solution would be to allow that “the field may be created in a slightly different form depending on the limitations of the format driver” [0] in v.out.ogr. The ERROR 6 would then change to e.g.

Warning 6: Normalized/laundered field name: ‘s_third_quartile’ to ‘s_third_qu’

another solution would be that QGIS uses GeoPackage instead of ESRI Shapefile as vector data exchange format

Markus M

[0] https://gdal.org/ogr__api_8h.html#aab585ef1166c61c4819f7fd46ee4a275

4

On Sat, Sep 1, 2018 at 7:14 PM Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Sat, Sep 1, 2018 at 6:07 PM Markus Metz <markus.metz.giswork@gmail.com> wrote:

...

> this is a GDAL error message also happening in GRASS directly:
>
> [...]
> > v.out.ogr --overwrite -s -e input=tmp15356101025715 output="/tmp/processing5113522d9c174e7bb445f24ef38c5f5a/135e78c4ae1d430da568b502506b4ab0" format=ESRI_Shapefile output_layer=output
> > [...]
> > ERROR 6: Failed to add field named 's_third_quartile'
> > ERROR 6: Failed to add field named 's_percentile_2'
>
> the underscores are not the problem because exporting columns like 's_t_q' and 's_p_2' works. The field names are too long for dbf (max 10 characters).

Oh, in fact I didn't try with SHP file format (meanwhile I tend to
forget about it :slight_smile:

> The solution would be to allow that "the field may be created in a slightly different form depending on the limitations of the format driver" [0] in v.out.ogr. The ERROR 6 would then change to e.g.
>
> Warning 6: Normalized/laundered field name: 's_third_quartile' to 's_third_qu'

another solution would be that QGIS uses GeoPackage instead of ESRI Shapefile as vector data exchange format

I have proposed the latter in the now updated QGIS ticket:
https://issues.qgis.org/issues/19733

Thanks for your analysis!

markusN