[GRASS-user] WinGRASS not recognizing accented utf-8 (nor cp1252) attribute values

Hi,

I've got shape files with Swedish accented letters (ÄÖÅ) in the some of the attribute
values. The Attributes are shwon as they should in the GUI. SQL statements, however,
are not recognizing them. They're also messed up in the command output if another
(not accented) values are queried.

I sat GRASS_DB_ENCODING to cp1252 firstly and it didn't work. Then I converted the
dbf file into utf-8 and sat it as the value of the variable, to no avail. I also
tried using the 'encoding' parameter in v.in.ogr in both cases, didn't work.

I tried it on windows 8.1 and windows 10. The same is happening in both, stable GRASS
7.0.5 and GRASS 7.2.0RC1.

The problem is only happening on Windows. Fedora and Mac OsX don't have this issue
with the same shape files.

Any suggestion?

Salim

Salim Razzaz wrote

Hi,

I've got shape files with Swedish accented letters (ÄÖÅ) in the some of
the attribute
values. The Attributes are shwon as they should in the GUI. SQL
statements, however,
are not recognizing them. They're also messed up in the command output if
another
(not accented) values are queried.

I sat GRASS_DB_ENCODING to cp1252 firstly and it didn't work. Then I
converted the
dbf file into utf-8 and sat it as the value of the variable, to no avail.
I also
tried using the 'encoding' parameter in v.in.ogr in both cases, didn't
work.

I tried it on windows 8.1 and windows 10. The same is happening in both,
stable GRASS
7.0.5 and GRASS 7.2.0RC1.

The problem is only happening on Windows. Fedora and Mac OsX don't have
this issue
with the same shape files.

Any suggestion?

Salim

confirmed with

GRASS version: 7.3.svn
GRASS SVN revision: r70001
Build date: 2016-12-06
Build platform: x86_64-w64-mingw32
GDAL: 2.1.2
PROJ.4: 4.9.3
GEOS: 3.5.0
SQLite: 3.14.1
Python: 2.7.5
wxPython: 2.8.12.1
Platform: Windows-8-6.2.9200 (OSGeo4W)

and a test vector with following attributes

v.db.select map=test_points@data file=D:\temp\test_point.txt

cat|id|names
1|1|ÄÖÅ
2||Æ
3||Ø
4||Å,å,Æ,æ,Ø,ø
5||ø, Ø
6||Þ
7||Ð
8||Å
9||æ

e.g.

d.vect map=test_points2@data where="names = 'Å,å,Æ,æ,Ø,ø'" width=1
icon=basic/point size=10

doesn't show the selected point in the map display.

e.g.

v.report map=test_points@data option=coor
cat|id|names|x|y|z
1|1|ÄÖÅ|1.37409120951759|47.039352838731|0.0
2||Æ|2.62326503635168|28.5515802015863|0.0
3||Ø|44.095836087244|57.2825782187707|0.0
4||Å,å,Æ,æ,Ø,ø|30.8545935228025|49.787535257766|0.0
5||ø, Ø|10.1183079973563|51.0367090846001|0.0
6||Þ|20.361533377396|52.0360481460674|0.0
8||Ã…|15.1491119517375|60.3621017805262|0.0
9||æ|-1.26290587954035|52.5879880709736|0.0

Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\goutput.py", line 473, in OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\goutput.py", line 772, in AddStyledMessage

self.AddTextWrapped(message, wrap=None)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\goutput.py", line 721, in AddTextWrapped

txt = EncodeString(txt)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\co
re\gcmd.py", line 97, in EncodeString

return string.encode(_enc)
  File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 3: ordinal
not in range(128)

please open a ticket in trac (https://trac.osgeo.org/grass) with some
example data and an example workflow which can the devs test.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/WinGRASS-not-recognizing-accented-utf-8-nor-cp1252-attribute-values-tp5298708p5298806.html
Sent from the Grass - Users mailing list archive at Nabble.com.

Salim Razzaz wrote

Hi,

I've got shape files with Swedish accented letters (ÄÖÅ) in the some of
the attribute
values. The Attributes are shwon as they should in the GUI. SQL
statements, however,
are not recognizing them. They're also messed up in the command output if
another
(not accented) values are queried.

I sat GRASS_DB_ENCODING to cp1252 firstly and it didn't work. Then I
converted the
dbf file into utf-8 and sat it as the value of the variable, to no avail.
I also
tried using the 'encoding' parameter in v.in.ogr in both cases, didn't
work.

I tried it on windows 8.1 and windows 10. The same is happening in both,
stable GRASS
7.0.5 and GRASS 7.2.0RC1.

The problem is only happening on Windows. Fedora and Mac OsX don't have
this issue
with the same shape files.

Any suggestion?

for the record, just opened a ticket for this:

https://trac.osgeo.org/grass/ticket/3220

please add further information here if needed.

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/WinGRASS-not-recognizing-accented-utf-8-nor-cp1252-attribute-values-tp5298708p5299027.html
Sent from the Grass - Users mailing list archive at Nabble.com.

Hi,

2016-12-05 22:12 GMT+01:00 Salim Al Razzaz <salim.razzaz@gmail.com>:

dbf file into utf-8 and sat it as the value of the variable, to no avail. I also
tried using the 'encoding' parameter in v.in.ogr in both cases, didn't work.

strange, it should help. Can you share with us a sample? Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

Martin Landa wrote

Hi,

2016-12-05 22:12 GMT+01:00 Salim Al Razzaz &lt;

salim.razzaz@

&gt;:

dbf file into utf-8 and sat it as the value of the variable, to no avail.
I also
tried using the 'encoding' parameter in v.in.ogr in both cases, didn't
work.

strange, it should help. Can you share with us a sample? Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa
_______________________________________________
grass-user mailing list

grass-user@.osgeo

http://lists.osgeo.org/mailman/listinfo/grass-user

I've added some test data here:

https://trac.osgeo.org/grass/ticket/3220

-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/WinGRASS-not-recognizing-accented-utf-8-nor-cp1252-attribute-values-tp5298708p5299059.html
Sent from the Grass - Users mailing list archive at Nabble.com.