[GRASS-dev] [GRASS GIS] #3423: UnicodeDecodeError in r.category in wxGUI when category labels contain special characters

GDAL · October 9, 2017, 2:42pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
---------------------------------+-------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Keywords: r.category encoding | CPU: Unspecified
Platform: Unspecified |
---------------------------------+-------------------------
I have a map with the following categories:

{{{
> r.category liege_ucl
0
10 eau
15 sol nu
20 bâti
21 route
30 terre labourée
35 herbage non naturel
40 zone ouverte (semi)naturelle 1
45 zone ouverte (semi)naturelle 2
48 zone ouverte (semi)naturelle 3
50 résineux
55 feuillus
}}}

In the GUI, r.category gives:

{{{
r.category map=liege_ucl@RW
0
10 eau
15 sol nu
21 route
35 herbage non naturel
40 zone ouverte (semi)naturelle 1
45 zone ouverte (semi)naturelle 2
48 zone ouverte (semi)naturelle 3
55 feuillus
}}}

i.e. all categories with special characters in the labels are just
ignored.

The traceback in the Console window:

{{{
Traceback (most recent call last):
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 4: ordinal
not in range(128)
Traceback (most recent call last):
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 15: ordinal
not in range(128)
Traceback (most recent call last):
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 4: ordinal
not in range(128)
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 3:18am

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by annakrat):

Hmm, I can't reproduce it on linux. Could you test if it is still the
problem?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:1>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 12:46pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by marisn):

Works for me from CLI, GUI and Layer Manager Console tab:
{{{
r.category map=labels
1 arumi
2 vagas
3 blīvs tīrums
4 skrajš tīrums
}}}

GRASS versija: 7.3.svn\\
GRASS SVN revīzija: r71685M\\
Būvējuma datums: 2017-11-12\\
Būvēšanas platforma: x86_64-pc-linux-gnu\\
wxPython: 3.0.2.0\\

LANG=lv_LV\\
LC_CTYPE=lv_LV.UTF-8\\
LC_NUMERIC=C\\

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:2>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 4:23pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:1 annakrat]:
> Hmm, I can't reproduce it on linux. Could you test if it is still the
problem?

I have just checked. There is no issue if the GUI is defined in the
system's locale. However, choosing English as language also resets all the
locale settings and so when the GUI is in English, any accented characters
cannot be displayed.

So the question becomes: should choice of the language in which the GUI
appears influence the locale settings in this manner ? If you say yes,
than we can close this.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:3>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 4:48pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by marisn):

Replying to [comment:3 mlennert]:
> So the question becomes: should choice of the language in which the GUI
appears influence the locale settings in this manner ? If you say yes,
than we can close this.

Yes, it should. There are messages also provided by non-GUI apps or GUI
apps can be started from CLI thus bypassing wxgui language switching
mechanisms.

Here is a test case triggering a failure without switching GUI language
(NC basic; sorry for my Russian):

{{{
g.copy rast=landuse,landuse
r.category map=landuse separator=":" rules=- << EOF
1:развитая
2:сельское хозяйство
3:травянистый
4:кустарниковые
5:лесной
6:водный
7:осадок
EOF
r.category map=landuse
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:4>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 5:45pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:4 marisn]:
> Replying to [comment:3 mlennert]:
> > So the question becomes: should choice of the language in which the
GUI appears influence the locale settings in this manner ? If you say yes,
than we can close this.
>
> Yes, it should. There are messages also provided by non-GUI apps or GUI
apps can be started from CLI thus bypassing wxgui language switching
mechanisms.

So, the GUI language should influence all the rest (which it doesn't by
the way, since r.category works like a charm with special characters on
the command line even when the GUI is set to 'en', i.e. to locale=C).
Personally, I would prefer to be able to choose to show the GUI in another
language without loosing half of the attribute info...

One related question is why 'en' should lead to locale=C. We could have
the GUI in en_EN.UTF-8 or something like that.

>
> Here is a test case triggering a failure without switching GUI language
(NC basic; sorry for my Russian):

When you say 'without switching GUI language', do you mean that the GUI is
set to Russian ?

>
> {{{
> g.copy rast=landuse,landuse
> r.category map=landuse separator=":" rules=- << EOF
> 1:развитая
> 2:сельское хозяйство
> 3:травянистый
> 4:кустарниковые
> 5:лесной
> 6:водный
> 7:осадок
> EOF
> r.category map=landuse
> }}}

I do not have any failure here, with the GUI language set to 'system'
which is

{{{
locale
LANG=fr_BE
LANGUAGE=fr_BE
LC_CTYPE=fr_BE.UTF-8
LC_NUMERIC=C
LC_TIME=fr_BE.UTF-8
LC_COLLATE=fr_BE.UTF-8
LC_MONETARY=fr_BE.UTF-8
LC_MESSAGES=fr_BE.UTF-8
LC_PAPER=fr_BE.UTF-8
LC_NAME=fr_BE.UTF-8
LC_ADDRESS=fr_BE.UTF-8
LC_TELEPHONE=fr_BE.UTF-8
LC_MEASUREMENT=fr_BE.UTF-8
LC_IDENTIFICATION=fr_BE.UTF-8
LC_ALL=
}}}

However, when I switch the GUI language to 'en', I get:

{{{
r.category map=landuse@user1
(Sun Nov 12 18:44:33 2017) Command finished (0 sec)
}}}

i.e. nothing...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:5>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2017, 10:16pm

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by annakrat):

Replying to [comment:5 mlennert]:
> Replying to [comment:4 marisn]:
> > Replying to [comment:3 mlennert]:
> > > So the question becomes: should choice of the language in which the
GUI appears influence the locale settings in this manner ? If you say yes,
than we can close this.
> >
>
> One related question is why 'en' should lead to locale=C. We could have
the GUI in en_EN.UTF-8 or something like that.

So it seems the problem happens only for 'en'. I looked at it but I don't
quite understand the problem, maybe what you suggest would work...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:6>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 13, 2017, 8:56am

#3423: UnicodeDecodeError in r.category in wxGUI when category labels contain
special characters
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: worksforme | Keywords: r.category encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by marisn):

* status: new => closed
* resolution: => worksforme

Comment:

Closing this one as the bug is not in r.category but in switching to en
language (#3441).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3423#comment:7>
GRASS GIS <https://grass.osgeo.org>