[GRASS-dev] [GRASS GIS] #2120: wxgui: encoding errors

GDAL · October 25, 2013, 2:38pm

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------
Using grass64release in French (locale: fr_BE.utf8), I seem to be seeing
more encoding errors than before in the wxGUI. Examples are when running
v.in.ogr, r.in.gdal, v.build. In fact whenever a module emits a message in
the locale's language which contains special characters, I get errors such
as

{{{
'ascii' codec can't decode byte 0xef in position 16: ordinal
not in range(128)
}}}

I don't get such errors on the command line.

These errors do not keep the module from functioning, but it makes reading
its messages nearly impossible unless you really know what to look for.

I don't have the time now to go back and try to find out if this worked
better before, but I do think so. It might just be that more strings have
been translated to French and that is why I see these errors now.

I think this also adds to the recent discussions about test in that it
makes it evident that if we want to take multi-locale usage of GRASS
seriously, we probably need to think of a series of tests that developers
should use to make sure any changes do not create such encoding problems.

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120>
GRASS GIS <http://grass.osgeo.org>

GDAL · April 9, 2014, 10:09am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by marisn):

Is this still an issue in GRASS 7? There has been a huge progress related
to ironing out encoding issues in WXGUI.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2120#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · April 9, 2014, 11:59am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by mlennert):

Replying to [comment:1 marisn]:
> Is this still an issue in GRASS 7? There has been a huge progress
related to ironing out encoding issues in WXGUI.

I've been testing in the last days, and it seems that everything is
working smoothly now in GRASS7. Great job !

However, in GRASS6release I still get the same errors.

Maybe some of what was done in grass7 should be backported to grass6?

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2120#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 14, 2014, 10:53pm

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by neteler):

Replying to [comment:2 mlennert]:
> I've been testing in the last days, and it seems that everything
> is working smoothly now in GRASS7. Great job !
>
> However, in GRASS6release I still get the same errors.
> Maybe some of what was done in grass7 should be backported to grass6?

See also #1856

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 6:37am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

I see the same error message in 7 trunk when I check "Redirect to console"
in the wx monitor.
{{{
Traceback (most recent call last):
   File "/home/grass/trunk/dist.x86_64-unknown-linux-
gnu/gui/wxpython/gui_core/query.py", line 65, in <lambda>
     self.redirect.Bind(wx.EVT_CHECKBOX, lambda evt:
self._onRedirect(evt.IsChecked()))
   File "/home/grass/trunk/dist.x86_64-unknown-linux-
gnu/gui/wxpython/gui_core/query.py", line 143, in _onRedirect
     self.redirectOutput.emit(output=self._textToRedirect())
   File "/home/grass/trunk/dist.x86_64-unknown-linux-
gnu/gui/wxpython/gui_core/query.py", line 148, in _textToRedirect
     text = printResults(self._model, self._colNames[1])
   File "/home/grass/trunk/dist.x86_64-unknown-linux-
gnu/gui/wxpython/gui_core/query.py", line 213, in printResults
     return '\n'.join(textList)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xea in position 2:
ordinal not in range(128)
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 6:37am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

I mean Redirect to console in Query results.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 6:41am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

Maybe, this error message is useful:
{{{
/usr/lib64/python2.7/site-
packages/wx-2.8-gtk2-unicode/wx/lib/mixins/treemixin.py:463:
UnicodeWarning: Unicode unequal comparison failed to convert both
arguments to Unicode - interpreting them as being unequal
if getattr(self, 'Get%s'%attribute)(item, *args) != value:
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 7:34am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

OK. I solved this issue by creating sitecustomize.py. Python's default
encoding is 'ascii', so whenever it tries to compare a string (not u"...",
but "...") with a translated message, which may not be ASCII, it complains
about different encodings and throws an exception. My sitecustomize.py
file looks like
{{{
import sys
sys.setdefaultencoding("utf-8")
}}}
and I put it in gui/wxpython.

I'm not sure if it's a good idea to include this file in GRASS SVN because
it overrides and forces to use UTF-8 encoding. I don't know why the
encoding setting in Preferences doesn't work. It seems like we cannot
change the default encoding outside sitecustomize.py.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 7:37am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

FYI, Python 3's default encoding is UTF-8, so I guess it doesn't hurt to
add sitecustomize.py in GRASS SVN.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 8:06am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by hcho):

Fixed in r60307

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:9>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 18, 2014, 8:02pm

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: Unspecified
Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by glynn):

Replying to [comment:7 hcho]:

> I'm not sure if it's a good idea to include this file in GRASS SVN

It isn't.

It's working around a bug, namely that Python's default encoding is being
used. Fix the bug, not the symptoms.

> I don't know why the encoding setting in Preferences doesn't work. It
seems like we cannot change the default encoding outside sitecustomize.py.

That's correct. The reason is related to the use of hashtables to
implement dictionaries (including those which underlie Python objects).

When a Unicode object is hashed, if it can be converted to a byte string
using the default encoding then it is, and the resulting string's hash is
used. This allows a dictionary key which is a string to be matched using
the string's Unicode equivalent, and vice-versa.

However, if the default encoding is changed after any non-ASCII keys have
been added to dictionaries, any lookups for those keys would use the wrong
hash and fail. To protect against this possibility, the site module
deletes the setdefaultencoding() function from the sys module after
loading the sitecustomize and/or usercustomize modules. Basically, once
the interpreter is fully "up and running", the default encoding is fixed.

Note that Python's default encoding (as per sys.getdefaultencoding()) is
only used for implicit conversions between str and unicode objects, e.g.
via the str() and unicode() constructors, or via the C API (e.g.
`PyString_AsString`). But not for filenames (see below).

It is not the same thing as the locale's default encoding as per
locale.getpreferredencoding() or locale.getdefaultlocale(). Nor is it the
same thing as the filesystem encoding as per sys.getfilesystemencoding(),
which is used for converting filenames (from unicode to str on Unix, Mac
and Windows 9x, from str to Unicode on Windows NT). These are what
application code should be using.

So, the implicit conversions should never be used by wxGUI. Conversions
between Unicode strings (which is what wxPython normally uses) and byte
strings should be explicit, with a specified encoding (e.g. that from the
locale, unless information is available to indicate a different encoding).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:10>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 19, 2014, 2:56pm

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: major | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: All
Cpu: All |
-----------------------------+----------------------------------------------
Changes (by hcho):

  * priority: normal => major
  * platform: Unspecified => All
  * cpu: Unspecified => All

Comment:

There seems to be a real fix other than changing the default encoding.
Hopefully your explanation is helpful to the wxGUI devs for fixing the bug
in the correct way.

Meanwhile the users who are bothered by the issue, including me, can work
around it by creating usercustomize.py.

Also I would say this is major or even blocker?, not normal because some
functionalities are not working at all because of this issue. It's not
simply an annoyance.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:11>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 19, 2014, 2:57pm

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: major | Milestone: 6.4.4
Component: wxGUI | Version: svn-releasebranch64
Keywords: locale encoding | Platform: All
Cpu: All |
-----------------------------+----------------------------------------------

Comment(by hcho):

For our records, we still have this bug in 7 trunk.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:12>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 8, 2015, 10:44am

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: major | Milestone: 7.0.1
Component: wxGUI | Version: 7.0.0
Keywords: locale encoding | Platform: All
Cpu: All |
-----------------------------+----------------------------------------------
Changes (by marisn):

* version: svn-releasebranch64 => 7.0.0
* milestone: 6.4.4 => 7.0.1

Comment:

As fixing all issues would involve too invasive changes in GRASS 6.x
maintenance releases, bumping up versions to 7.

Reporters: please fill a separate bug report for each case, as it is hard
to track progress by comments alone. Add bug # here for easier tracking.
As most of issues are spotted on Windows running non-english GRASS
version, do NOT close any of them before testing with (insert your
favourite language) translated version of GRASS on Windows.

  * Raster query redirection to console issue reported as #2617
  * Raster query (no redirection) #2601
  * Save console output #2614
  * Command console fails if username is not ascii only #2390
  * Network analysis tool fails to start #2145

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2120#comment:13>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 26, 2015, 1:11pm

#2120: wxgui: encoding errors
-----------------------+-----------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.0.1
Component: wxGUI | Version: 7.0.0
Resolution: | Keywords: locale encoding
       CPU: All | Platform: All
-----------------------+-----------------------------

Comment (by mlennert):

I know that this will probably not get fixed in the grass6 branch, but in
view of the upcoming release of 6.4.5, I just wanted to note that this
problem still exists. When I create a new map with any command (e.g.
v.surf.idw, v.net, ...) and this new map is then automatically displayed,
I get the following type of error message in the gui console:

{{{
v.surf.idw --overwrite input=elev_lid792_randpts@PERMANENT output=elev
column=value

Traceback (most recent call last):
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/gui_core/goutput.py", line
759, in OnCmdOutput

self.cmdOutput.AddTextWrapped(message, wrap = None)
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/gui_core/goutput.py", line
1238, in AddTextWrapped

txt = EncodeString(txt)
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/core/gcmd.py", line 103, in
EncodeString

return string.encode(enc)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xe9 in position 22: ordinal
not in range(128)

v.net op=connect

Traceback (most recent call last):
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/gui_core/goutput.py", line
759, in OnCmdOutput

self.cmdOutput.AddTextWrapped(message, wrap = None)
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/gui_core/goutput.py", line
1238, in AddTextWrapped

txt = EncodeString(txt)
File
"/data/home/mlennert/SRC/GRASS/grass-6.4.5RC1/dist.x86_64
-unknown-linux-gnu/etc/wxpython/core/gcmd.py", line 103, in
EncodeString

return string.encode(enc)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xef in position 16: ordinal
not in range(128)
}}}

All this in a French locale setting:

{{{
> locale
LANG=fr_BE.utf8
LANGUAGE=fr_BE.utf8
LC_CTYPE="fr_BE.utf8"
LC_NUMERIC="fr_BE.utf8"
LC_TIME="fr_BE.utf8"
LC_COLLATE="fr_BE.utf8"
LC_MONETARY="fr_BE.utf8"
LC_MESSAGES="fr_BE.utf8"
LC_PAPER="fr_BE.utf8"
LC_NAME="fr_BE.utf8"
LC_ADDRESS="fr_BE.utf8"
LC_TELEPHONE="fr_BE.utf8"
LC_MEASUREMENT="fr_BE.utf8"
LC_IDENTIFICATION="fr_BE.utf8"
LC_ALL=
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:14>
GRASS GIS <http://grass.osgeo.org>