[GRASS-dev] [GRASS GIS] #3405: g.search.modules: 'ascii' codec can't encode character u'\xe0' in position 11: ordinal not in range(128)

#3405: g.search.modules: 'ascii' codec can't encode character u'\xe0' in position
11: ordinal not in range(128)
---------------------------------------+-------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.2.2
Component: wxGUI | Version: svn-trunk
Keywords: g.search.modules encoding | CPU: Unspecified
Platform: Unspecified |
---------------------------------------+-------------------------
And another encoding issue:

In the GUI, going to the "Modules" tab and clicking on "Advanced search",
the g.search.modules module window opens, but I get the following error
message:

{{{
'ascii' codec can't encode character u'\xe0' in position 11: ordinal not
in range(128)
}}}

When I then click 'OK' I can access the g.search.modules window and
execute the command without any further issue.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Apparently this is a larger issue:

{{{
i.segment --ui
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
     task.get_cmd(ignoreErrors=True, ignoreRequired=True))
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 248, in get_cmd
     errList = self.get_cmd_error()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
     {'name': p['name'], 'desc': encode(desc)})
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/utils.py", line 217, in encode
     return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
position 29: ordinal not in range(128)
}}}

{{{
v.lrs.create --ui
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
     task.get_cmd(ignoreErrors=True, ignoreRequired=True))
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 248, in get_cmd
     errList = self.get_cmd_error()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
     {'name': p['name'], 'desc': encode(desc)})
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/utils.py", line 217, in encode
     return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
position 36: ordinal not in range(128)
}}}

{{{
r.surf.idw --ui
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
     task.get_cmd(ignoreErrors=True, ignoreRequired=True))
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 248, in get_cmd
     errList = self.get_cmd_error()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
     {'name': p['name'], 'desc': encode(desc)})
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/utils.py", line 217, in encode
     return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
position 35: ordinal not in range(128)
}}}

Interestingly, for all these modules no GUI window is created
automatically if they are launched without '--ui' from the command line. I
have the feeling this is new.

When I launch the modules from the GUI menu, I get a window with the error
message, but clicking 'OK' lets me use the module normally.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------
Changes (by mlennert):

* milestone: 7.2.2 => 7.4.0

Comment:

This is not an issue in release72, so changing milestone.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

I just tested on my office computer with a fresh svn checkout, and cannot
reproduce. It must be something on my home computer... I'll try again
tonight, but maybe this is just noise...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

I'm completely stumped on this one: my office and home machine both run
Debian testing. I recompiled a fresh svn trunk checkout from scratch
(r71433). I moved the .grass7 directories to avoid any cruft. locale
settings are equal between the two. In neither do I have special
characters anywhere in the gisenv paths.

Result: on the home machine I get the above errors, on the office machine
I don't.

???

Setting LANGUAGE=C allows the module GUI to launch without an error on my
home machine. The only difference I see is that on the office machine
LANGUAGE=C and the launching the module does not give any messages in the
terminal, while on the home machine, the same combination gives me:

{{{
(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",
}}}

Could that be the issue ?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by neteler):

Replying to [comment:4 mlennert]:
> {{{
> (forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",
>
> (forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",
> }}}

Probably you are lacking the adwaita-gtk2-theme package? Here the resp.
Fedora RPM:

{{{
rpm -qil adwaita-gtk2-theme
Name : adwaita-gtk2-theme
Version : 3.22.3
Release : 2.fc26
...
URL : http://git.gnome.org/browse/gnome-themes-standard
Summary : Adwaita gtk2 theme
Description : The adwaita-gtk2-theme package contains a gtk2 theme for
presenting widgets with a GNOME look and feel.
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Replying to [comment:5 neteler]:
> Replying to [comment:4 mlennert]:
> > {{{
> > (forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",
> >
> > (forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in
module_path: "adwaita",
> > }}}
>
> Probably you are lacking the adwaita-gtk2-theme package? Here the resp.
Fedora RPM:

Thanks for the hint. In Debian, it's the gnome-themes-standard package.

Installing that gets rid of the Gtk-WARNINGs, but not of the bug trying to
run the modules.

But I'm getting closer. In lib/python/script/utils.py we have the function
encode, with the following code:

{{{
     if isinstance(string, bytes):
         return string
     if isinstance(string, unicode):
         enc = _get_encoding()
         return string.encode(enc)
     return bytes(string)
}}}

On the work computer the string gets recognized as unicode and therefore
the code in the second if is run. On the home computer, the string is not
recognized as unicode and the last line is run and causes the error.

However, if I modify the code as such:

{{{
     print type(string)
     if isinstance(string, bytes):
         print "isinstance bytes"
         return string
     if isinstance(string, unicode):
         print "isinstance unicode"
         enc = _get_encoding()
         return string.encode(enc)
     print "isinstance none"
     return bytes(string)
}}}

the output I get is

{{{
> v.to.rast --ui
<type 'str'>
isinstance bytes
<type 'str'>
isinstance bytes
<type 'unicode'> <==================
isinstance none <==================
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
     task.get_cmd(ignoreErrors=True, ignoreRequired=True))
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 248, in get_cmd
     errList = self.get_cmd_error()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
     {'name': p['name'], 'desc': encode(desc)})
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/utils.py", line 221, in encode
     return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
position 36: ordinal not in range(128)
}}}

In other words, it is recognized as type unicode, but isinstance(string,
unicode) returns false...

Actually, when type is 'str' it is recognized as unicode:

{{{
     print type(string)
     print isinstance(string, unicode)
}}}

gives

{{{
<type 'str'>
True
<type 'str'>
True
<type 'unicode'>
False
}}}

Any hints explaining why this is happening on one machine, but not the
other ? Python versions are identical (Python 2.7.13).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Replying to [comment:6 mlennert]:
> Actually, when type is 'str' it is recognized as unicode:
>
>
> {{{
> print type(string)
> print isinstance(string, unicode)
> }}}
>
> gives
>
> {{{
> <type 'str'>
> True
> <type 'str'>
> True
> <type 'unicode'>
> False
> }}}
>
>

This is not the case on my office machine, where I get:

{{{
<type 'str'>
False
<type 'str'>
False
<type 'unicode'>
True
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Bingo ! I found the problem. At the top of lib/python/script/utils.py,
there is this:

{{{
try:
     from builtins import unicode
except ImportError:
     # python3
     unicode = str
}}}

However, in order to be able to import builtins I have to install the
package python-future which was not the case on my home computer. Does
this mean we assume this to be a dependence ?

This said, if I just comment out this part of the code, I do not get the
errors either, as apparently the decode function works as is with the
builtin unicode type in python 2.

Anna, could you enlighten me why this import from future is necessary ?
And if it is, we will have to make this an official dependence...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Replying to [comment:8 mlennert]:
> Bingo ! I found the problem. At the top of lib/python/script/utils.py,
there is this:
>
> {{{
> try:
> from builtins import unicode
> except ImportError:
> # python3
> unicode = str
> }}}
>
> However, in order to be able to import builtins I have to install the
package python-future which was not the case on my home computer. Does
this mean we assume this to be a dependence ?
>
> This said, if I just comment out this part of the code, I do not get the
errors either, as apparently the decode function works as is with the
builtin unicode type in python 2.
>
> Anna, could you enlighten me why this import from future is necessary ?
And if it is, we will have to make this an official dependence...

Sorry, wrong adressee: I just saw that Pietro introduced this just
recently. Will contact him.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Apparently fixed in r71435. I tested by removing the python-future package
from my office computer and I get the GUI as expected. I'll test on my
home computer ASAP before closing.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by neteler):

G7.2.svn tested ok on Linux (Fedora 26) with

{{{
  export LANG=de_DE.UTF-8
  export LANGUAGE=de_DE.UTF-8
  export LC_MESSAGES=de_DE.UTF-8
}}}

and

{{{
  export LANG=fr_FR.UTF-8
  export LANGUAGE=fr_FR.UTF-8
  export LC_MESSAGES=fr_FR.UTF-8
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:11&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------

Comment (by mlennert):

Replying to [comment:11 neteler]:
> G7.2.svn tested ok on Linux (Fedora 26) with
>
> {{{
> export LANG=de_DE.UTF-8
> export LANGUAGE=de_DE.UTF-8
> export LC_MESSAGES=de_DE.UTF-8
> }}}
>
> and
>
> {{{
> export LANG=fr_FR.UTF-8
> export LANGUAGE=fr_FR.UTF-8
> export LC_MESSAGES=fr_FR.UTF-8
> }}}

I think we can release 7.2.2RC2 now. With a big fat call for testing on
the lists...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:12&gt;
GRASS GIS <https://grass.osgeo.org>

#3405: encoding issue when launching modules
--------------------------+---------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.4.0
Component: wxGUI | Version: svn-trunk
Resolution: fixed | Keywords: g.search.modules encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------------
Changes (by mlennert):

* status: new => closed
* resolution: => fixed

Comment:

Closing as fixed in trunk and 7.2.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3405#comment:13&gt;
GRASS GIS <https://grass.osgeo.org>