[GRASS-dev] [GRASS GIS] #1941: wxGUI error

GRASS GIS GRASS GIS developer
1

#1941: wxGUI error
---------------------+------------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: MSWindows 7
      Cpu: x86-32 |
---------------------+------------------------------------------------------
I get following error with the wxGUI while running
GRASS7-svn when selecting modules form the menu.
on Windows-7 in Japanese locale.
I guess it is related to utf-8 encoding.

------------------------------------------------
Traceback (most recent call last):
   File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 737, in
OnMenuCmd

cmd = self.GetMenuCmd(event)
   File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 722, in
GetMenuCmd

input = GUI().GetCommandInputMapParamKey(cmdlist[0])
   File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\gui_core\forms.py", line 2301, in
GetCommandInputMapParamKey

gtask.get_interface_description(cmd).decode(enc).encode('utf
-8')))
UnicodeDecodeError
:
'cp932' codec can't decode bytes in position 7074-7075:
illegal multibyte sequence

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941&gt;
GRASS GIS <http://grass.osgeo.org>

2

#1941: wxGUI fails with Japanese locale
---------------------+------------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: MSWindows 7
      Cpu: x86-32 |
---------------------+------------------------------------------------------

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

3

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------
Changes (by neteler):

  * keywords: => wingrass

Comment:

Two tickets related to cp932:

  * http://trac.osgeo.org/osgeo4w/ticket/294#comment:4
  * http://trac.osgeo.org/grass/ticket/1193#comment:2

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

4

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by venkat):

This issue seems to have been fixed in GRASS-6.4 but I think still remains
in GRASS-7.0.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

5

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by mlennert):

Could this be related to #2021 ?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

6

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by glynn):

Replying to [comment:4 mlennert]:
> Could this be related to #2021 ?

That seems unlikely, as this relates to Windows which doesn't use
environment variables for locale settings.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

7

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by improgrammer):

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to
return xml in "utf-8" encoding. see attachment
"patch_for_ticket_1941.patch".

Rational:
  1. The following code segment in task.py and forms.py is too limited
because ("cp936","cp932", etc) need similar converting:
{{{
     enc = locale.getdefaultlocale()[1]
     if enc and enc.lower() == "cp932":
         p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
         tree = etree.fromstring(p.sub('encoding="utf-8"',
get_interface_description(name).decode(enc).encode("utf-8")))
     else:
         tree = etree.fromstring(get_interface_description(name))
}}}
  1. Any where the gtask.get_interface_description() is called, the
"convert-to-utf8" is needed;
  1. Properly constructed interface description text is xml text with
proper <?xml ... encoding="some-text-encoding"?> statement, extract text
encoding schema from these xml text is more rational than extract enc from
locale package.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

8

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by mlennert):

Replying to [comment:6 improgrammer]:
> I encounter similar problem.
>
> I suggest modifying the sementics of gtask.get_interface_description()
to return xml in "utf-8" encoding. see attachment
"patch_for_ticket_1941.patch".
>
> Rational:
> 1. The following code segment in task.py and forms.py is too limited
because ("cp936","cp932", etc) need similar converting:
> {{{
> enc = locale.getdefaultlocale()[1]
> if enc and enc.lower() == "cp932":
> p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
> tree = etree.fromstring(p.sub('encoding="utf-8"',
>
get_interface_description(name).decode(enc).encode("utf-8")))
> else:
> tree = etree.fromstring(get_interface_description(name))
> }}}
> 1. Any where the gtask.get_interface_description() is called, the
"convert-to-utf8" is needed;
> 1. Properly constructed interface description text is xml text with
proper <?xml ... encoding="some-text-encoding"?> statement, extract text
encoding schema from these xml text is more rational than extract enc from
locale package.

Can someone else working in a Japanese Windows environment test this patch
?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

9

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by neteler):

Replying to [comment:7 mlennert]:
...
> Can someone else working in a Japanese Windows environment test this
patch ?

Proposal: Since it so much easier to update winGRASS via installer, I
would suggest that we submit the patch to SVN and test from there. In
the worst case we revert after 24hs/whatever time.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

10

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by annakrat):

Replying to [comment:8 neteler]:
> Replying to [comment:7 mlennert]:
> ...
> > Can someone else working in a Japanese Windows environment test this
patch ?
>
> Proposal: Since it so much easier to update winGRASS via installer, I
> would suggest that we submit the patch to SVN and test from there. In
> the worst case we revert after 24hs/whatever time.

Done in r57501. Please test!

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

11

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by neteler):

Replying to [comment:9 annakrat]:
> > Replying to [comment:7 mlennert]:
> > ...
> > > Can someone else working in a Japanese Windows environment test this
patch ?
> Done in r57501. Please test!

The related winGRASS 7 binary is now available here:
http://wingrass.fsv.cvut.cz/grass70/

as well as in OSGEo4W.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

12

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by venkat):

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe
from http://wingrass.fsv.cvut.cz/grass70/ but this
gives me

ImportError: No module named gis when I try to start the
wxGUI. Details in the attachment.

Venka

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

13

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by annakrat):

Replying to [comment:11 venkat]:
> tried with WinGRASS-7.0.svn-r57518-685-Setup.exe
> from http://wingrass.fsv.cvut.cz/grass70/ but this
> gives me
>
> ImportError: No module named gis when I try to start the
> wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis
library (which apparently uses ctypes) so that wxGUI should at least
start.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1941#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

14

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by mlennert):

Replying to [comment:12 annakrat]:
> Replying to [comment:11 venkat]:
> > tried with WinGRASS-7.0.svn-r57518-685-Setup.exe
> > from http://wingrass.fsv.cvut.cz/grass70/ but this
> > gives me
> >
> > ImportError: No module named gis when I try to start the
> > wxGUI. Details in the attachment.
>
> You seem to have a problem with ctypes. I moved importing temporal gis
library (which apparently uses ctypes) so that wxGUI should at least
start.

Venka,

Have you been able to test these last changes ?

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

15

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by venkat):

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and
get errors as shown in the attachment.

Thanks

Venka

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

16

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by annakrat):

Replying to [comment:14 venkat]:
> Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and
> get errors as shown in the attachment.

Please try the OSGeo4W installer, you shouldn't get this error there.

Anybody has an idea how to fix this annoying MAXREPEAT problem?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>

17

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by neteler):

Replying to [comment:15 annakrat]:
> Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:16&gt;
GRASS GIS <http://grass.osgeo.org>

18

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by martinl):

Replying to [comment:16 neteler]:
> Replying to [comment:15 annakrat]:
> > Anybody has an idea how to fix this annoying MAXREPEAT problem?
>
> Did you try this?
>
>
http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

Usually it's caused by Python installed together with Esri ArcGIS. The
newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be
good to find solution on our side without overwriting system-installed
Python.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:17&gt;
GRASS GIS <http://grass.osgeo.org>

19

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by glynn):

Replying to [comment:17 martinl]:

> Usually it's caused by Python installed together with Esri ArcGIS. The
newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be
good to find solution on our side without overwriting system-installed
Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect
that the "system" Python is sufficient and not try to use the bundled
Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system
version, etc).

AFAICT, the issue is that the system python27.dll (from `Windows/System32`
or `Windows/SysWOW64`) is being used, but PYTHONHOME points to the bundled
version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use
[http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx Process
Explorer] to view the list of DLLs which are actually being used by that
process. AFAIK, System32/SysWOW64 take precedence over everything except
the directory containing the exe, and %PATH% comes last.

[http://msdn.microsoft.com/en-
us/library/windows/desktop/ms682586%28v=vs.85%29.aspx DLL Search Order].

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:18&gt;
GRASS GIS <http://grass.osgeo.org>

20

#1941: wxGUI fails with Japanese locale
----------------------+-----------------------------------------------------
Reporter: venkat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: wingrass | Platform: MSWindows 7
      Cpu: x86-32 |
----------------------+-----------------------------------------------------

Comment(by martinl):

Replying to [comment:18 glynn]:
> Replying to [comment:17 martinl]:
>
> > Usually it's caused by Python installed together with Esri ArcGIS. The
newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be
good to find solution on our side without overwriting system-installed
Python.
>
> 2.7.3 should be fine for GRASS, right? So the solution should be to
detect that the

No, the problem still remains. Note that we have this problem only with
standalone installer not with osgeo4w installer.

> "system" Python is sufficient and not try to use the bundled Python
(i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system
version, etc).

But then we will have problem with other python packages (GRASS
dependecies) which are installed through osgeo4w environment as `python-
matplotlib` or `python-numpy`, right? Version of "system" python will be
most probably different from the python version available in osgeo4w
environment.

> AFAICT, the issue is that the system python27.dll (from
`Windows/System32` or `Windows/SysWOW64`) is being used, but PYTHONHOME
points to the bundled version of the Python standard library.
>
> If you can start a Python shell using the GRASS Python, you can use
[http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx Process
Explorer] to view the list of DLLs which are actually being used by that
process. AFAIK, System32/SysWOW64 take precedence over everything except
the directory containing the exe, and %PATH% comes last.

Strangely `C:\Program Files (x86)\GRASS GIS 7.0.svn\extrabin\python.exe`
launches python 2.7.3 (same as system python from `C:\Python27\ArcGIS
10.2`)! When launching `C:\osgeo4w\bin\python.exe` I got expected version
of python, so 2.7.4 (current osgeo4w python version). This partly explains
the fact that we have this problem only with standalone installer.

> [http://msdn.microsoft.com/en-
us/library/windows/desktop/ms682586%28v=vs.85%29.aspx DLL Search Order].

Right, when I launch `python.exe` which comes with standalone installer it
points to `C:Windows\SysWOW64\python27.dll`. When I launch `python.exe`
from `C:\OSGeo4W\bin` it points to the right dll file, ie.
`C:\OSGeo4W\bin\python27.dll`.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1941#comment:19&gt;
GRASS GIS <http://grass.osgeo.org>