[GRASS-dev] [GRASS GIS] #1672: gcpmanager fails to list source mapsets if non-ascii characters in grass data dir path

#1672: gcpmanager fails to list source mapsets if non-ascii characters in grass
data dir path
----------------------------------------------+-----------------------------
Reporter: hamish | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: wxGUI | Version: 6.4.2
Keywords: georectifier, UnicodeDecodeError | Platform: Linux
      Cpu: x86-64 |
----------------------------------------------+-----------------------------
[Hi, one fwd'd from the Debian bug tracker, http://bugs.debian.org/676190
--H]

{{{
From: Torsti Schulz <torsti.schulz at iki dot fi>
To: Debian Bug Tracking System
Subject: grass-gui: gcpmanager fails to list source mapsets if non-ascii
characters
  in grass data dir path
Date: Tue, 05 Jun 2012 14:35:48 +0300

Package: grass-gui
Version: 6.4.2-1
Severity: normal

Hi,

Using the wxpython GUI Georectify tool is impossible if the path to the
grass
data directory (e.g. GRASSDATA) contains non ascii characters (e.g.
'/home/user/Työpöytä/GRASSDATA'). Selecting the source location works, but
the 'Select source mapset:' dropdown menu remains greyed out. The command
console shows the following error message, after selecting the source
location:

***
Traceback (most recent call last):
   File
"/usr/lib/grass64/etc/wxpython/gui_modules/gcpmanager.py",
line 377, in OnLocation

tmplist = os.listdir(os.path.join(self.grassdatabase,
self.xylocation))
   File "/usr/lib/python2.7/posixpath.py", line 71, in join

path += '/' + b
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 15: ordinal
not in range(128)
***

Changing the grass data directory path to only contain ascii characters
(e.g.
'/home/user/GRASSDATA') makes the Georectify tool usable. This seems to an
issue with how the python code handles path names and unicode characters.

Cheers,
Torsti

-- System Information:
Debian Release: wheezy/sid
   APT prefers testing
   APT policy: (700, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=fi_FI.utf8, LC_CTYPE=fi_FI.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages grass-gui depends on:
ii gnome-terminal [x-terminal-emulator] 3.4.1.1-1
ii grass-core 6.4.2-1
ii lesstif2 1:0.95.2-1
ii libc6 2.13-33
ii libfreetype6 2.4.9-1
ii libgdal1 1.9.0-2
ii libgl1-mesa-glx [libgl1] 8.0.2-2
ii libglu1-mesa [libglu1] 8.0.2-2
ii libice6 2:1.0.8-2
ii libpng12-0 1.2.49-1
ii libsm6 2:1.2.1-2
ii libtiff5 4.0.1-7
ii libx11-6 2:1.4.99.901-2
ii libxext6 2:1.3.1-2
ii libxmu6 2:1.1.1-1
ii libxt6 1:1.1.3-1
ii python 2.7.2-10
ii python-opengl 3.0.1-1
ii python-wxgtk2.8 2.8.12.1-11
ii tcl8.5 8.5.11-2
ii tk8.5 8.5.11-2
ii xterm [x-terminal-emulator] 278-1
ii zlib1g 1:1.2.7.dfsg-11

Versions of packages grass-gui recommends:
ii ghostscript 9.05~dfsg-6
ii python-imaging 1.1.7-4

grass-gui suggests no packages.
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1672&gt;
GRASS GIS <http://grass.osgeo.org>

#1672: gcpmanager fails to list source mapsets if non-ascii characters in grass
data dir path
----------------------------------------------+-----------------------------
Reporter: hamish | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: wxGUI | Version: 6.4.2
Keywords: georectifier, UnicodeDecodeError | Platform: Linux
      Cpu: x86-64 |
----------------------------------------------+-----------------------------

Comment(by hamish):

see also

* http://bugs.debian.org/595815
  "sqlite database update failure with Unicode characters in wxpython gui"

and

* http://bugs.debian.org/595910
  "wxpython: save of workspace on exit crasches on non-ascii layer name and
corrupts workspace file"

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1672#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#1672: gcpmanager fails to list source mapsets if non-ascii characters in grass
data dir path
----------------------------------------------+-----------------------------
Reporter: hamish | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 6.4.3
Component: wxGUI | Version: svn-trunk
Keywords: georectifier, UnicodeDecodeError | Platform: Linux
      Cpu: x86-64 |
----------------------------------------------+-----------------------------
Changes (by torsti):

  * version: 6.4.2 => svn-trunk

Comment:

This one is still around (r55643). It can be traced back to
'self.grassdatabase' in '!OnLocation' (gui/wxpython/gcp/manager.py, line
376), which gets its value at line 94:

{{{
self.grassdatabase = grass.gisenv()['GISDBASE']
}}}

'gisenv' is defined at line 720 in lib/python/script/core.py:
{{{
s = read_command("g.gisenv", flags='n')
return parse_key_val(s)
}}}

Neither the string 's' read from g.gisenv nor the dictionary returned by
'parse_key_val' is explicitly decoded to Unicode, so when it is finally
implicitly cast to Unicode somewhere in Python's bowels, the default ascii
codec is used, which leads to errors if $GISDBASE contains non ascii
characters.

This a wider python (at least for < 3.x) problem, and fixing it for this
specific issue won't probably solve all existing !UnicodeDecodeErrors nor
prevent new ones from surfacing, but at least any GUI component calling
gisenv can probably be fixed at one location.

Any filenames/paths read into wxpython/pygrass should probably be
explicitly decoded to Unicode, e.g. with the 'decode()' function defined
at line 79 in script/core.py. Enforcing only ascii chars works for mapset
and location names, but can't/shouldn't be the solution for path
components not under GRASS's direct control.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1672#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>