[GRASS-dev] [GRASS GIS] #1293: Creating mapset with non-latin letter gives python ascii error

GDAL · March 3, 2011, 8:15pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Start GRASS WXGUI. In startup screen create a new mapset called "āšņļ" ->
Unable to create mapset: 'ascii' codec cant decode blahblah (can not
copy/paste text ). Still it creates new mapset and I can select it and
start GRASS session.

WinGRASS-6.4.1RC1-1 on Windows Vista
{{{
C:/Program Files/GRASS-64/etc/wxpython/gis_set.py:623: UnicodeWarning:
Unicode e
qual comparison failed to convert both arguments to Unicode - interpreting
them
as being unequal
   if mapset not in self.listOfMapsetsSelectable or \
C:/Program Files/GRASS-64/etc/wxpython/gis_set.py:725: UnicodeWarning:
Unicode e
qual comparison failed to convert both arguments to Unicode - interpreting
them
as being unequal
   self.lbmapsets.SetSelection(self.listOfMapsets.index(mapset))
C:/Program Files/GRASS-64/etc/wxpython/gis_set.py:680: UnicodeWarning:
Unicode e
qual comparison failed to convert both arguments to Unicode - interpreting
them
as being unequal
   if event.GetText() not in self.listOfMapsetsSelectable:

}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 3, 2011, 8:19pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by marisn):

Seems that not only startup screen is failing with non-latin mapsets. No
idea where this error comes from. Got it in WXGUI "Command console" by
using mapset named "šaursliežu dzelzceļš".

{{{
C:\Program
Files\GRASS-64\etc\wxpython\gui_modules\goutput.py:748:
UnicodeWarning: Unicode equal comparison failed to convert
both arguments to Unicode - interpreting them as being
unequal
if mapName in mapLayers:
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 3, 2011, 8:55pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by marisn):

Another place, that fails: g.help

{{{
Traceback (most recent call last):
File "c:/osgeo4w/usr/src/grass-6.4.1RC1/dist.i686-pc-
mingw32/etc/wxpython/gui_modules/prompt.py", line 777, in
OnItemSelected
File "C:\Program
Files\GRASS-64\etc\wxpython\gui_modules\menuform.py", line
2022, in ParseInterface

tree = etree.fromstring(getInterfaceDescription(cmd[0]))
File "C:\Program
Files\GRASS-64\etc\wxpython\gui_modules\menuform.py", line
1975, in getInterfaceDescription

"Details: %s") % (cmd, cmderr)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xd0 in position 30: ordinal
not in range(128)

}}}

And afterwards things are going only downhill from there. Starting a new
GRASS WXGUI session greets with python crash and skipping of startup
screen:

{{{
KŽ█DA:KARąU KOPA ┼Īaurslie┼Šu dzelzce─╝┼Ī nav atrasta
C:/Program Files/GRASS-64/etc/wxpython/gis_set.py:623: UnicodeWarning:
Unicode e
qual comparison failed to convert both arguments to Unicode - interpreting
them
as being unequal
   if mapset not in self.listOfMapsetsSelectable or \
Traceback (most recent call last):
   File "C:/Program Files/GRASS-64/etc/wxpython/gis_set.py", line 858, in
<module
>
     GRASSStartUp = StartUp(0)
   File "C:\OSGeo4W\apps\Python25\lib\site-packages\wx-2.8-msw-
unicode\wx\_core.p
y", line 7935, in __init__
   File "C:\OSGeo4W\apps\Python25\lib\site-packages\wx-2.8-msw-
unicode\wx\_core.p
y", line 7509, in _BootstrapApp
   File "C:/Program Files/GRASS-64/etc/wxpython/gis_set.py", line 829, in
OnInit
     StartUp = GRASSStartup()
   File "C:/Program Files/GRASS-64/etc/wxpython/gis_set.py", line 166, in
__init_
_
     self._set_properties()
   File "C:/Program Files/GRASS-64/etc/wxpython/gis_set.py", line 241, in
_set_pr
operties
     (utils.UnicodeString(mapset))
   File "C:\Program Files\GRASS-64\etc\wxpython\gui_modules\utils.py", line
669,
in UnicodeString
     return unicode(string, enc)
   File "C:\Program Files\GRASS-64\Python25\lib\encodings\cp1257.py", line
15, in
  decode
     return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0xa1 in position 1:
charac
ter maps to <undefined>
}}}

Contents of .grassrc6 ("šaursliežu dzelzceļš" is a mapset in location
"spearfish60" and not "demo"):

{{{
GISDBASE: C:\Users\tests\Documents\GIS_DataBase
LOCATION_NAME: demolocation
MAPSET: šaursliežu dzelzceļš
GRASS_GUI: wxpython
MAPSET : PERMANENT
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 4, 2011, 2:41pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by glynn):

Replying to [ticket:1293 marisn]:

Note that 8-bit characters are not valid within names of locations,
mapsets or maps. It should refuse to create the mapset, albeit with a more
meaningful error message.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 4, 2011, 3:03pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by martinl):

Replying to [comment:3 glynn]:

> Note that 8-bit characters are not valid within names of locations,
mapsets or maps. It should refuse to create the mapset, albeit with a more
meaningful error message.

How complicated would be to allow 8-bit characters in the
location/mapset/map names? BTW, on Linux, I am able to create location,
mapset with 8-bit characters, start GRASS with created mapset and display
maps from this mapset.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 4, 2011, 4:03pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by martinl):

* cc: martinl (added)

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 5, 2011, 10:31am

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by marisn):

Replying to [comment:3 glynn]:
> Note that 8-bit characters are not valid within names of locations,
mapsets or maps. It should refuse to create the mapset, albeit with a more
meaningful error message.
URL to documentation? RFC?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 5, 2011, 11:42am

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by glynn):

Replying to [comment:4 martinl]:

> How complicated would be to allow 8-bit characters in the
location/mapset/map names?

It's not particularly complicated, just a lot of work. We would have to
audit the whole of GRASS for any issues, then fix them.

For code which simply passes strings around within GRASS, it isn't an
issue. It becomes an issue when code tries to interpret strings or pass
them outside of GRASS.

The main issue is that once you use characters outside of ASCII, encoding
becomes an issue. Many functions (e.g. case conversions) are affected by
the current locale. Also, if the map name is written to a file, some file
formats require the use of specific encodings, so we would have to convert
the text to that encoding to avoid creating invalid files. E.g. if a map
name is in ISO-8859-* and it gets written directly to a file format which
uses UTF-8, the result is an invalid file.

If you stick to ASCII, none of this matters. All commonly-used encodings
are supersets of ASCII, so conversion between ASCII and such encodings is
a no-op. Once you move outside of ASCII you have to perform conversions,
which means knowing which encoding you're converting to/from.

Another significant issue is that it's unspecified whether "char" is
signed or unsigned. Signed is more common, which can break code which
asssumes that a char ranges from 0 to 255.

> BTW, on Linux, I am able to create location, mapset with 8-bit
characters, start GRASS with created mapset and display maps from this
mapset.

That indicates that G_legal_name() isn't being called when it should. This
function tests for invalid characters:
{{{
         if (*s == '/' || *s == '"' || *s == '\'' || *s <= ' ' ||
             *s == '@' || *s == ',' || *s == '=' || *s == '*' || *s > 0176)
{
             G_warning(_("Illegal filename <%s>. Character <%c> not
allowed.\n"), name, *s);
             return -1;
         }
}}}

This prohibits all 8-bit characters (due to either the "*s > 0176" or the
"*s <= ' '", depending upon whether char is signed or unsigned).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 9, 2011, 8:59pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: wingrass | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by hellik):

* keywords: => wingrass

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 9, 2011, 9:47pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: 6.4.1 RCs
Keywords: wingrass | Platform: MSWindows Vista
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by glynn):

Replying to [comment:8 hellik]:
> keywords wingrass added

Is this actually Windows-specific?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:9>
GRASS GIS <http://grass.osgeo.org>

GDAL · April 3, 2013, 1:50pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Unspecified
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by torsti):

* cc: torsti (added)
  * keywords: wingrass =>
  * version: 6.4.1 RCs => svn-trunk
  * platform: MSWindows Vista => Unspecified

Comment:

This bug persists in a fairly recent grass7 on Linux. But it's really two
separate issues: Unicode handling in the python GUI (lost of other tickets
on that...) and the fact that the validity of the mapset name isn't being
checked.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:10>
GRASS GIS <http://grass.osgeo.org>

Robert_Lagace3 · April 3, 2013, 2:02pm

Can you give me an example?

Thank,

Robert Lagacé
________________________________________
De : grass-dev-bounces@lists.osgeo.org [grass-dev-bounces@lists.osgeo.org] de la part de GRASS GIS [trac@osgeo.org]
Date d'envoi : 3 avril 2013 09:50
Objet : Re: [GRASS-dev] [GRASS GIS] #1293: Creating mapset with non-latin letter gives python ascii error

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Unspecified
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by torsti):

* cc: torsti (added)
  * keywords: wingrass =>
  * version: 6.4.1 RCs => svn-trunk
  * platform: MSWindows Vista => Unspecified

Comment:

This bug persists in a fairly recent grass7 on Linux. But it's really two
separate issues: Unicode handling in the python GUI (lost of other tickets
on that...) and the fact that the validity of the mapset name isn't being
checked.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:10>
GRASS GIS <http://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

GDAL · April 3, 2013, 2:28pm

#1293: Creating mapset with non-latin letter gives python ascii error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 6.4.1
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Unspecified
Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by annakrat):

Replying to [comment:10 torsti]:
> This bug persists in a fairly recent grass7 on Linux. But it's really
two separate issues: Unicode handling in the python GUI (lost of other
tickets on that...) and the fact that the validity of the mapset name
isn't being checked.

When you create new mapset within the startup screen, it is checked (added
1 or 2 months ago). I don't understand why g.mapset itself doesn't check
it, it prints only warning but the mapset with illegal name is created
anyway.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1293#comment:11>
GRASS GIS <http://grass.osgeo.org>