[GRASS-dev] [GRASS GIS] #1681: WXGUI vector editing fails with python unicode error

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Linux
      Cpu: Unspecified |
-------------------------+--------------------------------------------------
Steps to reproduce:
  * Create a new GRASS vector data set;
  * In "Attribute table" dialog choose "Manage table" and add a single text
attribute;
  * Rightclick layer and choose "Start editing";
  * Add a feature and fill text field with: "āšņļ";
  * Click "Submit" and observe infamous unicode error in WXGUI console.

{{{
Traceback (most recent call last):
   File "/home/maris/soft/grass_trunk/dist.x86_64-unknown-
linux-gnu/etc/gui/wxpython/dbmgr/dialogs.py", line 318, in
OnSubmit

for sql in self.GetSQLString(updateValues = True):
   File "/home/maris/soft/grass_trunk/dist.x86_64-unknown-
linux-gnu/etc/gui/wxpython/dbmgr/dialogs.py", line 241, in
GetSQLString

updatedValues.append("'" + str(newvalue) + "'")
UnicodeEncodeError
:
'ascii' codec can't encode characters in position 3-4:
ordinal not in range(128)
}}}

"Data browser" encoding is set to "utf-8"

{{{
GRASS 7.0.svn (Igaunija):~/soft/grass_trunk > locale
LANG=lv_LV
LC_CTYPE="lv_LV.utf8"
LC_NUMERIC="lv_LV.utf8"
LC_TIME="lv_LV.utf8"
LC_COLLATE="lv_LV.utf8"
LC_MONETARY="lv_LV.utf8"
LC_MESSAGES="lv_LV.utf8"
LC_PAPER="lv_LV.utf8"
LC_NAME="lv_LV.utf8"
LC_ADDRESS="lv_LV.utf8"
LC_TELEPHONE="lv_LV.utf8"
LC_MEASUREMENT="lv_LV.utf8"
LC_IDENTIFICATION="lv_LV.utf8"
LC_ALL=lv_LV.utf8
}}}

Notice to all developers - if You do anything user visible or provided
text related, keep "āšņļ" and "йцукен" at Your hand all the time, please.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Linux
      Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by mlennert):

I can confirm this bug.

Setting utf-8 in the GUI preferences Attribut tab, I can edit the
attribute table and add 'āšņļ', but even with that setting, I cannot edit
the attributes in the digitizer. I get the same error message as Maris.

I think it might be helpful to try to develop guidelines for developers on
how to handle non-ascii. Especially in the wxGUI this issue comes up very
often.

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1681#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Linux
      Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by marisn):

  * priority: major => blocker

Comment:

It is still an issue with current trunk. As it prevents users from
creating new vector data sets, I'm setting this as a blocker. Unless GRASS
7.0 ships without wxGUI, UnicodeEncodeError needs to be eliminated in it.

Some reading for those, who have problems with strings in Python:
http://blog.notdot.net/2010/07/Getting-unicode-right-in-Python

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: | Platform: Linux
      Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by annakrat):

Replying to [comment:2 marisn]:
> It is still an issue with current trunk. As it prevents users from
creating new vector data sets, I'm setting this as a blocker. Unless GRASS
7.0 ships without wxGUI, UnicodeEncodeError needs to be eliminated in it.
>
> Some reading for those, who have problems with strings in Python:
http://blog.notdot.net/2010/07/Getting-unicode-right-in-Python

I attached a diff which solves some of the issues. I didn't commit it
because it's not complete (I don't have much time for it). So feel free to
improve it.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: encoding | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------
Changes (by annakrat):

  * keywords: => encoding
  * platform: Linux => Unspecified

Comment:

Replying to [comment:3 annakrat]:
>
> I attached a diff which solves some of the issues. I didn't commit it
because it's not complete (I don't have much time for it). So feel free to
improve it.

I extended the diff and it seems to work somehow so I committed it in
r61897. Needs testing.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: encoding | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by neteler):

Replying to [comment:4 annakrat]:
> I extended the diff and it seems to work somehow so I committed it in
r61897. Needs testing.

Can this be backported?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
-------------------------+--------------------------------------------------
Reporter: marisn | Owner: grass-dev@…
     Type: defect | Status: new
Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Keywords: encoding | Platform: Unspecified
      Cpu: Unspecified |
-------------------------+--------------------------------------------------

Comment(by mlennert):

Replying to [comment:5 neteler]:
> Replying to [comment:4 annakrat]:
> > I extended the diff and it seems to work somehow so I committed it in
r61897. Needs testing.
>
> Can this be backported?

It works for me in trunk at least. So I would say +1.

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#1681: WXGUI vector editing fails with python unicode error
--------------------------+-------------------------------------------------
  Reporter: marisn | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: blocker | Milestone: 7.0.0
Component: wxGUI | Version: svn-trunk
Resolution: fixed | Keywords: encoding
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Changes (by annakrat):

  * status: new => closed
  * resolution: => fixed

Comment:

Replying to [comment:6 mlennert]:
> Replying to [comment:5 neteler]:
> > Replying to [comment:4 annakrat]:
> > > I extended the diff and it seems to work somehow so I committed it
in r61897. Needs testing.
> >
> > Can this be backported?
>
> It works for me in trunk at least. So I would say +1.

Thanks for testing, backported in r62378. I will close it for now, however
encoding issues generally are not solved properly yet.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1681#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>