[GRASS-dev] [GRASS GIS] #882: i18n enabled　winGRASS: properties dialog not opening

GRASS GRASS developer
1

#882: i18n enabled winGRASS: properties dialog not opening
---------------------+------------------------------------------------------
Reporter: neteler | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Keywords: | Platform: MSWindows XP
      Cpu: x86-32 |
---------------------+------------------------------------------------------
In the current 6.4.svn version, with NLS enabled, the wxGUI dialog window
for selecting raster and vector maps does not open. Also right-mouse
button/properties fail to open.

Using Japanese XP.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882&gt;
GRASS GIS <http://grass.osgeo.org>

2

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by martinl):

Hopefully fixed in r40553, try
http://josef.fsv.cvut.cz/wingrass/grass64/WinGRASS-6.4.SVN-
r40553-1-Setup.exe. Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

3

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by neteler):

Unfortunately the problem persists (WinGRASS-6.4.SVN-r40553-1-Setup.exe).
Using XP (Japanese).

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

4

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by neteler):

I just discovered that in the terminal some problems are indicated:
{{{
Traceback (most recent call last):
   File "C:/Program
Files/GRASS-64-SVN/etc/wxpython/wxgui.py", line 1260, in
OnAddRaster

self.AddRaster(event)
   File "C:/Program
Files/GRASS-64-SVN/etc/wxpython/wxgui.py", line 1385, in
AddRaster

self.curr_page.maptree.AddLayer('raster')
   File "C:\Program
Files\GRASS-64-SVN\etc\wxpython\gui_modules\wxgui_utils.py",
line 792, in AddLayer

self.PropertiesDialog(layer, show=True)
   File "C:\Program
Files\GRASS-64-SVN\etc\wxpython\gui_modules\wxgui_utils.py",
line 858, in PropertiesDialog

parentframe=self)
   File "C:\Program
Files\GRASS-64-SVN\etc\wxpython\gui_modules\menuform.py",
line 1819, in ParseCommand

handler)
   File "C:\Program
Files\GRASS-64-SVN\Python25\lib\xml\sax\__init__.py", line
49, in parseString

parser.parse(inpsrc)
   File "C:\Program
Files\GRASS-64-SVN\Python25\lib\xml\sax\expatreader.py",
line 107, in parse

xmlreader.IncrementalParser.parse(self, source)
   File "C:\Program
Files\GRASS-64-SVN\Python25\lib\xml\sax\xmlreader.py", line
123, in parse

self.feed(buffer)
   File "C:\Program
Files\GRASS-64-SVN\Python25\lib\xml\sax\expatreader.py",
line 211, in feed

self._err_handler.fatalError(exc)
   File "C:\Program
Files\GRASS-64-SVN\Python25\lib\xml\sax\handler.py", line
38, in fatalError

raise exception
xml.sax._exceptions
.
SAXParseException
:
<unknown>:1:30: unknown encoding
}}}

Maybe that gives an idea.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

5

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by glynn):

Replying to [comment:3 neteler]:

> I just discovered that in the terminal some problems are indicated:
{{{
SAXParseException
:
<unknown>:1:30: unknown encoding
}}}
>
> Maybe that gives an idea.

Look at the first line of the output from `d.rast --interface-
description`. The encoding= parameter is probably something Windows-
specific like "CP932". In which case, we may need to use something other
than locale_charset() (e.g. bind_textdomain_codeset() with NULL for the
codeset parameter), or even convert the text to UTF-8.

Except ... GUI.!ParseCommand() assumes that the text is in the locale's
encoding and is explicitly converting it to UTF-8. But it doesn't change
the encoding= in the XML header, which will still contain the locale's
encoding (maybe this pre-dates r17034 being back-ported?).

The fact that it's xml.sax.parseString() which is failing rather than the
".decode(enc)" in the argument suggests that Python understands the
encoding, so it may suffice to do something like:

{{{
- ... .decode(enc).encode("utf-8") ...
+ ... .decode(enc).split('\n',1)[1].replace('', '<?xml version="1.0"
encoding="utf-8"?>\n', 1).encode("utf-8") ...
}}}

The situation is complicated by the fact that 6.5/7.0 has abandoned
xml.sax in favour of xml.etree.!ElementTree.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

6

On Wed, Jan 20, 2010 at 7:18 AM, GRASS GIS <trac@osgeo.org> wrote:

#882: i18n enabled winGRASS: properties dialog not opening

...
http://trac.osgeo.org/grass/ticket/882

Look at the first line of the output from `d.rast --interface-
description`. The encoding= parameter is probably something Windows-
specific like "CP932". In which case, we may need to use something other
than locale_charset() (e.g. bind_textdomain_codeset() with NULL for the
codeset parameter), or even convert the text to UTF-8.

(posting here since trac doesn't let me login right now)

The first line on the Japanese XP box reports:
<?xml version="1.0" encoding="CP932" ?>

Markus

7

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by neteler):

On a Japanese Linux box, the current encoding is:
{{{
r.cost --interface-description
<?xml version="1.0" encoding="EUC-JP"?>
}}}

but should be UTF8.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

8

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by neteler):

On a Japanese Windows-XP box, the current encoding is:

{{{
<?xml version="1.0" encoding="CP932" ?>
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

9

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords:
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by glynn):

Replying to [comment:5 neteler]:
> On a Japanese Linux box, the current encoding is:
{{{
r.cost --interface-description
<?xml version="1.0" encoding="EUC-JP"?>
}}}
>
> but should be UTF8.

What do you mean by "should be"? Are the descriptions encoded in EUC-JP or
in UTF-8? First and foremost, the encoding specified in the header must be
the encoding which is actually used for the data.

The strings to which `opt->label` and `opt->description` point will be in
the locale's encoding, as that's how `dgettext()` works, and that's how
they need to be encoded so that the --help output displays correctly on
the user's terminal. `G__usage_xml()` doesn't recode the data; it just
copies the strings into the output.

If you want the --interface-description output to always use UTF-8, then
`G__usage_xml()` will need to recode any localised strings with `iconv()`.

There shouldn't be any problem with encoding=EUC-JP, but encoding=CP932
may be problematic as "CP932" isn't an IANA-registered encoding (the IANA
name is "Windows-31J"), so it may not be recognised by XML parsers.

[http://www.iana.org/assignments/character-sets IANA-registered encodings]

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

10

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Changes (by neteler):

  * keywords: => wingrass
  * platform: MSWindows XP => All

Comment:

Replying to [comment:7 glynn]:
> Replying to [comment:5 neteler]:
> > On a Japanese Linux box, the current encoding is:
> {{{
> r.cost --interface-description
> <?xml version="1.0" encoding="EUC-JP"?>
> }}}
> >
> > but should be UTF8.
>
> What do you mean by "should be"?

OK: s/should be/IMHO/

> Are the descriptions encoded in EUC-JP or in UTF-8?

The grass*_ja.po files are in UTF8.

> First and foremost, the encoding specified in the header must be the
encoding which is actually used for the data.

That's why I assumed a mismatch (both for Win and Linux, but in different
ways).

> The strings to which `opt->label` and `opt->description` point will be
in the locale's encoding, as that's how `dgettext()` works, and that's how
they need to be encoded so that the --help output displays correctly on
the user's terminal. `G__usage_xml()` doesn't recode the data; it just
copies the strings into the output.
>
> If you want the --interface-description output to always use UTF-8, then
`G__usage_xml()` will need to recode any localised strings with `iconv()`.
>
> There shouldn't be any problem with encoding=EUC-JP, but encoding=CP932
may be problematic as "CP932" isn't an IANA-registered encoding (the IANA
name is "Windows-31J"), so it may not be recognised by XML parsers.

I did not have success with encoding=EUC-JP while the .po files are in
UTF8. Same of course for encoding=CP932.

> [http://www.iana.org/assignments/character-sets IANA-registered
encodings]

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

11

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Comment (by glynn):

Replying to [comment:8 neteler]:

> > Are the descriptions encoded in EUC-JP or in UTF-8?
>
> The grass*_ja.po files are in UTF8.

That's not what I'm asking. Is the text in the <description> and <label>
elements in the --interface-description output encoded in EUC-JP or in
UTF-8? If it's EUC-JP, then the header should say encoding=EUC-JP. If it's
in UTF-8, the header should say encoding=UTF-8.

> I did not have success with encoding=EUC-JP while the .po files are in
UTF8. Same of course for encoding=CP932.

The encoding of the .po files is irrelevant. `dgettext()` converts the
strings from the .mo files to the locale's encoding, so that you don't
need separate .mo files for e.g. LANG=ja_JP (Japanese, EUC-JP) and
LANG=ja_JP.utf-8 (Japanese, UTF-8).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

12

On Fri, Jan 22, 2010 at 4:30 PM, GRASS GIS <trac@osgeo.org> wrote:

#882: i18n enabled winGRASS: properties dialog not opening

..

Comment (by glynn):

...

That's not what I'm asking. Is the text in the <description> and <label>
elements in the --interface-description output encoded in EUC-JP or in
UTF-8? If it's EUC-JP, then the header should say encoding=EUC-JP. If it's
in UTF-8, the header should say encoding=UTF-8.

Right. But how to find out? Maybe I don't see the obvious.

Markus

13

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Comment (by neteler):

Testing on Linux (EN is default locale):

{{{
GRASS 6.4.0svn (nc_spm_08):~ > r.walk --interface-description | grep -v
encoding | grep -v DOCTYPE > en.txt

GRASS 6.4.0svn (nc_spm_08):~ > . ja.sh
GRASS 6.4.0svn (nc_spm_08):~ > r.walk --interface-description | grep
encoding
<?xml version="1.0" encoding="EUC-JP"?>
GRASS 6.4.0svn (nc_spm_08):~ > r.walk --interface-description | grep -v
encoding | grep -v DOCTYPE > ja.txt

GRASS 6.4.0svn (nc_spm_08):~ > file en.txt ja.txt
en.txt: ASCII English text
ja.txt: ISO-8859 English text

GRASS 6.4.0svn (nc_spm_08):~ > r.walk
Traceback (most recent call last):
   File "/home/neteler/grass64/dist.x86_64-unknown-linux-
gnu/etc/wxpython/gui_modules/menuform.py", line 1937, in <module>
...
xml.sax._exceptions.SAXParseException: <unknown>:12:4: not well-formed
(invalid token)
}}}

So: while it advertises EUC-JP, the content appears to be ISO-8859 if I
got the test right (thanks for hint to Hamish).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

14

Markus Neteler wrote:

> #882: i18n enabled winGRASS: properties dialog not opening
..
> Comment (by glynn):
...
> That's not what I'm asking. Is the text in the <description> and <label>
> elements in the --interface-description output encoded in EUC-JP or in
> UTF-8? If it's EUC-JP, then the header should say encoding=EUC-JP. If it's
> in UTF-8, the header should say encoding=UTF-8.

Right. But how to find out? Maybe I don't see the obvious.

1.
  r3.info --interface-description > desc.txt

[Pick any command which actually has Japanese translations.]

2.
  iconv -f utf-8 -t utf-8 desc.txt >/dev/null

  iconv -f euc-jp -t utf-8 desc.txt >/dev/null

One of these commands should work, the other should produce an
"illegal input sequence" error.

--
Glynn Clements <glynn@gclements.plus.com>

15

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Comment (by glynn):

Replying to [comment:10 neteler]:
{{{
GRASS 6.4.0svn (nc_spm_08):~ > file en.txt ja.txt
en.txt: ASCII English text
ja.txt: ISO-8859 English text
}}}
>
> So: while it advertises EUC-JP, the content appears to be ISO-8859 if I
got the test right (thanks for hint to Hamish).

Nah. "file"'s encoding detection is rather rudimentary. If it sees a mix
of ASCII and 8-bit characters, and it isn't UTF-8, it concludes that it's
ISO-8859.

Based upon this I'd say that it's EUC-JP; a Japanese locale on Unix is
likely to use either UTF-8 or EUC-JP, and file can detect UTF-8. If you
want a more accurate test, try iconv which each encoding in turn and see
which one works, or load the file in Firefox and see which encoding it
selects.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

16

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Comment (by neteler):

Here the test:

{{{
[neteler@north ~]$ iconv -f utf-8 -t utf-8 ja.txt >/dev/null
iconv: illegal input sequence at position 465

[neteler@north ~]$ iconv -f euc-jp -t utf-8 ja.txt >/dev/null
[neteler@north ~]$
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

17

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Comment (by glynn):

Replying to [comment:12 neteler]:
> Here the test:

Yep; that's EUC-JP, which is what I'd expect for LANG=ja_JP (UTF-8 locales
normally have a .UTF-8 suffix, e.g. ja_JP.UTF-8).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

18

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Changes (by neteler):

  * platform: All => MSWindows XP

Comment:

Replying to [comment:13 glynn]:
> Replying to [comment:12 neteler]:
> > Here the test:
>
> Yep; that's EUC-JP, which is what I'd expect for LANG=ja_JP (UTF-8
locales normally have a .UTF-8 suffix, e.g. ja_JP.UTF-8).

I see, so it was an improper setting here. So, setting on Linux:
{{{
export LANG=ja_JP.UTF-8
export LANGUAGE=ja_JP.UTF-8
export LC_MESSAGES=ja_JP.UTF-8
}}}

enables r.walk to come up with a translated wxGUI. Excellent.

This reduces the problem to the Windows encoding issue.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

19

#882: i18n enabled winGRASS: properties dialog not opening
---------------------------+------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: MSWindows XP | Cpu: x86-32
---------------------------+------------------------------------------------
Comment (by glynn):

Replying to [comment:14 neteler]:

> I see, so it was an improper setting here. So, setting on Linux:
{{{
export LANG=ja_JP.UTF-8
export LANGUAGE=ja_JP.UTF-8
export LC_MESSAGES=ja_JP.UTF-8
}}}
>
> enables r.walk to come up with a translated wxGUI. Excellent.
>
> This reduces the problem to the Windows encoding issue.

Hmm; no. If it doesn't work with LANG=ja_JP, that's a bug.

Did you try the fix suggested in comment:4?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>

20

#882: i18n enabled winGRASS: properties dialog not opening
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: blocker | Milestone: 6.4.0
Component: wxGUI | Version: svn-releasebranch64
Resolution: | Keywords: wingrass
  Platform: All | Cpu: x86-32
----------------------+-----------------------------------------------------
Changes (by neteler):

  * platform: MSWindows XP => All

Comment:

I have tried (now attached) patch from comment 4, on Linux box.

{{{
# session 1:
GRASS 6.4.0svn (nc_spm_08):~ > set | grep ja_
LANG=ja_JP
LANGUAGE=ja_JP
LC_MESSAGES=ja_JP
GRASS 6.4.0svn (nc_spm_08):~ > r.walk --interface-description | grep
encoding
<?xml version="1.0" encoding="UTF-8"?>
}}}

and

{{{
# session 2:
GRASS 6.4.0svn (nc_spm_08):~ > set | grep ja_
LANG=ja_JP.UTF-8
LANGUAGE=ja_JP.UTF-8
LC_MESSAGES=ja_JP.UTF-8
GRASS 6.4.0svn (nc_spm_08):~ > r.walk --interface-description | grep
encoding
<?xml version="1.0" encoding="UTF-8"?>
}}}

In both cases the wxGUI of r.walk comes now up in Japanese.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/882#comment:16&gt;
GRASS GIS <http://grass.osgeo.org>