[GRASS-dev] [GRASS GIS] #2021: encoding information in locale gets lost

#2021: encoding information in locale gets lost
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Startup | Version: svn-trunk
Keywords: locale encoding | Platform: Linux
      Cpu: Unspecified |
-----------------------------+----------------------------------------------
My locale settings before launching GRASS 7:

{{{
$ locale
LANG=fr_BE.UTF-8
LANGUAGE=fr_BE:fr
LC_CTYPE="fr_BE.UTF-8"
LC_NUMERIC="fr_BE.UTF-8"
LC_TIME="fr_BE.UTF-8"
LC_COLLATE="fr_BE.UTF-8"
LC_MONETARY="fr_BE.UTF-8"
LC_MESSAGES="fr_BE.UTF-8"
LC_PAPER="fr_BE.UTF-8"
LC_NAME="fr_BE.UTF-8"
LC_ADDRESS="fr_BE.UTF-8"
LC_TELEPHONE="fr_BE.UTF-8"
LC_MEASUREMENT="fr_BE.UTF-8"
LC_IDENTIFICATION="fr_BE.UTF-8"
LC_ALL=
}}}

locale settings after launching GRASS 7:

{{{
> locale
LANG=fr_BE
LANGUAGE=fr_BE
LC_CTYPE=fr_BE
LC_NUMERIC=C
LC_TIME=fr_BE
LC_COLLATE=fr_BE
LC_MONETARY=fr_BE
LC_MESSAGES=fr_BE
LC_PAPER=fr_BE
LC_NAME=fr_BE
LC_ADDRESS=fr_BE
LC_TELEPHONE=fr_BE
LC_MEASUREMENT=fr_BE
LC_IDENTIFICATION=fr_BE
LC_ALL=
}}}

Thus the special characters in translated messages are not displayed
correctly.

This is with svn revision 56918.

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
-----------------------------+----------------------------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Startup | Version: svn-trunk
Keywords: locale encoding | Platform: Linux
      Cpu: Unspecified |
-----------------------------+----------------------------------------------

Comment(by mlennert):

Replying to [ticket:2021 mlennert]:
> My locale settings before launching GRASS 7:
>
>
> {{{
> $ locale
> LANG=fr_BE.UTF-8
> LANGUAGE=fr_BE:fr
> LC_CTYPE="fr_BE.UTF-8"
> LC_NUMERIC="fr_BE.UTF-8"
> LC_TIME="fr_BE.UTF-8"
> LC_COLLATE="fr_BE.UTF-8"
> LC_MONETARY="fr_BE.UTF-8"
> LC_MESSAGES="fr_BE.UTF-8"
> LC_PAPER="fr_BE.UTF-8"
> LC_NAME="fr_BE.UTF-8"
> LC_ADDRESS="fr_BE.UTF-8"
> LC_TELEPHONE="fr_BE.UTF-8"
> LC_MEASUREMENT="fr_BE.UTF-8"
> LC_IDENTIFICATION="fr_BE.UTF-8"
> LC_ALL=
> }}}
>
>
> locale settings after launching GRASS 7:
>
>
> {{{
> > locale
> LANG=fr_BE
> LANGUAGE=fr_BE
> LC_CTYPE=fr_BE
> LC_NUMERIC=C
> LC_TIME=fr_BE
> LC_COLLATE=fr_BE
> LC_MONETARY=fr_BE
> LC_MESSAGES=fr_BE
> LC_PAPER=fr_BE
> LC_NAME=fr_BE
> LC_ADDRESS=fr_BE
> LC_TELEPHONE=fr_BE
> LC_MEASUREMENT=fr_BE
> LC_IDENTIFICATION=fr_BE
> LC_ALL=
> }}}
>
>
> Thus the special characters in translated messages are not displayed
correctly.
>
> This is with svn revision 56918.

The culprit seems to be r52517 which includes the following line in
lib/init.grass.py:

{{{
language = language.split('.')[0] # Split off ignored .encoding part if
present
}}}

Why should the encoding part be split off, especially since it is _not_
ignored as implied by the comment. Module messages that have been
translated (in my case into French) and include special characters are not
displayed correctly.

When I comment out this line, I get messages in the correct encoding.

Maris, could you explain the reason for taking away the encoding info ?
I'm tempted to comment out the line for now and see if anyone reports
negative consequences, but I'll wait a few days to see if there is a
reason for it.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
-----------------------+----------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.0
Component: Startup | Version: svn-trunk
Resolution: fixed | Keywords: locale encoding
  Platform: Linux | Cpu: Unspecified
-----------------------+----------------------------------------------------
Changes (by mlennert):

  * status: new => closed
  * resolution: => fixed

Comment:

This seems to have been fixed by r59163, so closing.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
--------------------------+-----------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 7.0.1
Component: Startup | Version: svn-trunk
Resolution: | Keywords: locale encoding
       CPU: Unspecified | Platform: Linux
--------------------------+-----------------------------
Changes (by mlennert):

* status: closed => reopened
* resolution: fixed =>
* milestone: 7.0.0 => 7.0.1

Comment:

Reopening this, as with a fairly fresh trunk (r65364), I get:

Before running the grass startup script:

{{{
$ locale
LANG=fr_BE.utf8
LANGUAGE=
LC_CTYPE="fr_BE.utf8"
LC_NUMERIC="fr_BE.utf8"
LC_TIME="fr_BE.utf8"
LC_COLLATE="fr_BE.utf8"
LC_MONETARY="fr_BE.utf8"
LC_MESSAGES="fr_BE.utf8"
LC_PAPER="fr_BE.utf8"
LC_NAME="fr_BE.utf8"
LC_ADDRESS="fr_BE.utf8"
LC_TELEPHONE="fr_BE.utf8"
LC_MEASUREMENT="fr_BE.utf8"
LC_IDENTIFICATION="fr_BE.utf8"
LC_ALL=
}}}

After running the script:

{{{
> locale
LANG=C
LANGUAGE=C
LC_CTYPE=C
LC_NUMERIC=C
LC_TIME=C
LC_COLLATE=C
LC_MONETARY=C
LC_MESSAGES=C
LC_PAPER=C
LC_NAME=C
LC_ADDRESS=C
LC_TELEPHONE=C
LC_MEASUREMENT=C
LC_IDENTIFICATION=C
LC_ALL=
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
--------------------------+-----------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 7.0.1
Component: Startup | Version: svn-trunk
Resolution: | Keywords: locale encoding
       CPU: Unspecified | Platform: Linux
--------------------------+-----------------------------

Comment (by marisn):

Replying to [comment:3 mlennert]:
> Reopening this, as with a fairly fresh trunk (r65364), I get:
Please provide locale line from GRASS wx configuration file.

set_language() kicks in only if a language override is requested within
wxgui settings.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
--------------------------+-----------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 7.0.1
Component: Startup | Version: svn-trunk
Resolution: | Keywords: locale encoding
       CPU: Unspecified | Platform: Linux
--------------------------+-----------------------------

Comment (by mlennert):

Replying to [comment:4 marisn]:
> Replying to [comment:3 mlennert]:
> > Reopening this, as with a fairly fresh trunk (r65364), I get:
> Please provide locale line from GRASS wx configuration file.
>
> set_language() kicks in only if a language override is requested within
wxgui settings.

Yep, that was it. There was an override in the wx config file.

However, this is quite confusing as I was not even using the wxgui. I
don't think wxgui config should interfere with command line.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#2021: encoding information in locale gets lost
--------------------------+-----------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 7.0.1
Component: Startup | Version: svn-trunk
Resolution: | Keywords: locale encoding
       CPU: Unspecified | Platform: Linux
--------------------------+-----------------------------

Comment (by marisn):

Hm. Probably set_language should be moved to some place to kick-in only if
a GUI session is started.

Rant:
The architectural problem lies in the fact that GRASS is not a monolithic
app as i.e. QGIS. GUI's can be started from CLI and there are CLI apps too
(and, of course, "native" widgets like "open file"!). Thus setting a
language in wxgui tries to cover all potential aspects to ensure a
constant user experience.
The problem areas are: lack of target locale on particular system (only
GRASS messages will be translated); incorrect default encoding (i.e. for
Latvian language Python is certain that Win-1257 is the best thing on my
UTF-8 Linux box and then hell breaks loose!); enforcing language on
everything (cal; "OK" and "Cancel" buttons; all GRASS module output as it
might be piped back to GUI; did I forgot something?
As one can see - it is easy to mess something up and "quick fixes" need to
ensure covering most of aspects as it is too easy to make things worse.
(just check my own commits for language switching and you will see that it
is true :wink: )

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2021#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>