[GRASS-dev] [GRASS GIS] #3392: t.register: encoding error

#3392: t.register: encoding error
---------------------------------+-------------------------
Reporter: mlennert | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Keywords: t.register encoding | CPU: Unspecified
Platform: Unspecified |
---------------------------------+-------------------------
Using the data and examples from [http://ncsu-geoforall-lab.github.io
/grass-temporal-workshop/ 1] in a french (fr_BE@UTF-8) encoding
environment, I get an encoding error using t.register:

{{{
g.list type=raster pattern="*tempmean" output=fichier_cartes_tempmean
t.create output=pluies_nc semantictype=sum title=precipitation_mois
description="Précipitation totale mensuelle NC"
t.register -i input=pluies_nc@climate_2000_2012
file=fichier_cartes_tempmean start=2000-01-01 increment="1 months"
Collecte des informations sur la carte ...
  100%
Registering maps in the temporal database...
Registering maps in the space time dataset...
Updating space time dataset...
ERREUR :ascii,Mettre à jour les métadonnées, l'emprise spatiale et
         temporelle de toutes les cartes enregistrées de <,7,8,ordinal not
         in range(128)
}}}

And when I run t.register again:

{{{
t.register -i input=pluies_nc@climate_2000_2012
file=fichier_cartes_tempmean start=2000-01-01 increment="1 months"
Collecte des informations sur la carte ...
ATTENTION: Map is already registered in temporal database. Unable to
update
            raster map <2000_01_tempmean@climate_2000_2012>. Overwrite flag
            is not set.
[...]
ATTENTION: Map is already registered in temporal database. Unable to
update
            raster map <2012_12_tempmean@climate_2000_2012>. Overwrite flag
            is not set.
  100%
Registering maps in the space time dataset...
ERREUR :ascii,%s> est déjà enregistrée.,9,10,ordinal not in range(128)
}}}

ould be supercool if this could still be solved for 7.2.2 :slight_smile:

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by zarch):

* Attachment "temporal.patch" added.

from str() => unicode()

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by zarch):

Replying to [ticket:3392 mlennert]:
> Using the data and examples from [http://ncsu-geoforall-lab.github.io
/grass-temporal-workshop/ 1] in a french (fr_BE@UTF-8) encoding
environment, I get an encoding error using t.register:

I was able to reproduce the error, actually the #3394, and with the
attached patch is working on my PC.
Can you test it?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by zarch):

* Attachment "temporal_encoding.diff" added.

Convert str => encoding

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by zarch):

* Attachment "temporal_script_utils_encoding.diff" added.

Convert value to bytes/unicode if is not already

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by zarch):

Replying to [comment:1 zarch]:
> Replying to [ticket:3392 mlennert]:
> > Using the data and examples from [http://ncsu-geoforall-lab.github.io
/grass-temporal-workshop/ 1] in a french (fr_BE@UTF-8) encoding
environment, I get an encoding error using t.register:
>
> I was able to reproduce the error, actually the #3394, and with the
attached patch is working on my PC.
> Can you test it?

I remembered that once Glynn in grass-dev (2016-05-05) said:

''"Anything using the script module should be using byte strings
throughout (in spite of how awkward Python 3 tries to make this)."''

So I think that instead of returning unicode as I did in the first patch
we should return bytes...
Please apply the two diff files, and let me know if it is working for you.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by huhabla):

Dear Pietro,
thanks a lot for taking care of this issue. Unfortunately i have no clue
howto fix this. If your patch works well, i would like to put it in trunk
for more tests.

Replying to [comment:2 zarch]:
> Replying to [comment:1 zarch]:
> > Replying to [ticket:3392 mlennert]:
> > > Using the data and examples from [http://ncsu-geoforall-
lab.github.io/grass-temporal-workshop/ 1] in a french (fr_BE@UTF-8)
encoding environment, I get an encoding error using t.register:
> >
> > I was able to reproduce the error, actually the #3394, and with the
attached patch is working on my PC.
> > Can you test it?
>
> I remembered that once Glynn in grass-dev (2016-05-05) said:
>
> ''"Anything using the script module should be using byte strings
throughout (in spite of how awkward Python 3 tries to make this)."''
>
> So I think that instead of returning unicode as I did in the first patch
we should return bytes...
> Please apply the two diff files, and let me know if it is working for
you.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:2 zarch]:
> Replying to [comment:1 zarch]:
> > Replying to [ticket:3392 mlennert]:
> > > Using the data and examples from [http://ncsu-geoforall-
lab.github.io/grass-temporal-workshop/ 1] in a french (fr_BE@UTF-8)
encoding environment, I get an encoding error using t.register:
> >
> > I was able to reproduce the error, actually the #3394, and with the
attached patch is working on my PC.
> > Can you test it?
>
> I remembered that once Glynn in grass-dev (2016-05-05) said:
>
> ''"Anything using the script module should be using byte strings
throughout (in spite of how awkward Python 3 tries to make this)."''
>
> So I think that instead of returning unicode as I did in the first patch
we should return bytes...
> Please apply the two diff files, and let me know if it is working for
you.

I've applied your patches (BTW: it is always best to make diffs relative
to the source tree root. That way one doesn't have to search for where the
patched files are...).

{{{
t.create output=pluies_nc semantictype=sum title=precipitation_mois
description="Précipitation totale mensuelle NC"Traceback (most recent call
last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.create", line 86, in <module>
     main()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.create", line 82, in main
     semantic, None, grass.overwrite())
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/open_stds.py", line 174, in open_new_stds
     sp.insert(dbif)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 381,
in insert
     statement += AbstractDataset.insert(self, dbif=dbif, execute=False)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_dataset.py", line 398, in insert
     statement += self.metadata.get_insert_statement_mogrified(dbif)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/base.py", line 404, in
get_insert_statement_mogrified
     mapset=self.mapset)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 946, in mogrify_sql_statement
     return self.connections[mapset].mogrify_sql_statement(content)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 1216, in
mogrify_sql_statement
     unicode(args[count]),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2:
ordinal not in range(128)
}}}

However:

{{{
t.create output=pluies_nc semantictype=sum title=precipitation_mois
description="Precipitation totale mensuelle NC"
}}}

Note the accent in the description of the first try, but not the second.
This wasn't a problem before applying your patch.

Then

{{{
t.register -i input=pluies_nc@climate_2000_2012
file=fichier_cartes_tempmean start=2000-01-01 increment="1 months"
Collecte des informations sur la carte ...
  100%
Enregistrement des cartes dans la base temporelle ...
Enregistrement des cartes dans le jeu de données temporel ...
Mise à jour du jeu de données temporel ...
ERREUR :ascii,Mettre à jour les métadonnées, l'emprise spatiale et
         temporelle de toutes les cartes enregistrées de <,7,8,ordinal not
         in range(128)
}}}

And :

{{{
t.unregister file=fichier_cartes_tempmean
Unregister maps
  100%
Unregister maps from the temporal database
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.unregister", line 183, in <module>
     tgis.profile_function(main)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 85, in profile_function
     func()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.unregister", line 169, in main
     sp.update_from_registered_maps(dbif)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 2295,
in update_from_registered_maps
     " all registered maps of <%s>") % (self.get_id()))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
ordinal not in range(128)
}}}

And:

{{{
t.remove pluies_nc -fNote: registered maps themselves have not been
removed, only the strds
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.remove", line 171, in <module>
     tgis.profile_function(main)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 85, in profile_function
     func()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.remove", line 157, in main
     statement += sp.delete(dbif=dbif, execute=False)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 1959,
in delete
     self.get_id()))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23:
ordinal not in range(128)
}}}

If this is a fundamental issue in the temporal modules then we should
probably keep the discussion centralized here. I'll close the other bug
reports as duplicates of this one.

t.info (#3394) and t.rast.extract (#3395) now work, but I suspect that the
errors there came from the accent in the description of the strds.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by zarch):

Replying to [comment:4 mlennert]:
> Replying to [comment:2 zarch]:
> > So I think that instead of returning unicode as I did in the first
patch we should return bytes...
> > Please apply the two diff files, and let me know if it is working for
you.
>
> I've applied your patches (BTW: it is always best to make diffs relative
to the source tree root. That way one doesn't have to search for where the
patched files are...).

Ops, sorry I did not notice that the diff was sensitive to my current
working directory.

\\

> t.info (#3394) and t.rast.extract (#3395) now work, but I suspect that
the errors there came from the accent in the description of the strds.

Indeed the error was due to `é` character in the description. I was able
to reproduce only the error with `t.info`. I added a test r71391.

\\

I'm not able to reproduce these errors in: `t.create`, `t.register`,
`t.unregister`, `t.remove`.

I executed the testsuite in `grass/lib/python/temporal` and
`grass/temporal` with no regressions.
But, since I was able to reproduce only the `t.info` error, it would be
nice, before do the commit, to add tests in: t.create, t.register,
t.unregister, t.remove that problem.

At the moment I'm not able to write the tests for them, someone can help
me?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by huhabla):

Moritz,
can you please provide more information about the errors? Some of them
occurred by calling self.get_id(). The ascii conversion of ids that are
build using "name@mapset" produces an error. Does the map or the mapset
contain special characters?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:5 zarch]:
> I'm not able to reproduce these errors in: `t.create`, `t.register`,
`t.unregister`, `t.remove`.

Ok, here I go again (leaving the stdrs desciption issue aside for the
moment, so t.create is not an issue), with a completely fresh svn checkout
of trunk, applied your two patches, 'temporal_encoding.diff​' and
'temporal_script_utils_encoding.diff​', compiled.

Get the [http://fatra.cnr.ncsu.edu/temporal-grass-
workshop/NC_spm_temporal_workshop.zip demo location] from the [http
://ncsu-geoforall-lab.github.io/grass-temporal-workshop/ FOSS4G 2014
workshop site]. Use the climate_2000_2012 mapset.

{{{
GRASS 7.3.svn (NC_spm_temporal_workshop):~ > g.version -g
version=7.3.svn
date=2017
revision=r71392M
build_date=2017-08-11
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
}}}

{{{
GRASS 7.3.svn (NC_spm_temporal_workshop):~ > locale
LANG=fr_BE
LANGUAGE=fr_BE
LC_CTYPE=fr_BE.UTF-8
LC_NUMERIC=C
LC_TIME=fr_BE.UTF-8
LC_COLLATE=fr_BE.UTF-8
LC_MONETARY=fr_BE.UTF-8
LC_MESSAGES=fr_BE.UTF-8
LC_PAPER=fr_BE.UTF-8
LC_NAME=fr_BE.UTF-8
LC_ADDRESS=fr_BE.UTF-8
LC_TELEPHONE=fr_BE.UTF-8
LC_MEASUREMENT=fr_BE.UTF-8
LC_IDENTIFICATION=fr_BE.UTF-8
LC_ALL=
}}}

Then follow the steps described there (with light modifications for ease):

{{{
t.create output=tempmean type=strds temporaltype=absolute title="Average
temperature" description="Monthly temperature average in NC [deg C]"
g.list type=raster pattern="*tempmean" > rasters_tempmean.list
t.register -i input=tempmean type=raster start=2000-01-01 increment="1
months" file=rasters_tempmean.list
Collecte des informations sur la carte ...
  100%
Enregistrement des cartes dans la base temporelle ...
Enregistrement des cartes dans le jeu de données temporel ...
Mise à jour du jeu de données temporel ...
ERREUR :ascii,Mettre à jour les métadonnées, l'emprise spatiale et
         temporelle de toutes les cartes enregistrées de <,7,8,ordinal not
         in range(128)
}}}

t.list and t.list raster seem to show that everything is alright, but IIUC
the output of t.info and t.topology seems to indicate that something went
wrong:

{{{
t.info tempmean@climate_2000_2012
  +-------------------- Space Time Raster Dataset
-----------------------------+
  |
|
  +-------------------- Basic information
-------------------------------------+
  | Id: ........................ tempmean@climate_2000_2012
  | Name: ...................... tempmean
  | Mapset: .................... climate_2000_2012
  | Creator: ................... mlennert
  | Temporal type: ............. absolute
  | Creation time: ............. 2017-08-11 18:12:14.819514
  | Modification time:.......... 2017-08-11 18:12:14.819546
  | Semantic type:.............. mean
  +-------------------- Absolute time
-----------------------------------------+
  | Start time:................. None
  | End time:................... None
  | Granularity:................ None
  | Temporal type of maps:...... None
  +-------------------- Spatial extent
----------------------------------------+
  | North:...................... None
  | South:...................... None
  | East:.. .................... None
  | West:....................... None
  | Top:........................ None
  | Bottom:..................... None
  +-------------------- Metadata information
----------------------------------+
  | Raster register table:......
raster_map_register_031cdb28ede04a64b03d115101b86c0b
  | North-South resolution min:. None
  | North-South resolution max:. None
  | East-west resolution min:... None
  | East-west resolution max:... None
  | Minimum value min:.......... None
  | Minimum value max:.......... None
  | Maximum value min:.......... None
  | Maximum value max:.......... None
  | Aggregation type:........... None
  | Number of registered maps:.. None
  |
  | Title:
  | Average temperature
  | Description:
  | Monthly temperature average in NC [deg C]
  | Command history:
  | # 2017-08-11 18:12:14
  | t.create output="tempmean" type="strds"
  | temporaltype="absolute" title="Average temperature"
  | description="Monthly temperature average in NC [deg C]"
  |
+----------------------------------------------------------------------------+
}}}

{{{
t.topology tempmean@climate_2000_2012
  +-------------------- Basic information
-------------------------------------+
  | Id: ........................ tempmean@climate_2000_2012
  | Name: ...................... tempmean
  | Mapset: .................... climate_2000_2012
  | Creator: ................... mlennert
  | Temporal type: ............. absolute
  | Creation time: ............. 2017-08-11 18:12:14.819514
  | Modification time:.......... 2017-08-11 18:12:14.819546
  | Semantic type:.............. mean
  +-------------------- Temporal topology
-------------------------------------+
  | Is subset of dataset: ...... False
  | Temporal topology is: ...... invalid
  | Number of intervals: ....... 156
  | Invalid time stamps: ....... 0
  | Number of points: .......... 0
  | Number of gaps: ............ 0
  | Granularity: ............... 1 month
  +-------------------- Topological relations
---------------------------------+
  | Overlaps: .................. 0
  | Overlapped: ................ 0
  | Finishes: .................. 0
  | Started: ................... 0
  | Follows: ................... 155
  | Contains: .................. 0
  | Equal:...................... 0
  | Finished: .................. 0
  | Precedes: .................. 155
  | Starts: .................... 0
  | During: .................... 0
+----------------------------------------------------------------------------+
}}}

{{{
t.remove -f tempmean@climate_2000_2012
Note: registered maps themselves have not been removed, only the strds
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.remove", line 171, in <module>
     tgis.profile_function(main)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 85, in profile_function
     func()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.remove", line 157, in main
     statement += sp.delete(dbif=dbif, execute=False)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 1959,
in delete
     self.get_id()))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23:
ordinal not in range(128)
}}}

{{{
t.unregister file=rasters_tempmean.list
Unregister maps
  100%
Unregister maps from the temporal database
Traceback (most recent call last):
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.unregister", line 183, in <module>
     tgis.profile_function(main)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/core.py", line 85, in profile_function
     func()
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/scripts/t.unregister", line 169, in main
     sp.update_from_registered_maps(dbif)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/temporal/abstract_space_time_dataset.py", line 2295,
in update_from_registered_maps
     " all registered maps of <%s>") % (self.get_id()))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7:
ordinal not in range(128)
}}}

So, I still have exactly the same issues. My guess is that most of them
come from the fact that the message strings are flagged as translatable
and that in the French translation there are accents... The t.register one
I find more difficult to understand.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:6 huhabla]:
> Moritz,
> can you please provide more information about the errors? Some of them
occurred by calling self.get_id(). The ascii conversion of ids that are
build using "name@mapset" produces an error. Does the map or the mapset
contain special characters?

No, as I said, I use a demo data set. I'm pretty sure it's an issue with
the handling of message translations.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Maybe the solution needs to be found in lib/python/pygrass/messages ?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by huhabla):

* Attachment "pygrass_temporal_gis_utf_handling.diff" added.

Approach to fix the handling of utf-8 strings and translations in the
temporal framework

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by huhabla):

Replying to [comment:9 mlennert]:
> Maybe the solution needs to be found in lib/python/pygrass/messages ?

Good point. I have tried to fix this issue and attached the
"pygrass_temporal_gis_utf_handling.diff​" that must be applied to fresh
trunk.
Can you please test the changes? I am not able to convince my GRASS
installation to use the German locale that i have set, so its a kind of
dry test.

https://trac.osgeo.org/grass/raw-
attachment/ticket/3392/pygrass_temporal_gis_utf_handling.diff

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:10 huhabla]:
> Replying to [comment:9 mlennert]:
> > Maybe the solution needs to be found in lib/python/pygrass/messages ?
>
> Good point. I have tried to fix this issue and attached the
"pygrass_temporal_gis_utf_handling.diff​" that must be applied to fresh
trunk.
> Can you please test the changes?

They don't seem to work:

{{{
t.create output=tempmean type=strds temporaltype=absolute title="Average
temperature" description="Monthly temperature average in NC [deg C]"
Process Process-1:
Traceback (most recent call last):
   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
     self.run()
   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
     self._target(*self._args, **self._kwargs)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/messages/__init__.py", line 95, in
message_server
     libgis.G_verbose_message(message.encode("utf-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2:
ordinal not in range(128)
}}}

and, even worse:

{{{
t.register -i input=tempmean type=raster start=2000-01-01 increment="1
months" file=rasters_tempmean.list
Collecte des informations sur la carte ...
Process Process-1:
Traceback (most recent call last):
   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
     self.run()
   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
     self._target(*self._args, **self._kwargs)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/messages/__init__.py", line 93, in
message_server
     libgis.G_debug(level, message.encode("utf-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)
ATTENTION: Needed to restart the messenger server
Process Process-3:
Traceback (most recent call last):
   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
     self.run()
   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
     self._target(*self._args, **self._kwargs)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/messages/__init__.py", line 93, in
message_server
     libgis.G_debug(level, message.encode("utf-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)
ATTENTION: Needed to restart the messenger server
Process Process-4:
Traceback (most recent call last):
   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
_bootstrap
     self.run()
   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
     self._target(*self._args, **self._kwargs)
   File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/messages/__init__.py", line 93, in
message_server
     libgis.G_debug(level, message.encode("utf-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)
ATTENTION: Needed to restart the messenger server
[...]
}}}

I'm also not sure whether using u' strings is the solution, nor how
portable this is between Python 2 and 3. Anna should probably have a look
at this as she has quite some experience with the encoding issues...

>I am not able to convince my GRASS installation to use the German locale
that i have set, so its a kind of dry test.

You can use the GUI: Settings->Preferences->Appearance->Language settings.

However: looking at grasslibs_de.po I have the feeling that most temporal
lib strings have not been translated to German, yet. Maybe these strings
have only recently been translated to French, making the errors appear now
?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:11&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by annakrat):

The problem mostly comes from calling gisenv function from script.utils
which returns unicode. This was done some time ago to make the library
Python3 compatible. So we eventually need to move towards unicode, but to
have a quick fix for the release, we can keep the temporal library using
bytestrings. This is a possible solution:

{{{
Index: lib/python/temporal/core.py

--- lib/python/temporal/core.py (revision 71395)
+++ lib/python/temporal/core.py (working copy)
@@ -542,9 +542,9 @@
      grassenv = gscript.gisenv()

      # Set the global variable for faster access
- current_mapset = grassenv["MAPSET"]
- current_location = grassenv["LOCATION_NAME"]
- current_gisdbase = grassenv["GISDBASE"]
+ current_mapset = gscript.encode(grassenv["MAPSET"])
+ current_location = gscript.encode(grassenv["LOCATION_NAME"])
+ current_gisdbase = gscript.encode(grassenv["GISDBASE"])

      # Check environment variable GRASS_TGIS_RAISE_ON_ERROR
      if os.getenv("GRASS_TGIS_RAISE_ON_ERROR") == "True" or \
@@ -1107,6 +1107,7 @@
                          detect_types=self.dbmi.PARSE_DECLTYPES |
self.dbmi.PARSE_COLNAMES)
                  self.connection.row_factory = self.dbmi.Row
                  self.connection.isolation_level = None
+ self.connection.text_factory = str
                  self.cursor = self.connection.cursor()
                  self.cursor.execute("PRAGMA synchronous = OFF")
                  self.cursor.execute("PRAGMA journal_mode = MEMORY")

}}}

plus all gisenv calls in temporal modules need to be encoded to not let
any unicode into the library. I would appreciate if somebody could test it
more thoroughly. This would be a temporary solution, we need to move to
unicode eventually, but I think that would require more time to fix and
more significant changes.

Also, this works only for sqlite3 backend, I haven't looked at postgres...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:12&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by zarch):

Hi Anna,

Replying to [comment:12 annakrat]:
> The problem mostly comes from calling gisenv function from script.utils
which returns unicode. This was done some time ago to make the library
Python3 compatible. So we eventually need to move towards unicode, but to
have a quick fix for the release, we can keep the temporal library using
bytestrings.

I didn't have time to test your solution yet.

Actually for me it is still not clear where should we use `unicode` and
where `bytes`.
Do you have some general guidelines?

In the changes that I did, I transformed everything in bytes, and now the
tests that I added to reproduce the ticket's error are passing, so the
ticket is "theoretically" fixed. However this solution can only work with
python2, because in the temporal library now I'm summing bytes with
unicode, that can work only with python2 that does not distinguish between
them.
In python3 this operation raise an error. In general due to bytes
limitation in python3 (mainly there is not format) perhaps we should use
just unicode within the python code...

Therefore I was thinking to replace all the `grass.script.utils` `encode`
function that I added in `grass/lib/temporal` with `decode` to ensure that
everything is unicode. And then replace the `print` function with
`pygrass.messages.Messager` to eventually translate all the python unicode
back to bytes just before printing them to system terminal.

So I'm not sure when is the right moment to transform unicode to bytes.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:13&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Changes (by zarch):

* Attachment "temporal_encode.patch" added.

Transform every variable to bytes

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392&gt;
GRASS GIS <https://grass.osgeo.org>

#3392: t.register: encoding error
--------------------------+---------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.2.2
Component: Temporal | Version: svn-trunk
Resolution: | Keywords: t.register encoding
       CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------

Comment (by mlennert):

Replying to [comment:12 annakrat]:
> The problem mostly comes from calling gisenv function from script.utils
which returns unicode.

So you are saying that this has nothing to do with translated message
strings ?

>This was done some time ago to make the library Python3 compatible. So we
eventually need to move >towards unicode, but to have a quick fix for the
release, we can keep the temporal library using >bytestrings. This is a
possible solution:
>
>
> {{{
> Index: lib/python/temporal/core.py
> ===================================================================
> --- lib/python/temporal/core.py (revision 71395)
> +++ lib/python/temporal/core.py (working copy)
> @@ -542,9 +542,9 @@
> grassenv = gscript.gisenv()
>
> # Set the global variable for faster access
> - current_mapset = grassenv["MAPSET"]
> - current_location = grassenv["LOCATION_NAME"]
> - current_gisdbase = grassenv["GISDBASE"]
> + current_mapset = gscript.encode(grassenv["MAPSET"])
> + current_location = gscript.encode(grassenv["LOCATION_NAME"])
> + current_gisdbase = gscript.encode(grassenv["GISDBASE"])
>
> # Check environment variable GRASS_TGIS_RAISE_ON_ERROR
> if os.getenv("GRASS_TGIS_RAISE_ON_ERROR") == "True" or \
> @@ -1107,6 +1107,7 @@
> detect_types=self.dbmi.PARSE_DECLTYPES |
self.dbmi.PARSE_COLNAMES)
> self.connection.row_factory = self.dbmi.Row
> self.connection.isolation_level = None
> + self.connection.text_factory = str
> self.cursor = self.connection.cursor()
> self.cursor.execute("PRAGMA synchronous = OFF")
> self.cursor.execute("PRAGMA journal_mode = MEMORY")
>
> }}}
>

Just the line

{{{
self.connection.text_factory = str
}}}

already suffices to solve the above-mentioned issues (and I've gone a bit
further in the tutorial and tested a series of other t.* modules, but
haven't encountered any other error) but it's better to also protect
against non-ascii gisdbase variables which wasn't an issue here for me.

Thanks Anna !!

I have to admit that I have no idea how this works. Some form of
documentation of the entire system would be very helpful, in line with
what Pietro said.

> plus all gisenv calls in temporal modules need to be encoded to not let
any unicode into the library. I would appreciate if somebody could test it
more thoroughly. This would be a temporary solution, we need to move to
unicode eventually, but I think that would require more time to fix and
more significant changes.

Can we at least already commit this small change which already makes a big
difference ?

> Also, this works only for sqlite3 backend, I haven't looked at
postgres...

How does the database backend come in when dealing with these strings ?

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3392#comment:14&gt;
GRASS GIS <https://grass.osgeo.org>