[GRASS-dev] python script utils encoding/decoding overriding unicode with python-future

GRASS GRASS developer
1

Pietro,

In r71394 [1] you changed the unicode handling in decode()/encode() in the python script library. However, this broke things on one of my machines [2] because I didn't have the python-future deb package installed, which is normally not supposed to be a compulsory dependency AFAIK.

Is the override of the definition of unicode really necessary ? If I comment the part of the code where you do the override, the errors I had go away (as they do if install python-future).

Moritz

[1] http://trac.osgeo.org/grass/changeset/71394/grass/trunk/lib/python/script/utils.py
[2] https://trac.osgeo.org/grass/ticket/3405

2

Hi Moritz,

On Thu, Aug 24, 2017 at 4:58 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

Is the override of the definition of unicode really necessary ?

No it is not.
it is sufficient replace the try/except with an if.
Done in r71435 [0]

http://trac.osgeo.org/grass/changeset/71435/grass/trunk/lib/python/script/utils.py

3

Unfortunately builtins is used also in other files, and I’m afraid
that I should also fix these:

$ rg "builtins" . -B 1 -A 5
script/task.py
24-try:
25: from builtins import unicode
26- bytes = str
27-except ImportError:
28- # python3
29- unicode = str
30-

temporal/core.py
38-try:
39: from builtins import long
40-except ImportError:
41- # python3
42- long = int
43-
44-from .c_libraries_interface import *

script/vector.py
27- # python3
28: import builtins as __builtin__
29- unicode = str
30-
31-from .utils import parse_key_val
32-from .core import *
33-from grass.exceptions import CalledModuleError

script/temporal_script_utils_encoding.diff
10-+try:
11:+ from builtins import unicode
12-+except ImportError:
13-+ # python3
14-+ unicode = str
15-+
16-+

script/raster.py
31-try:
32: from builtins import unicode
33- bytes = str
34-except ImportError:
35- # python3
36- unicode = str
37-

gunittest/utils.py
59- except ImportError:
60: import builtins as __builtin__
61- __builtin__._ = new_translator
62-
63-
64-_MAX_LENGTH = 80
65-

pygrass/vector/table.py
13-try:
14: from builtins import long, unicode
15-except ImportError:
16- # python3
17- long = int
18- unicode = str
19-

script/core.py
43- # python3
44: import builtins as __builtin__
45- from os import environb as environ
46- unicode = str
47-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
48-
49-

ctypes/OBJ.x86_64-pc-linux-gnu/temporal.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/cluster.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/ogsf.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/proj.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/vector.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/stats.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/arraystats.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/vedit.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/rowio.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/display.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/date.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/raster3d.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/segment.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/gis.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/nviz.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/gmath.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/rtree.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/raster.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/dbmi.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/imagery.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes_pregit/ctypesgencore/processor/operations.py
97-def fix_conflicting_names(data, opts):
98: """If any descriptions from the C code would overwrite Python builtins or
99- other important names, fix_conflicting_names() adds underscores
to resolve
100- the name conflict."""
101-
102- # This is the order of priority for names
103- descriptions = data.functions + data.variables + data.structs + \
--
124- important_names[name] = "a name needed by ctypes or ctypesgen"
125: for name in dir(__builtins__):
126- important_names[name] = "a Python builtin"
127- for name in opts.other_known_names:
128- important_names[name] = "a name from an included Python module"
129- for name in keyword.kwlist:
130- important_names[name] = "a Python keyword"

ctypes/ctypesgencore/printer/preamble.py
1-try:
2: from builtins import long
3-except ImportError:
4- long = int
5-
6-import os
7-import sys

ctypes/ctypesgencore/parser/lex.py
31-try:
32: from builtins import bytes
33- PY3 = True
34-except ImportError:
35- # python2
36- bytes = str
37- PY3 = False

ctypes/ctypesgencore/parser/cgrammar.py
13-try:
14: from builtins import long
15-except ImportError:
16- # python3
17- long = int
18-
19-__docformat__ = 'restructuredtext'

ctypes/ctypesgencore/processor/operations.py
97-def fix_conflicting_names(data, opts):
98: """If any descriptions from the C code would overwrite Python builtins or
99- other important names, fix_conflicting_names() adds underscores
to resolve
100- the name conflict."""
101-
102- # This is the order of priority for names
103- descriptions = data.functions + data.variables + data.structs + \
--
124- important_names[name] = "a name needed by ctypes or ctypesgen"
125: for name in dir(__builtins__):
126- important_names[name] = "a Python builtin"
127- for name in opts.other_known_names:
128- important_names[name] = "a name from an included Python module"
129- for name in keyword.kwlist:
130- important_names[name] = "a Python keyword"

pygrass/vector/testsuite/test_table.py
7-try:
8: from builtins import long
9-except ImportError:
10- # python3
11- long = int
12-
13-import os
--
25-try:
26: from builtins import long
27-except ImportError:
28- long = int
29-
30-# dictionary that generate random data
31-COL2VALS = {'INT': lambda n: np.random.randint(9, size=n),

ctypes/ctypesgencore/parser/pplexer.py
13-try:
14: from builtins import long
15- PY2 = True
16-except ImportError:
17- # python3
18- PY2 = False
19- long = int

4

On 25/08/17 17:06, Pietro wrote:

Unfortunately builtins is used also in other files, and I’m afraid
that I should also fix these:

I think that this would be better, yes.

Is this in grass72 as well, or only trunk ?

Moritz

$ rg "builtins" . -B 1 -A 5
script/task.py
24-try:
25: from builtins import unicode
26- bytes = str
27-except ImportError:
28- # python3
29- unicode = str
30-

temporal/core.py
38-try:
39: from builtins import long
40-except ImportError:
41- # python3
42- long = int
43-
44-from .c_libraries_interface import *

script/vector.py
27- # python3
28: import builtins as __builtin__
29- unicode = str
30-
31-from .utils import parse_key_val
32-from .core import *
33-from grass.exceptions import CalledModuleError

script/temporal_script_utils_encoding.diff
10-+try:
11:+ from builtins import unicode
12-+except ImportError:
13-+ # python3
14-+ unicode = str
15-+
16-+

script/raster.py
31-try:
32: from builtins import unicode
33- bytes = str
34-except ImportError:
35- # python3
36- unicode = str
37-

gunittest/utils.py
59- except ImportError:
60: import builtins as __builtin__
61- __builtin__._ = new_translator
62-
63-
64-_MAX_LENGTH = 80
65-

pygrass/vector/table.py
13-try:
14: from builtins import long, unicode
15-except ImportError:
16- # python3
17- long = int
18- unicode = str
19-

script/core.py
43- # python3
44: import builtins as __builtin__
45- from os import environb as environ
46- unicode = str
47-__builtin__.__dict__['_'] = __builtin__.__dict__['_'].__self__.lgettext
48-
49-

ctypes/OBJ.x86_64-pc-linux-gnu/temporal.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/cluster.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/ogsf.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/proj.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/vector.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/stats.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/arraystats.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/vedit.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/rowio.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/display.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/date.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/raster3d.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/segment.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/gis.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/nviz.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/gmath.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/rtree.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/raster.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/dbmi.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes/OBJ.x86_64-pc-linux-gnu/imagery.py
13-try:
14: from builtins import long
15-except ImportError:
16- long = int
17-
18-import os
19-import sys

ctypes_pregit/ctypesgencore/processor/operations.py
97-def fix_conflicting_names(data, opts):
98: """If any descriptions from the C code would overwrite Python builtins or
99- other important names, fix_conflicting_names() adds underscores
to resolve
100- the name conflict."""
101-
102- # This is the order of priority for names
103- descriptions = data.functions + data.variables + data.structs + \
--
124- important_names[name] = "a name needed by ctypes or ctypesgen"
125: for name in dir(__builtins__):
126- important_names[name] = "a Python builtin"
127- for name in opts.other_known_names:
128- important_names[name] = "a name from an included Python module"
129- for name in keyword.kwlist:
130- important_names[name] = "a Python keyword"

ctypes/ctypesgencore/printer/preamble.py
1-try:
2: from builtins import long
3-except ImportError:
4- long = int
5-
6-import os
7-import sys

ctypes/ctypesgencore/parser/lex.py
31-try:
32: from builtins import bytes
33- PY3 = True
34-except ImportError:
35- # python2
36- bytes = str
37- PY3 = False

ctypes/ctypesgencore/parser/cgrammar.py
13-try:
14: from builtins import long
15-except ImportError:
16- # python3
17- long = int
18-
19-__docformat__ = 'restructuredtext'

ctypes/ctypesgencore/processor/operations.py
97-def fix_conflicting_names(data, opts):
98: """If any descriptions from the C code would overwrite Python builtins or
99- other important names, fix_conflicting_names() adds underscores
to resolve
100- the name conflict."""
101-
102- # This is the order of priority for names
103- descriptions = data.functions + data.variables + data.structs + \
--
124- important_names[name] = "a name needed by ctypes or ctypesgen"
125: for name in dir(__builtins__):
126- important_names[name] = "a Python builtin"
127- for name in opts.other_known_names:
128- important_names[name] = "a name from an included Python module"
129- for name in keyword.kwlist:
130- important_names[name] = "a Python keyword"

pygrass/vector/testsuite/test_table.py
7-try:
8: from builtins import long
9-except ImportError:
10- # python3
11- long = int
12-
13-import os
--
25-try:
26: from builtins import long
27-except ImportError:
28- long = int
29-
30-# dictionary that generate random data
31-COL2VALS = {'INT': lambda n: np.random.randint(9, size=n),

ctypes/ctypesgencore/parser/pplexer.py
13-try:
14: from builtins import long
15- PY2 = True
16-except ImportError:
17- # python3
18- PY2 = False
19- long = int

5

On 25/08/17 16:57, Pietro wrote:

Hi Moritz,

On Thu, Aug 24, 2017 at 4:58 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

Is the override of the definition of unicode really necessary ?

No it is not.
it is sufficient replace the try/except with an if.
Done in r71435 [0]

Great, thanks ! I confirm that this now works.

Moritz

6

On Fri, Aug 25, 2017 at 5:14 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

On 25/08/17 17:06, Pietro wrote:

Unfortunately builtins is used also in other files, and I’m afraid
that I should also fix these:

Actually most of them are working properly, I was thinking that the
module builtins was present in both python2 and python3 and only
contains different functions.

Instead builtins exists only in python3, so most of them works because
in python2 try to import some function from builtins and it raise an
exception not because the function does not exists but because the
module does not exists!

Do you think that should we fix them in any case?

Is this in grass72 as well, or only trunk ?

Changes in scripts.utils is only in trunk:

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/script/utils.py

the other are already in the code.

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/temporal/core.py#L38

Pietro

7

On 25/08/17 17:32, Pietro wrote:

On Fri, Aug 25, 2017 at 5:14 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

On 25/08/17 17:06, Pietro wrote:

Unfortunately builtins is used also in other files, and I’m afraid
that I should also fix these:

Actually most of them are working properly, I was thinking that the
module builtins was present in both python2 and python3 and only
contains different functions.

Instead builtins exists only in python3, so most of them works because
in python2 try to import some function from builtins and it raise an
exception not because the function does not exists but because the
module does not exists!

Yes, but the exception then acts as if it is in Python 3, so this might cause problems (just as we saw for utils.py).

Do you think that should we fix them in any case?

In light of the above, yes.

Is this in grass72 as well, or only trunk ?

Changes in scripts.utils is only in trunk:

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/script/utils.py

  the other are already in the code.

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/temporal/core.py#L38

So this probably needs to be taken care of before the imminent release.

Moritz

8

On Fri, Aug 25, 2017 at 5:39 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/temporal/core.py#L38

So this probably needs to be taken care of before the imminent release.

Done in r71438.
it seems ok now.

9

On 26/08/17 14:42, Pietro wrote:

On Fri, Aug 25, 2017 at 5:39 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/temporal/core.py#L38

So this probably needs to be taken care of before the imminent release.

Done in r71438.
it seems ok now.

Can you backport to release72 ?

Moritz

10

On Sat, Aug 26, 2017 at 3:55 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

On 26/08/17 14:42, Pietro wrote:

On Fri, Aug 25, 2017 at 5:39 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

https://trac.osgeo.org/grass/browser/grass/tags/release_20170810_grass_7_2_2RC1/lib/python/temporal/core.py#L38

So this probably needs to be taken care of before the imminent release.

Done in r71438.
it seems ok now.

Can you backport to release72 ?

Will you also submit ctypes related pull requests upstream?

https://github.com/davidjamesca/ctypesgen

We had several "local" ctypes fixes which should go there, too.

Markus