[GRASS-dev] [GRASS GIS] #3446: GRASS python library is not working with python 3

GDAL · November 16, 2017, 5:16pm

#3446: GRASS python library is not working with python 3
-------------------------------------+-------------------------
Reporter: lrntct | Owner: grass-dev@…
Type: defect | Status: new
Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Keywords: python python3 temporal | CPU: x86-64
Platform: Linux |
-------------------------------------+-------------------------
Various pygrass modules fail to import in Python 3.6.

Python version:
{{{
Python 3.6.3 (default, Oct 3 2017, 21:45:48)
[GCC 7.2.0] on linux
}}}

For example:
{{{
import grass.pygrass.gis
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/grass72/etc/python/grass/pygrass/gis/__init__.py", line
14, in <module>
     libgis.G_gisinit('')
   File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in
G_gisinit
     return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
}}}

This issue prevents pygrass and the temporal framework to work with
python3:

{{{
>>> import grass.temporal as tgis
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/grass72/etc/python/grass/temporal/__init__.py", line 3,
in <module>
     from .core import *
   File "/usr/lib/grass72/etc/python/grass/temporal/core.py", line 45, in
<module>
     from .c_libraries_interface import *
   File
"/usr/lib/grass72/etc/python/grass/temporal/c_libraries_interface.py",
line 28, in <module>
     from grass.pygrass.rpc.base import RPCServerBase
   File "/usr/lib/grass72/etc/python/grass/pygrass/rpc/__init__.py", line
21, in <module>
     from grass.pygrass.vector import *
   File "/usr/lib/grass72/etc/python/grass/pygrass/vector/__init__.py",
line 6, in <module>
     libgis.G_gisinit('')
   File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in
G_gisinit
     return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 17, 2017, 6:14am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------
Changes (by zarch):

* keywords: python python3 temporal => python python3 ctypes

Comment:

Just remove the keyword: temporal and add ctypes, since the issue is due
to ctypes and affect all the library based on them: temporal, pygrass.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:1>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 17, 2017, 6:30am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by neteler):

Just FYI - upstream there are a few new Python 3 fixes as PR available:

https://github.com/davidjamesca/ctypesgen/pull/58

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:2>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 19, 2017, 10:27pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by neteler):

See also #2708

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:3>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 11, 2017, 3:40pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by lrntct):

Replying to [comment:2 neteler]:
> Just FYI - upstream there are a few new Python 3 fixes as PR available:
>
> https://github.com/davidjamesca/ctypesgen/pull/58

If GRASS relies on ctypesgen, we have a problem, because the project is
likely dead. The latest commit was 3 years ago, and the maintainer does
not seems interested anymore (see
https://github.com/davidjamesca/ctypesgen/issues/52).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:4>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 11, 2017, 3:49pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by neteler):

Replying to [comment:4 lrntct]:
> Replying to [comment:2 neteler]:
> > Just FYI - upstream there are a few new Python 3 fixes as PR
available:
> >
> > https://github.com/davidjamesca/ctypesgen/pull/58
>
> If GRASS relies on ctypesgen, we have a problem, because the project is
likely dead. The latest commit was 3 years ago, and the maintainer does
not seems interested anymore (see
https://github.com/davidjamesca/ctypesgen/issues/52).

Yeah, but the beauty of open source development is that someone else can
take over:

https://github.com/olsonse/ctypesgen

"This branch is 154 commits ahead of davidjamesca:master." (see also the
many forks at https://github.com/olsonse/ctypesgen/network)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:5>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 21, 2018, 1:58pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.1
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by martinl):

BTW, currently ctypes compilation with Python3 enabled fails with

{{{
python3 -t -m py_compile /var/local/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.py
   File "/var/local/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.py", line 5779
     PORT_LONG_MAX = 2147483647L
                               ^
SyntaxError: invalid syntax
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:7>
GRASS GIS <https://grass.osgeo.org>

GDAL · April 4, 2018, 9:41am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.1
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by epifanio):

Hi, do you have any directions on how to test and debug `pygrass` build
against `python3`? Which version/branch of `ctypegen`? Is anyone from the
dev team in contact (open issues?) with ctypegen devs?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:8>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 6, 2018, 5:00pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.4.2
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by neteler):

This fork seems to contain Python3 changes:
https://github.com/billhill00/ctypesgen

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:10>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 7, 2018, 10:00pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------
Changes (by martinl):

* milestone: 7.4.2 => 7.8.0

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:11>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 7, 2018, 10:02pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------

Comment (by martinl):

For record, currently compilation fails (using Python 3.7) here

{{{
Status: gcc -E -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/include -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include
-D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline=" "-D__inline__="
"-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" "-D__asm__(x)="
"-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpi1g2g39s.h"
Traceback (most recent call last):
File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line
1021, in parse_template
this = chr(ESCAPES[this][1])
KeyError: '\\x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "./ctypesgen.py", line 162, in <module>
     descriptions = ctypesgencore.parser.parse(options.headers, options)
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/__init__.py",
line 22, in parse
     parser.parse()
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py",
line 74, in parse
     ctypesparser.CtypesParser.parse(self, fname, False)
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/cparser.py",
line 122, in parse
     self.preprocessor_parser.parse(filename)
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py",
line 226, in parse
     token = self.lexer.token()
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py",
line 65, in token
     result = lex.Lexer.token(self)
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/lex.py",
line 359, in token
     newtok = func(tok)
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py",
line 286, in t_ANY_string_literal
     t.value = StringLiteral(encode(t.value))
   File
"/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py",
line 84, in __new__
     r'\x0\1', value[1:-1])
   File "/home/martin/python3-grass7/lib/python3.7/re.py", line 192, in sub
     return _compile(pattern, flags).sub(repl, string, count)
   File "/home/martin/python3-grass7/lib/python3.7/re.py", line 309, in
_subx
     template = _compile_repl(template, pattern)
   File "/home/martin/python3-grass7/lib/python3.7/re.py", line 300, in
_compile_repl
     return sre_parse.parse_template(repl, pattern)
   File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line
1024, in parse_template
     raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \x at position 0
make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1
make[1]: Leaving directory '/opt/src/grass7_trunk/lib/python/ctypes'
make: *** [Makefile:81: default] Error 2
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:12>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 7, 2018, 10:10pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------
Changes (by martinl):

* keywords: python python3 ctypes => python python3 ctypes, python 3.7

Comment:

Good news: compilation does not fail with Python 3.6.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:13>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 13, 2018, 8:51am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by epifanio):

@martinl do you have any direction on how to test a grass build against
python3? @all, will be worth it to have a dedicated wiki page for it?
Thanks!!!

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:14>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 13, 2018, 9:44am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by hellik):

Replying to [comment:14 epifanio]:
> @martinl do you have any direction on how to test a grass build against
python3? @all, will be worth it to have a dedicated wiki page for it?
Thanks!!!

see

https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:15>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 13, 2018, 11:51am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by martinl):

Replying to [comment:15 hellik]:

> https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

instead of building wx package I used system packages (python3-wx and
python3-six) + `virtualenv --system-site-packages`.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:16>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 13, 2018, 12:09pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by martinl):

Replying to [comment:16 martinl]:
> Replying to [comment:15 hellik]:
>
> > https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest
>
> instead of building wx package I used system packages (python3-wx and
python3-six) + `virtualenv --system-site-packages`.

I have updated wiki to describe also quick system-side approach.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:17>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 10, 2018, 9:50pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by neteler):

Replying to [comment:12 martinl]:
> For record, currently compilation fails (using Python 3.7) here

For a related patch attempt, see #3704: ctypes compilation error with
Python-3.7

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:18>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 10, 2018, 9:55pm

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by neteler):

Now I have an issue with g.extension:

{{{
GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
Traceback (most recent call last):
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1758, in <module>
     sys.exit(main())
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1737, in main
     install_extension(source=source, url=url, xmlurl=xmlurl)
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 697, in install_extension
     source=source, url=url)
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
     outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1113, in download_source_code
     fix_newlines(directory)
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1043, in fix_newlines
     if '\0' in data:
TypeError: a bytes-like object is required, not 'str'
}}}

Likewise with other addon modules to be installed...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:19>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 11, 2018, 12:23am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by lucadelu):

Replying to [comment:19 neteler]:
> Now I have an issue with g.extension:
>
> {{{
> GRASS 7.7.svn (utm32N):~ > python --version
> Python 3.7.1
>
> GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
> Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
> Traceback (most recent call last):
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1758, in <module>
> sys.exit(main())
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1737, in main
> install_extension(source=source, url=url, xmlurl=xmlurl)
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 697, in install_extension
> source=source, url=url)
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
> outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1113, in download_source_code
> fix_newlines(directory)
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1043, in fix_newlines
> if '\0' in data:
> TypeError: a bytes-like object is required, not 'str'
> }}}
>
> Likewise with other addon modules to be installed...

I think most of the addons since they are mostly written for Python >= 2.6
We need to port the addons to python3

PS
I think the Python3 support will required major release also to use the
updated addons

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:20>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 11, 2018, 6:09am

#3446: GRASS python library is not working with python 3
---------------------+-----------------------------------------------
  Reporter: lrntct | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone: 7.8.0
Component: Python | Version: 7.2.1
Resolution: | Keywords: python python3 ctypes, python 3.7
       CPU: x86-64 | Platform: Linux
---------------------+-----------------------------------------------

Comment (by neteler):

Replying to [comment:20 lucadelu]:
> Replying to [comment:19 neteler]:
> > Now I have an issue with g.extension:
> >
> > {{{
> > GRASS 7.7.svn (utm32N):~ > python --version
> > Python 3.7.1
> >
> > GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
> > ...
> > File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts/g.extension", line 1043, in fix_newlines
> > if '\0' in data:
> > TypeError: a bytes-like object is required, not 'str'
> > }}}
> >
> > Likewise with other addon modules to be installed...
>
> I think most of the addons since they are mostly written for Python >=
2.6
> We need to port the addons to python3

I had updated some and it also fails with those.

To me it looks like a bug in g.extension itself:

https://trac.osgeo.org/grass/browser/grass/trunk/scripts/g.extension/g.extension.py#L1042

When I delete these lines
{{{
#if '\0' in data:
# continue # ignore binary files
}}}

then g.extension works fine.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3446#comment:21>
GRASS GIS <https://grass.osgeo.org>