[GRASS-dev] [GRASS GIS] #3704: ctypes compilation error with Python-3.7

#3704: ctypes compilation error with Python-3.7
-----------------------------+-------------------------
Reporter: neteler | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Keywords: ctypes, python3 | CPU: Unspecified
Platform: Unspecified |
-----------------------------+-------------------------
I checked compilation of trunk with Python3 (using virtualenv) and get a
compile error in ctypes:

{{{
(grasspy3) [mneteler@oboe ctypes ]$ cd
/home/mneteler/software/grass77/lib/python/ctypes/

(grasspy3) [mneteler@oboe ctypes ]$ make
make /home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib
make[1]: Entering directory
'/home/mneteler/software/grass77/lib/python/ctypes'
make[1]: '/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib' is up to date.
make[1]: Leaving directory
'/home/mneteler/software/grass77/lib/python/ctypes'
make OBJ.x86_64-pc-linux-gnu/date.py OBJ.x86_64-pc-linux-gnu/gis.py
OBJ.x86_64-pc-linux-gnu/raster.py OBJ.x86_64-pc-linux-gnu/gmath.py
OBJ.x86_64-pc-linux-gnu/proj.py OBJ.x86_64-pc-linux-gnu/imagery.py
OBJ.x86_64-pc-linux-gnu/vector.py OBJ.x86_64-pc-linux-gnu/rtree.py
OBJ.x86_64-pc-linux-gnu/display.py OBJ.x86_64-pc-linux-gnu/stats.py
OBJ.x86_64-pc-linux-gnu/dbmi.py OBJ.x86_64-pc-linux-gnu/raster3d.py
OBJ.x86_64-pc-linux-gnu/arraystats.py OBJ.x86_64-pc-linux-gnu/cluster.py
OBJ.x86_64-pc-linux-gnu/vedit.py OBJ.x86_64-pc-linux-gnu/segment.py
OBJ.x86_64-pc-linux-gnu/rowio.py OBJ.x86_64-pc-linux-gnu/temporal.py
OBJ.x86_64-pc-linux-gnu/ogsf.py OBJ.x86_64-pc-linux-gnu/nviz.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/date.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/gis.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/raster.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/gmath.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/proj.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/imagery.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/rtree.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/display.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/stats.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/dbmi.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/raster3d.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/arraystats.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/cluster.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vedit.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/segment.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/rowio.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/temporal.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ogsf.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/nviz.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/__init__.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ctypes_preamble.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ctypes_loader.py
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/date.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/gis.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/raster.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/gmath.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/proj.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/imagery.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/rtree.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/display.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/stats.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/dbmi.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/raster3d.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/arraystats.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/cluster.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vedit.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/segment.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/rowio.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/temporal.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ogsf.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/nviz.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/__init__.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ctypes_preamble.pyc
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/ctypes_loader.pyc
make[1]: Entering directory
'/home/mneteler/software/grass77/lib/python/ctypes'
GISRC=/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/demolocation/.grassrc77
GISBASE=/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu
PATH="/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts:$PATH" PYTHONPATH="/home/mneteler/software/grass77/dist.x86_64
-pc-linux-gnu/etc/python:/home/mneteler/software/grass77/dist.x86_64-pc-
linux-gnu/gui/wxpython:$PYTHONPATH"
LD_LIBRARY_PATH="/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/scripts:/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/lib:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/lib:"
LC_ALL=C LANG=C LANGUAGE=C ./ctypesgen.py --cpp "gcc -E
-I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include
-I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include
-D__GLIBC_HAVE_LONG_LONG" -lgrass_datetime.7.7.svn
/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/include/grass/datetime.h /home/mneteler/software/grass77/dist.x86_64
-pc-linux-gnu/include/grass/defs/datetime.h -o OBJ.x86_64-pc-linux-
gnu/date.py
Status: Preprocessing /home/mneteler/tmp/tmpzwarkrsu.h
Status: gcc -E -I/home/mneteler/software/grass77/dist.x86_64-pc-
linux-gnu/include -I/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/include -D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline="
"-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const"
"-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1"
"/home/mneteler/tmp/tmpzwarkrsu.h"
Traceback (most recent call last):
   File "/usr/lib64/python3.7/sre_parse.py", line 1021, in parse_template
     this = chr(ESCAPES[this][1])
KeyError: '\\x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "./ctypesgen.py", line 162, in <module>
     descriptions = ctypesgencore.parser.parse(options.headers, options)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/__init__.py",
line 22, in parse
     parser.parse()
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py",
line 74, in parse
     ctypesparser.CtypesParser.parse(self, fname, False)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/cparser.py",
line 122, in parse
     self.preprocessor_parser.parse(filename)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/preprocessor.py",
line 222, in parse
     token = self.lexer.token()
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/preprocessor.py",
line 63, in token
     result = lex.Lexer.token(self)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/lex.py",
line 355, in token
     newtok = func(tok)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/pplexer.py",
line 284, in t_ANY_string_literal
     t.value = StringLiteral(t.value)
   File
"/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/pplexer.py",
line 82, in __new__
     r'\x0\1', value[1:-1])
   File "/usr/lib64/python3.7/re.py", line 192, in sub
     return _compile(pattern, flags).sub(repl, string, count)
   File "/usr/lib64/python3.7/re.py", line 309, in _subx
     template = _compile_repl(template, pattern)
   File "/usr/lib64/python3.7/re.py", line 300, in _compile_repl
     return sre_parse.parse_template(repl, pattern)
   File "/usr/lib64/python3.7/sre_parse.py", line 1024, in parse_template
     raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \x at position 0
make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1
make[1]: Leaving directory
'/home/mneteler/software/grass77/lib/python/ctypes'
make: *** [Makefile:81: default] Error 2
}}}

Any idea how to fix that?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-----------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: ctypes, python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-----------------------------

Comment (by martinl):

Python 3.7 only related, see
https://trac.osgeo.org/grass/ticket/3446#comment:12

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------
Changes (by martinl):

* keywords: ctypes, python3 => ctypes, python3.7

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------

Comment (by neteler):

Inspired by

  * bug: https://sourceforge.net/p/sbml/libsbml/461/
    * --> fix: https://sourceforge.net/p/sbml/code/25372/
  * and fix:
https://github.com/wbsoft/frescobaldi/commit/c27a9c24249cb19cbf99de7d7bfc46c561421539

... I have created attached patch attempt to fix the compilation issue
with Python-3.7.
Seems to work but testing/checks before applying is needed.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------
Changes (by neteler):

* Attachment "ctypesgencore_pplexer.diff" added.

Patch to fix ctypesgencore pplexer.py compilation error with Python3.7

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------
Changes (by martinl):

* status: new => closed
* resolution: => fixed

Comment:

In [changeset:"73827" 73827]:
{{{
#!CommitTicketReference repository="" revision="73827"
ctypes compilation error with Python-3.7, closes #3704
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------

Comment (by martinl):

I took liberty to apply the patch to trunk. It's working for me.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3704: ctypes compilation error with Python-3.7
--------------------------+-------------------------------
  Reporter: neteler | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: ctypes, python3.7
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------------

Comment (by hellik):

Replying to [comment:5 martinl]:
> I took liberty to apply the patch to trunk. It's working for me.

tested winGRASS compilation with Python 3.7.2, seems to work here too.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3704#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>