[GRASS-dev] [GRASS GIS] #3731: winGRASS7.7.svn - ctypes - python 3 issues

#3731: winGRASS7.7.svn - ctypes - python 3 issues
------------------------------------+-------------------------
Reporter: martinl | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Keywords: wingrass, python3, py3 | CPU: Unspecified
Platform: Unspecified |
------------------------------------+-------------------------
WinGRASS daily builds no. 372+ fails with

{{{
   File "C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\nviz\wxnviz.py",
line
53, in <module>
     from grass.lib.vector import *
   File
"C:\OSGEO4~1\apps\grass\grass-7.7.svn\etc\python\grass\lib\vector.py", li
ne 340
     SF_POINT25D = 2147483649L #
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mi
ngw32/include/grass/vect/dig_defines.h: 257
                             ^
SyntaxError: invalid syntax
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Comparing `vector.py` file compiled on

Windows:

{{{
'''Wrapper for vector.h

Generated with:
./ctypesgen.py --cpp gcc -E -I/c/OSGeo4W64/include -D_FILE_OFFSET_BITS=64
-I/c/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include
-I/c/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include
-D__GLIBC_HAVE_LONG_LONG -lgrass_vector.7.7.svn -IC:/OSGeo4W64/include
-IC:/OSGeo4W64/include
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/vector.h
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/defs/vector.h
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/vect/dig_structs.h
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/vect/dig_defines.h
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/vect/dig_externs.h
-o OBJ.x86_64-w64-mingw32/vector.py

Do not modify this file.
'''
...
SF_POINT25D = 2147483649L #
C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/include/grass/vect/dig_defines.h:
257
...
}}}

and Linux

{{{
'''Wrapper for vector.h

Generated with:
./ctypesgen.py --cpp gcc -E -I/opt/src/grass7_trunk/dist.x86_64-pc-
linux-gnu/include -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include
-D__GLIBC_HAVE_LONG_LONG -lgrass_vector.7.7.svn -I/usr/include/postgresql
-I/usr/include/gdal /opt/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/include/grass/vector.h /opt/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/include/grass/defs/vector.h /opt/src/grass7_trunk/dist.x86_64-pc-
linux-gnu/include/grass/vect/dig_structs.h
/opt/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/include/grass/vect/dig_defines.h /opt/src/grass7_trunk/dist.x86_64-pc-
linux-gnu/include/grass/vect/dig_externs.h -o OBJ.x86_64-pc-linux-
gnu/vector.py

Do not modify this file.
'''
...

SF_POINT25D = 2147483649 # /opt/src/grass7_trunk/dist.x86_64-pc-linux-
gnu/include/grass/vect/dig_defines.h: 257
...
}}}

Any idea how to avoid `L` suffix?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by neteler):

---------- Forwarded message ---------
From: Anna Petrášová <kratochanna@gmail.com>
Date: Do., 13. Sep. 2018, 22:30
Subject: Re: [GRASS-user] experimental Python 3 support in trunk

On Tue, Sep 4, 2018 at 2:00 AM Markus Neteler <neteler@osgeo.org> wrote:
On Tue, Sep 4, 2018 at 12:39 AM Anna Petrášová <kratochanna@gmail.com>
wrote:

{{{
     from grass.pygrass.vector import VectorTopo
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/vector/__init__.py",
line 7, in <module>
     import grass.lib.vector as libvect
   File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.py",
line 5863
     PORT_LONG_MAX = 2147483647L
                               ^
SyntaxError: invalid syntax

}}}

regarding this issue, make sure you compile GRASS with Python 3. I get
this if I compile it with Python2 and then open it with Python 3.

Anna

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by hellik):

Replying to [comment:2 neteler]:
>
>
> ---------- Forwarded message ---------
> From: Anna Petrášová <kratochanna@gmail.com>
> Date: Do., 13. Sep. 2018, 22:30
> Subject: Re: [GRASS-user] experimental Python 3 support in trunk
>
>
> On Tue, Sep 4, 2018 at 2:00 AM Markus Neteler <neteler@osgeo.org> wrote:
> On Tue, Sep 4, 2018 at 12:39 AM Anna Petrášová <kratochanna@gmail.com>
wrote:
>
> {{{
> from grass.pygrass.vector import VectorTopo
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/vector/__init__.py",
> line 7, in <module>
> import grass.lib.vector as libvect
> File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-
gnu/etc/python/grass/lib/vector.py",
> line 5863
> PORT_LONG_MAX = 2147483647L
> ^
> SyntaxError: invalid syntax
>
> }}}
>
> regarding this issue, make sure you compile GRASS with Python 3. I get
this if I compile it with Python2 and then open it with Python 3.
>
> Anna

for compilation in package.sh we do:

{{{
# Force Python3
alias python=python3
}}}

adding later in the script e.g.

{{{
[...]
exec 3>&1 >> mswindows/osgeo4w/package.log 2>&1
python --version
[...]
}}}

I get

{{{
Python 3.7.2
Sat Jan 12 15:29:17 CET 2019: STARTING dll.to.a
/c/OSGeo4W64/bin/proj_5_2.dll => mswindows/osgeo4w/lib/libproj
[...]
}}}

it seems python3 is used for compilation.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

In [changeset:"73932" 73932]:
{{{
#!CommitTicketReference repository="" revision="73932"
attempt to fix ctypes compilation, force python3, see #3731
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Replying to [comment:3 hellik]:

{{{
> alias python=python3
}}}

Defining `alias` does not seems to be enough. Hopefully fixed in r73932.
Let's wait for build no. 373+.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn - ctypes - python 3 issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Unfortunately last build is still no. 372. There is a new issue with
building manual pages on build server

{{{
Traceback (most recent call last):
   File
"C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/tools/g.html2man.py",
line 77, in <module>
     main()
   File
"C:/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/tools/g.html2man.py",
line 73, in main
     outf.write(s)
   File "C:/OSGeo4W64/apps/Python37\lib\encodings\cp1250.py", line 19, in
encode
     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2264' in
position 2582: character maps to <undefined>
../../include/Make/Html.make:11: recipe for target
'/c/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/docs/man/man1/r.clump.1'
failed
make[4]: ***
[/c/msys64/usr/src/grass_trunk/dist.x86_64-w64-mingw32/docs/man/man1/r.clump.1]
Error 1
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

In [changeset:"73939" 73939]:
{{{
#!CommitTicketReference repository="" revision="73939"
attempt to fix g.html2man wingrass python3 issue, see #3731
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

In [changeset:"73940" 73940]:
{{{
#!CommitTicketReference repository="" revision="73940"
fix r73939 for python2, see #3731
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Build no. 375 fails with new issues

{{{
Traceback (most recent call last):
   File "C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\wxgui.py", line
103, in OnInit
     workspace=self.workspaceFile)
   File "C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\lmgr\frame.py",
line 131, in __init__
     self._moduleTreeBuilder =
LayerManagerModuleTree(message_handler=add_menu_error)
   File
"C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\lmgr\menudata.py", line
69, in __init__
     message_handler=message_handler)
   File
"C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\core\menutree.py", line
72, in __init__
     xmlTree = etree.parse(filename)
   File "C:\OSGEO4~1\apps\Python37\lib\xml\etree\ElementTree.py", line
1197, in parse
     tree.parse(source, parser)
   File "C:\OSGEO4~1\apps\Python37\lib\xml\etree\ElementTree.py", line 598,
in parse
     self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
}}}

Reason:
`C:\OSGEO4~1\apps\grass\grass-7.7.svn\gui\wxpython\xml\module_tree_menudata.xml`
is empty.

Related traceback:

{{{
Traceback (most recent call last):
   File "core/toolboxes.py", line 914, in <module>
     sys.exit(main())
   File "core/toolboxes.py", line 898, in main
     userDefined=False)
   File "core/toolboxes.py", line 307, in createTree
     moduleItems=moduleItems)
   File "core/toolboxes.py", line 344, in toolboxes2menudata
     _expandRuntimeModules(root)
   File "core/toolboxes.py", line 645, in _expandRuntimeModules
     desc, keywords = _loadMetadata(name)
   File "core/toolboxes.py", line 682, in _loadMetadata
     task = gtask.parse_interface(module)
   File
"C:\msys64\usr\src\grass_trunk\dist.x86_64-w64-mingw32\etc\python\grass\script\task.py",
line 524, in parse_interface
     tree = etree.fromstring(get_interface_description(name))
   File
"C:\msys64\usr\src\grass_trunk\dist.x86_64-w64-mingw32\etc\python\grass\script\task.py",
line 476, in get_interface_description
     stderr=PIPE)
   File
"C:\msys64\usr\src\grass_trunk\dist.x86_64-w64-mingw32\etc\python\grass\script\core.py",
line 66, in __init__
     cmd = shutil_which(args[0])
   File
"C:\msys64\usr\src\grass_trunk\dist.x86_64-w64-mingw32\etc\python\grass\script\core.py",
line 222, in shutil_which
     if any(cmd.lower().endswith(ext) for ext in pathext):
   File
"C:\msys64\usr\src\grass_trunk\dist.x86_64-w64-mingw32\etc\python\grass\script\core.py",
line 222, in <genexpr>
     if any(cmd.lower().endswith(ext) for ext in pathext):
TypeError: endswith first arg must be str or a tuple of str, not bytes
Makefile:46: recipe for target 'xml/module_tree_menudata.xml' failed
make[4]: *** [xml/module_tree_menudata.xml] Error 1
}}}

Seems to be related to r73930 (#3723)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

I added some debug messages around. The `thumbnails.py` scripts calls
before `g.region` command the `g.gisenv` command(but how and where?). The
problem is that once `cmd` is a string (should be bytes, if I am right(?)
and then bytes.

{{{
g.gisenv <class 'str'>
b'g.region' <class 'bytes'>
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:11&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

In [changeset:"73945" 73945]:
{{{
#!CommitTicketReference repository="" revision="73945"
grass.core.find_program(): force bytes, see #3731
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:12&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Replying to [comment:11 martinl]:
the `g.gisenv` command(but how and where?). The problem is that once `cmd`
is a string (should be

it was `grass.core.find_program()`, hopefully fixed in r73945. Let's wait
for next build (no. 376+).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:13&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

In [changeset:"73956" 73956]:
{{{
#!CommitTicketReference repository="" revision="73956"
attempt to fix grass.task.get_interface python3 issue, see #3731
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:14&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------
Changes (by martinl):

* Attachment "wingrass_python3.png" added.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Great, with build no. 376 GUI starts, let's celebrate :slight_smile:

[[Image(wingrass_python3.png, 500px)]]

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:15&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by hellik):

Replying to [comment:15 martinl]:
> Great, with build no. 376 GUI starts, let's celebrate :slight_smile:
>
> [[Image(wingrass_python3.png, 500px)]]

great!

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:16&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------

Comment (by martinl):

Reported issue seems to be solved, closing. Feel free to reopen if needed.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:17&gt;
GRASS GIS <https://grass.osgeo.org>

#3731: winGRASS7.7.svn python3 build issues
----------------------------+------------------------------------
  Reporter: martinl | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python ctypes | Version: svn-trunk
Resolution: fixed | Keywords: wingrass, python3, py3
       CPU: Unspecified | Platform: Unspecified
----------------------------+------------------------------------
Changes (by martinl):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3731#comment:18&gt;
GRASS GIS <https://grass.osgeo.org>