[GRASS-dev] [GRASS GIS] #2748: ctype.POINTER function overloading prevents automatic byref() conversion

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
---------------------------+-------------------------
Reporter: artegion | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Keywords: | CPU: Unspecified
Platform: Unspecified |
---------------------------+-------------------------
From "https://docs.python.org/2/library/ctypes.html#structured-data-types"
''"In addition, if a function argument is explicitly declared to be a
pointer type (such as POINTER(c_int)) in argtypes, an object of the
pointed type (c_int in this case) can be passed to the function.
ctypes will apply the required byref() conversion in this case
automatically."''

grass.lib module overwrites ctype.POINTER function in ctypes_preamble
module.
Consequently function arguments types are not recognized:
  i.e. Vect_list_append.argtypes reports

{{{
<class 'grass.lib.ctypes_preamble.LP_struct_ilist'>
}}}

  instead of

{{{
<class 'grass.lib.vector.LP_struct_ilist'>
}}}

and the required byref() conversion does't happen but ctpyes.byref
explicit call is needed

example:
{{{
>>> from ctypes import byref
>>> from grass.lib.vector import struct_ilist, Vect_list_append
>>> a= struct_ilist()
>>> a.__class__
<class 'grass.lib.vector.struct_ilist'>
>>> Vect_list_append.argtypes
[<class 'grass.lib.ctypes_preamble.LP_struct_ilist'>, <class
'ctypes.c_long'>]
>>>
>>>
>>> if Vect_list_append(a, 1):
... print "direct call failed"
... else:
... print a.n_values, a.value[a.n_values-1]
...
direct call failed
>>>
>>>
>>> if Vect_list_append(byref(a), 2):
... print "byref call failed"
... else:
... print a.n_values, a.value[a.n_values-1]
...
1 2
>>>
}}}

Using a modified version of grass.lib.vector (added
{{{
from ctypes import POINTER
}}}

after
{{{
from ctypes_preamble import *
}}}
everything seems fine...

{{{
>>> from ctypes import byref
>>> from grassmod.lib.vector import struct_ilist, Vect_list_append
>>> a= struct_ilist()
>>> a.__class__
<class 'grassmod.lib.vector.struct_ilist'>
>>> Vect_list_append.argtypes
[<class 'grassmod.lib.vector.LP_struct_ilist'>, <class 'ctypes.c_long'>]
>>>
>>>
>>> if Vect_list_append(a, 1):
... print "direct call failed"
... else:
... print a.n_values, a.value[a.n_values-1]
...
1 1
>>>
>>>
>>> if Vect_list_append(byref(a), 2):
... print "byref call failed"
... else:
... print a.n_values, a.value[a.n_values-1]
...
2 2
>>>
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by glynn):

Replying to [ticket:2748 artegion]:

> grass.lib module overwrites ctype.POINTER function in ctypes_preamble
module.

The replacement POINTER function has the following comment:
{{{
     # Convert None to a real NULL pointer to work around bugs
     # in how ctypes handles None on 64-bit platforms
}}}
Have you tested whether this is still an issue (it might have been fixed
since ctypesgen was last updated)?

Having to use byref() explicitly seems less of an issue than not working
on 64-bit systems.

Actually, if an argument is a pointer, the caller should probably be using
byref() explicitly regardless of whether or not ctypes has a workaround.
What's the problem with needing to use byref()?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by artegion):

It seems a known bug of ctypesgen
[https://github.com/davidjamesca/ctypesgen/issues/26\]

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by artegion):

Not a real problem, just a bit disappointing: python ctypes documentation
describes an interesting feature but our code presents a different
behavior (it is far more pythonic do not care about byreference/byvalue if
ctypes can do the dirty work).

The odd thing is that happens unnoticed: while ctypes raises exceptions
passing wrong argument type in this case only function return code (or
segmentation fault exception) reveals troubles.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by artegion):

Replying to [comment:1 glynn]:
> Replying to [ticket:2748 artegion]:
>
> > grass.lib module overwrites ctype.POINTER function in ctypes_preamble
module.
>
> The replacement POINTER function has the following comment:
> {{{
> # Convert None to a real NULL pointer to work around bugs
> # in how ctypes handles None on 64-bit platforms
> }}}
> Have you tested whether this is still an issue (it might have been fixed
since ctypesgen was last updated)?
>
> Having to use byref() explicitly seems less of an issue than not working
on 64-bit systems.
>
> Actually, if an argument is a pointer, the caller should probably be
using byref() explicitly regardless of whether or not ctypes has a
workaround. What's the problem with needing to use byref()?

Not a real problem, just a bit disappointing: python ctypes documentation
describes an interesting feature but our code presents a different
behavior (it is far more pythonic do not care about byreference/byvalue if
ctypes can do the dirty work).

The odd thing is that happens unnoticed: while ctypes raises exceptions
passing wrong argument type in this case only function return code (or
segmentation fault exception) reveals troubles.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by glynn):

Replying to [comment:2 artegion]:
> It seems a known bug of ctypesgen
> [https://github.com/davidjamesca/ctypesgen/issues/26\]

That suggests that it's working around an issue which is specific to
Python 2.5. Do we still support that? If not, the workaround can be
removed.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.1
Component: Python ctypes | Version: 7.0.1
Resolution: | Keywords:
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------
Changes (by neteler):

* milestone: => 7.0.1

Comment:

Replying to [comment:5 glynn]:
> That suggests that it's working around an issue which is specific to
Python 2.5. Do we still support that? If not, the workaround can be
removed.

According to
https://svn.osgeo.org/grass/grass/branches/releasebranch_7_0/REQUIREMENTS.html
we support Python >= 2.6

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.3
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: ctypes
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------
Changes (by neteler):

* keywords: => ctypes
* version: 7.0.1 => svn-trunk
* milestone: 7.0.1 => 7.0.3

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Python ctypes | Version: svn-trunk
Resolution: | Keywords: ctypes
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------

Comment (by neteler):

Replying to [comment:5 glynn]:
> Replying to [comment:2 artegion]:
> > It seems a known bug of ctypesgen
> > [https://github.com/davidjamesca/ctypesgen/issues/26\]
>
> That suggests that it's working around an issue which is specific to
Python 2.5. Do we still support that? If not, the workaround can be
removed.

Since we require at least Python >= 2.6, let's remove the workaround...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#2748: ctype.POINTER function overloading prevents automatic byref() conversion
----------------------------+-------------------------
  Reporter: artegion | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.6.2
Component: Python ctypes | Version: svn-trunk
Resolution: worksforme | Keywords: ctypes
       CPU: Unspecified | Platform: Unspecified
----------------------------+-------------------------
Changes (by martinl):

* status: new => closed
* resolution: => worksforme
* milestone: 7.0.7 => 7.6.2

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2748#comment:14&gt;
GRASS GIS <https://grass.osgeo.org>