[GRASS-dev] [GRASS GIS] #3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
-----------------------+-------------------------
Reporter: pmav99 | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone:
Component: Compiling | Version: svn-trunk
Keywords: | CPU: x86-64
Platform: Linux |
-----------------------+-------------------------
On a clean installation in a new VM, after checking out the SVN repo and
compiling with python 3.6, `svn diff` shows that the following file has
changed. On the exact same setup using python 2.7 `svn diff` shows
nothing.

If this file is being generated then shouldn't it be ".svnignored" (or
whatever that is called on SVN).

This is probably related to #3704 too

{{{
Index: lib/python/ctypes/ctypesgencore/parser/lextab.py

--- lib/python/ctypes/ctypesgencore/parser/lextab.py (revision 74039)
+++ lib/python/ctypes/ctypesgencore/parser/lextab.py (working copy)
@@ -1,8 +1,8 @@
  # lextab.py. This file automatically created by PLY (version 2.2). Don't
edit!
-_lextokens = {'PP_END_DEFINE': None, 'DIV_ASSIGN': None, 'AND_OP':
None, 'PERIOD': None, 'PTR_OP': None, 'PP_MACRO_PARAM': None, 'OTHER':
None, 'IDENTIFIER': None, 'PP_STRINGIFY': None, 'XOR_ASSIGN': None,
'PP_DEFINE_NAME': None, 'ELLIPSIS': None, 'INC_OP': None, 'GE_OP': None,
'LEFT_ASSIGN': None, 'MUL_ASSIGN': None, 'RIGHT_ASSIGN': None,
'SUB_ASSIGN': None, 'LPAREN': None, 'PP_IDENTIFIER_PASTE': None,
'PP_DEFINE': None, 'OR_ASSIGN': None, 'ADD_ASSIGN': None, 'HEADER_NAME':
None, 'LE_OP': None, 'LEFT_OP': None, 'PP_NUMBER': None, 'DEC_OP': None,
'RIGHT_OP': None, 'MOD_ASSIGN': None, 'STRING_LITERAL': None, 'EQ_OP':
None, 'OR_OP': None, 'AND_ASSIGN': None, 'CHARACTER_CONSTANT': None,
'PP_DEFINE_MACRO_NAME': None, 'NE_OP': None, 'NEWLINE': None}
+_lextokens = {'HEADER_NAME': None, 'IDENTIFIER': None, 'PP_NUMBER':
None, 'CHARACTER_CONSTANT': None, 'STRING_LITERAL': None, 'OTHER': None,
'PTR_OP': None, 'INC_OP': None, 'DEC_OP': None, 'LEFT_OP': None,
'RIGHT_OP': None, 'LE_OP': None, 'GE_OP': None, 'EQ_OP': None, 'NE_OP':
None, 'AND_OP': None, 'OR_OP': None, 'MUL_ASSIGN': None, 'DIV_ASSIGN':
None, 'MOD_ASSIGN': None, 'ADD_ASSIGN': None, 'SUB_ASSIGN': None,
'LEFT_ASSIGN': None, 'RIGHT_ASSIGN': None, 'AND_ASSIGN': None,
'XOR_ASSIGN': None, 'OR_ASSIGN': None, 'PERIOD': None, 'ELLIPSIS': None,
'LPAREN': None, 'NEWLINE': None, 'PP_DEFINE': None, 'PP_DEFINE_NAME':
None, 'PP_DEFINE_MACRO_NAME': None, 'PP_MACRO_PARAM': None,
'PP_STRINGIFY': None, 'PP_IDENTIFIER_PASTE': None, 'PP_END_DEFINE': None}
  _lexreflags = 0
  _lexliterals = ''
-_lexstateinfo = {'DEFINE': 'exclusive', 'INITIAL': 'inclusive'}
-_lexstatere = {'DEFINE':
[('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[
\\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))',
[None, ('t_ANY_directive', 'ANY_directive'), None, None,
('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-
Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier',
'DEFINE_identifier')]),
('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))',
[None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]),
('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))',
[None, ('t_ANY_int', 'ANY_int'), None, None]),
('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))',
[None, ('t_ANY_character_constant', 'ANY_character_constant'), None,
('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen',
'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'),
('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])], 'INITIAL':
[('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[
\\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))',
[None, ('t_ANY_directive', 'ANY_directive'), None, None,
('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a
-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier',
'INITIAL_identifier')]),
('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))',
[None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]),
('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))',
[None, ('t_ANY_int', 'ANY_int'), None, None]),
('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)',
[None, ('t_ANY_character_constant', 'ANY_character_constant'), None,
('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen',
'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'),
('t_INITIAL_pp_define', 'INITIAL_pp_define')])]}
-_lexstateignore = {'DEFINE': ' \t\x0b\x0c\r', 'INITIAL': ' \t\x0b\x0c\r'}
-_lexstateerrorf = {'DEFINE': 't_DEFINE_error', 'INITIAL':
't_INITIAL_error'}
+_lexstateinfo = {'INITIAL': 'inclusive', 'DEFINE': 'exclusive'}
+_lexstatere = {'INITIAL':
[('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[
\\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))',
[None, ('t_ANY_directive', 'ANY_directive'), None, None,
('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a
-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier',
'INITIAL_identifier')]),
('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))',
[None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]),
('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))',
[None, ('t_ANY_int', 'ANY_int'), None, None]),
('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)',
[None, ('t_ANY_character_constant', 'ANY_character_constant'), None,
('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen',
'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'),
('t_INITIAL_pp_define', 'INITIAL_pp_define')])], 'DEFINE':
[('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[
\\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))',
[None, ('t_ANY_directive', 'ANY_directive'), None, None,
('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-
Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier',
'DEFINE_identifier')]),
('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))',
[None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]),
('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))',
[None, ('t_ANY_int', 'ANY_int'), None, None]),
('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))',
[None, ('t_ANY_character_constant', 'ANY_character_constant'), None,
('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen',
'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'),
('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])]}
+_lexstateignore = {'INITIAL': ' \t\x0b\x0c\r', 'DEFINE': ' \t\x0b\x0c\r'}
+_lexstateerrorf = {'INITIAL': 't_INITIAL_error', 'DEFINE':
't_DEFINE_error'}
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone:
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------

Comment (by pmav99):

As far as `_lextokens` go, the only difference is the key ordering. So,
this might has to do with Python's dictionaries keeping the "insertion
order" in Python 3.6+. `_lexstatere` has different values too, though.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------
Changes (by martinl):

* milestone: => 7.8.0

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------

Comment (by neteler):

Issue confirmed also on Fedora 29 with Python-3.7

@pmav99: do you suggest to commit the modified lextab.py since it mainly
comes with a different order?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------

Comment (by pmav99):

Not really. I don't have much experience with ctypes, but I am under the
impression that this will file will be different under each python version
(2.7 vs 3.6/3.7). Maybe even under different minor versions too but I
haven't checked it.

So if it gets committed for Python 3, I guess that there is going to be a
dirty `svn diff` under Python 2.

Furthermore, since this file is clearly changing when GRASS is compiled, I
am not really sure why it is added to the repo, instead of getting
generated upon compilation.

Finally, there have already been 2 instances where the file was
accidentally added to the repo, only to be removed right away:
https://trac.osgeo.org/grass/log/grass/trunk/lib/python/ctypes/ctypesgencore?rev=74047

So, before deciding on a course of action, I think it is better to further
investigate how it is being used/generated. This might provide some
insight: #3331

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------

Comment (by pmav99):

Just checked with 3.6 vs 3.7. There is no difference between them (i.e.
they produce the same `svn diff`).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords:
       CPU: x86-64 | Platform: Linux
------------------------+-------------------------

Comment (by pmav99):

For the record, this seems to be a maintained ctypesgen fork:
https://github.com/olsonse/ctypesgen

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-----------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: ctypes, python3
       CPU: x86-64 | Platform: Linux
------------------------+-----------------------------
Changes (by neteler):

* keywords: => ctypes, python3

Comment:

Replying to [comment:6 pmav99]:
> For the record, this seems to be a maintained ctypesgen fork:
https://github.com/olsonse/ctypesgen

It is worth trying? Did anyone do so?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+-----------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: svn-trunk
Resolution: | Keywords: ctypes, python3
       CPU: x86-64 | Platform: Linux
------------------------+-----------------------------

Comment (by neteler):

Replying to [comment:6 pmav99]:
> For the record, this seems to be a maintained ctypesgen fork:
https://github.com/olsonse/ctypesgen

See related #3900.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

#3739: lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.
------------------------+---------------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Compiling | Version: git-releasebranch78
Resolution: fixed | Keywords: ctypes, python3
       CPU: x86-64 | Platform: Linux
------------------------+---------------------------------
Changes (by neteler):

* status: new => closed
* version: svn-trunk => git-releasebranch78
* resolution: => fixed

Comment:

Meanwhile `lib/python/ctypes/ctypesgencore/parser/lextab.py` has been
removed from the source code and thing now work.

Closing.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3739#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>