[GRASS-dev] [GRASS GIS] #3773: shutil_which() throws an exception on Linux + Python 3

GRASS GRASS developer
1

#3773: shutil_which() throws an exception on Linux + Python 3
-------------------------+-------------------------
Reporter: pmav99 | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone:
Component: Python | Version: svn-trunk
Keywords: | CPU: Unspecified
Platform: Unspecified |
-------------------------+-------------------------
{{{
Traceback (most recent call last):
   File "raster/r.report/testsuite/test_r_report.py", line 68, in
test_output
     self.assertModule('r.report', map='lakes', output=self.outfile)
   File "/home/feanor/Prog/git/grass-p3/repo/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/case.py", line 1145, in assertModule
     if not shutil_which(module.name):
   File "/home/feanor/Prog/git/grass-p3/repo/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/core.py", line 225, in shutil_which
     name = os.path.join(encode(dir), thefile)
   File "/usr/lib/python3.7/posixpath.py", line 94, in join
     genericpath._check_arg_types('join', a, *p)
   File "/usr/lib/python3.7/genericpath.py", line 151, in _check_arg_types
     raise TypeError("Can't mix strings and bytes in path components") from
None
TypeError: Can't mix strings and bytes in path components
}}}

This was introduced at https://trac.osgeo.org/grass/changeset/73930

A possible resolution could be:

{{{
import six

if six.PY3:
     shutil_which = shutil.which
else:
     def shutil_which(...):
         # ...
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773&gt;
GRASS GIS <https://grass.osgeo.org>

2

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by neteler):

* keywords: => python3
* milestone: => 7.8.0

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

3

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by pmav99):

{{{shutil_which}}} is being used by {{{assertModule}}} which practically
means that it is not currently possible to run {{{gunnitest}}} based tests
on Python 3.

Please prioritize this.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

4

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by neteler):

Replying to [ticket:3773 pmav99]:
> A possible resolution could be:
>
> {{{
> import six
>
> if six.PY3:
> shutil_which = shutil.which
> else:
> def shutil_which(...):
> # ...
> }}}

Could you please provide a patch? I can also grab it from github if that's
easier for you.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

5

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by pmav99):

That's the least invasive patch I can think off:

{{{
Index: lib/python/script/core.py

--- lib/python/script/core.py (revision 74173)
+++ lib/python/script/core.py (working copy)
@@ -164,7 +164,7 @@

      return set(cmd), scripts

-
+# TODO: Please replace this function with shutil.which() before 8.0 comes
out
  # replacement for which function from shutil (not available in all
versions)
  # from http://hg.python.org/cpython/file/6860263c05b3/Lib/shutil.py#l1068
  # added because of Python scripts running Python scripts on MS Windows
@@ -239,6 +239,8 @@
                      return name
      return None

+if sys.version_info.major > 2:
+ shutil_which = shutil.which

  # Added because of scripts calling scripts on MS Windows.
  # Module name (here cmd) differs from the file name (does not have
extension).
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

6

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by pmav99):

If there are not any problems with the patch, please apply it. It is
really needed when testing with Python 3.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

7

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by neteler):

* status: new => closed
* resolution: => fixed

Comment:

In [changeset:"74225" 74225]:
{{{
#!CommitTicketReference repository="" revision="74225"
python script core: shutil_which() exception fix for Python 3 (fixes
#3773)
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

8

#3773: shutil_which() throws an exception on Linux + Python 3
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: defect | Status: reopened
  Priority: normal | Milestone: 8.0.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: python3
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by neteler):

* status: closed => reopened
* resolution: fixed =>
* milestone: 7.8.0 => 8.0.0

Comment:

Reopened with G8.0.0 milestone to remove TODO of r74225 in
lib/python/script/core.py

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3773#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>