[GRASS-dev] [GRASS GIS] #3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes

GRASS GIS GRASS GIS developer
1

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------------+-------------------------
Reporter: AnikaBettge | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Keywords: Python3, testsuite | CPU: x86-64
Platform: Linux |
--------------------------------+-------------------------
Tzhe multirunner in trunk currently fails with Python3:

{{{
(grasspy3) :heavy_check_mark: ~/src/grass-7.7.svn/testsuite/examples
11:41 $ bash test_framework_GRASS_GIS_with_NC.sh
test_framework_GRASS_GIS_with_NC_AB.conf
--2019-01-23 13:42:48--
https://grass.osgeo.org/sampledata/north_carolina/nc_spm_08_grass7.tar.gz
...

Starting GRASS GIS...
Cleaning up temporary files...
Executing <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh> ...
WARNUNG: <basin> already exists. File not copied.
WARNUNG: <boundary> already exists. File not copied.
...
Execution of <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh>
finished.
Cleaning up default sqlite database ...
Cleaning up temporary files...
Traceback (most recent call last):
   File
"/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py",
line 125, in <module>
     sys.exit(main())
   File
"/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py",
line 76, in main
     os.environ['GISBASE'] = gisbase
   File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py",
line 730, in __setitem__
     value = self.encodevalue(value)
   File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py",
line 798, in encode
     raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes
}}}

Version:

{{{
grass77 --config svn_revision
74007
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737&gt;
GRASS GIS <https://grass.osgeo.org>

2

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by pmav99):

In order to make `multirunner.py` run on Python 3 you need to apply
something like this.

{{{
@@ -64,13 +64,13 @@
      # we assume that GRASS GIS' start script is available and in the PATH
      # the shell=True is here because of MS Windows? (code taken from
wiki)
      startcmd = grass7bin + ' --config path'
      p = subprocess.Popen(startcmd, shell=True,
                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
      out, err = p.communicate()
      if p.returncode != 0:
          print("ERROR: Cannot find GRASS GIS 7 start script (%s):\n%s" %
(startcmd, err), file=sys.stderr)
          return 1
- gisbase = out.strip('\n')
+ gisbase = out.strip().decode("utf-8")

      # set GISBASE environment variable
      os.environ['GISBASE'] = gisbase
@@ -95,7 +95,7 @@
      gsetup.init(gisbase, gisdb, location, mapset)

      reports =
- for location, location_type in itertools.izip(locations,
locations_types):
+ for location, location_type in zip(locations, locations_types):
          # here it is quite a good place to parallelize
          # including also type to make it unique and preserve it for sure
          report = 'report_for_' + location + '_' + location_type
}}}

Specifying `\n` is probably redundant. `str.strip()` already strips that.
More specifically it strips all the characters contained in
`string.whitespace`.

Other than that, in python3 subprocess output is bytes. The
`.decode("utf-8")` call is converting the bytes to python3 strings which
are the equivalent of unicode strings in python2.

Note: the patch should work with python2 too, but I have not tested it.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

3

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by pmav99):

That being said, even after applying the patch the testrunner is still not
running as expected. More specifically all the tests are failing with this
message on `stderr.txt`:

{{{
.../stderr.txt
Unknown option: -3
usage: python3 [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
}}}

Python 2 does indeed have a `-3` switch but Python 3 does not. I am
looking but I can't find where this switch is added. If anyone has any
idea, please post.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

4

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by pmav99):

This is also needed
{{{
Index: lib/python/gunittest/invoker.py

--- lib/python/gunittest/invoker.py (revision 74025)
+++ lib/python/gunittest/invoker.py (working copy)
@@ -161,7 +161,7 @@
              # ignoring shebang line to use current Python
              # and also pass parameters to it
              # add also '-Qwarn'?
- p = subprocess.Popen([sys.executable, '-tt', '-3',
+ p = subprocess.Popen([sys.executable, '-tt',
                                    module.abs_file_path],
                                   cwd=cwd, env=env,
                                   stdout=stdout, stderr=stderr)
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

5

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by pmav99):

After applying the previous patches, the vast majority of the tests were
again failing due to trying to mix strings and bytes in `shutil_which()`.
The following patch fixes that and the tests can finally run. There are
still failing ones, but that's a different issue. I opted to add an
additional `decode()` call since it seems less likely to break anything.

{{{
Index: script/core.py

--- script/core.py (revision 74025)
+++ script/core.py (working copy)
@@ -234,7 +234,9 @@
          if not normdir in seen:
              seen.add(normdir)
              for thefile in files:
- name = os.path.join(encode(dir), thefile)
+ name = os.path.join(encode(dir), encode(thefile))
                  if _access_check(name, mode):
                      return name
      return None
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

6

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by annakrat):

Thanks for looking into it, I fixed the things in gunittest in r74032, but
I have still some local changes related to shutil_which in core.py I need
to test more first.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

7

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by pmav99):

Hello Anna, is it necessary to redefine `encode()` and `decode()`? They
are already defined in a bunch of places already...
{{{
$ ag -l 'def encode' lib/python

lib/python/script/utils.py
lib/python/ctypes/preamble.py
lib/python/ctypes/ctypesgencore/printer/preamble.py
}}}
The one in `script/utils.py` seems to be OK.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

8

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by annakrat):

Replying to [comment:6 pmav99]:
> Hello Anna, is it necessary to redefine `encode()` and `decode()`? They
are already defined in a bunch of places already...
> {{{
> $ ag -l 'def encode' lib/python
>
> lib/python/script/utils.py
> lib/python/ctypes/preamble.py
> lib/python/ctypes/ctypesgencore/printer/preamble.py
> }}}
> The one in `script/utils.py` seems to be OK.

True, but the library is not loaded at that point when I need to decode.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

9

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+--------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: svn-trunk
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+--------------------------------

Comment (by AnikaBettge):

{{{
Cleaning up default sqlite database ...
Cleaning up temporary files...
raster3d_lib_test from ./lib/raster3d failed (3 tests failed)
test_assertions from ./lib/python/gunittest failed (3 tests failed)
test_checkers from ./lib/python/gunittest failed (1 test failed)
test_gmodules from ./lib/python/gunittest failed (4 tests failed)
test_assertions_vect from ./lib/python/gunittest failed (7 tests failed)
test_assertions_rast3d from ./lib/python/gunittest failed (2 tests failed)
test_module_assertions from ./lib/python/gunittest failed (2 tests failed)
test_doctests from ./lib/python/gunittest failed (2 tests failed)
unittests_temporal_raster_conditionals from ./lib/python/temporal failed
(50 tests failed)
unittests_temporal_raster_conditionals_complement_else from
./lib/python/temporal failed (4 tests failed)
unittests_temporal_vector_algebra from ./lib/python/temporal failed (14
tests failed)
unittests_temporal_conditionals from ./lib/python/temporal failed (34
tests failed)
unittests_temporal_raster_algebra_grs from ./lib/python/temporal failed
(13 tests failed)
unittests_temporal_algebra_grs from ./lib/python/temporal failed (6 tests
failed)
unittests_temporal_raster_algebra from ./lib/python/temporal failed (80
tests failed)
unittests_temporal_raster3d_algebra from ./lib/python/temporal failed (4
tests failed)
unittests_temporal_raster_algebra_equal_ts from ./lib/python/temporal
failed (10 tests failed)
unittests_temporal_algebra from ./lib/python/temporal failed (39 tests
failed)
unittests_temporal_raster_algebra_spatial_topology from
./lib/python/temporal failed (12 tests failed)
test_doctests from ./lib/python/temporal failed (15 tests failed)
unittests_temporal_algebra_mixed_stds from ./lib/python/temporal failed (9
tests failed)
test_doctests from ./lib/python/script failed (1 test failed)
Traceback (most recent call last):
   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
     "__main__", mod_spec)
   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
     exec(code, run_globals)
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
     sys.exit(main())
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/main.py", line 174, in main
     results_dir=results_dir)
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/invoker.py", line 237, in run_in_location
     gisdbase=gisdbase, location=location)
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/invoker.py", line 193, in _run_test_module
     self._file_anonymizer.anonymize([stdout_path, stderr_path])
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/reporters.py", line 103, in anonymize
     replace_in_file(filename, path + path_end, '')
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/reporters.py", line 61, in replace_in_file
     for line in old_file:
   File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/codecs.py",
line 321, in decode
     (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 439:
invalid continuation byte
GRASS Location <other_location> does not exist in GRASS Database
</home/abettge/grassdata/tests-grassdata>
Traceback (most recent call last):
   File
"/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multireport.py",
line 28, in <module>
     from grass.gunittest.checkers import text_to_keyvalue
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/checkers.py", line 18, in <module>
     from grass.script.core import KeyValue
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/__init__.py", line 5, in <module>
     from .core import *
   File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-
gnu/etc/python/grass/script/core.py", line 38, in <module>
     gettext.install('grasslibs', os.path.join(os.getenv("GISBASE"),
'locale'))
   File
"/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/posixpath.py",
line 80, in join
     a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
}}}

grass77 --config svn_revision
74046M

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

10

#3737: Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
--------------------------+---------------------------------
  Reporter: AnikaBettge | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Python | Version: git-releasebranch78
Resolution: | Keywords: Python3, testsuite
       CPU: x86-64 | Platform: Linux
--------------------------+---------------------------------
Changes (by neteler):

* version: svn-trunk => git-releasebranch78

Comment:

State as of today:

{{{
g.version -rge
version=7.8.dev
date=2019
revision=726b55ea2
build_date=2019-09-06
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj4=5.2.0
gdal=2.3.2
geos=3.7.1
sqlite=3.26.0

bash test_framework_GRASS_GIS_with_NC.sh
test_framework_GRASS_GIS_with_NC.conf
--2019-09-06 22:11:34--
http://fatra.cnr.ncsu.edu/data/nc_spm_full_v2alpha.tar.gz
Resolving fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)... 152.1.72.43
Connecting to fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)|152.1.72.43|:80...
connected.
HTTP request sent, awaiting response... 416 Requested Range Not
Satisfiable

     The file is already fully retrieved; nothing to do.

+ echo 'Testing of GRASS GIS started: 2019-09-06-20-11'
+ '[' no = yes ']'
+ cd testreports/reports_for_date-2019-09-06-20-11
+ python
/home/mneteler/software/grass78_git/lib/python/gunittest/multirunner.py
--grassbin /home/mneteler/software/grass78_git/bin.x86_64-pc-linux-
gnu/grass78 --grasssrc /home/mneteler/software/grass78_git --grassdata
/home/mneteler/grassdata --location nc_spm_full_v2alpha --location-type nc
test_v_out_lidar from ./vector/v.out.lidar failed (2 tests failed)
decimation_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_filter from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_basic from ./vector/v.in.lidar failed (1 test failed)
mask_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_pdal_basic from ./vector/v.in.pdal failed (1 test failed)
test_v_in_pdal_filter from ./vector/v.in.pdal failed (1 test failed)
Traceback (most recent call last):
   File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
     "__main__", fname, loader, pkg_name)
   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
     exec code in run_globals
   File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
     sys.exit(main())
   File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/main.py", line 174, in main
     results_dir=results_dir)
   File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/invoker.py", line 271, in run_in_location
     gisdbase=gisdbase, location=location)
   File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python/grass/gunittest/invoker.py", line 218, in _run_test_module
     stdout_file.write(stdout)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
596-597: ordinal not in range(128)
+ export PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-
linux-gnu/etc/python:
+ PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python:
+ python
/home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py
--output summary_report 'reports_for_date-*/*'
Traceback (most recent call last):
   File
"/home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py",
line 28, in <module>
     from grass.gunittest.checkers import text_to_keyvalue
   File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-
gnu/etc/python/grass/__init__.py", line 21, in <module>
     _LOCALE_DIR = os.path.join(os.getenv("GISBASE"), 'locale')
   File "/usr/lib64/python2.7/posixpath.py", line 70, in join
     elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3737#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>