[GRASS-dev] Failing addon releasebranch_7_8 (Python 3.6) pipeline

Hi devs,

I regularly receive

releasebranch_7_8 (Python 3.6)
Process completed with exit code 1.

(e.g., https://github.com/OSGeo/grass-addons/actions/runs/675694073)

which is a bit confusing for me since we (I believe) already fixed the issues:

Executing <python3 -m grass.gunittest.main --grassdata /home/runner
--location nc_spm_full_v2alpha2 --location-type nc --min-success 60>
...
generate_manpage_images from ./vector/v.stream.order failed
test_stream_order from ./vector/v.stream.order failed (2 tests failed)
test_pygbif_import from ./vector/v.in.pygbif failed (3 tests failed)
test_v_db_pyupdate from ./vector/v.db.pyupdate failed (2 tests failed)
r3_forestfrag_trivial from ./raster3d/r3.forestfrag failed (1 test failed)
test_whatcsv from ./temporal/t.rast.whatcsv failed (1 test failed)
test_whataggr from ./temporal/t.rast.what.aggr failed (6 tests failed)
test_gpot from ./raster/r.green/r.green.gshp/libgshp failed
test_numbers from ./raster/r.sample.category failed
r_extract_test from ./raster/r.extract failed (2 tests failed)
r_object_geometry_test from ./raster/r.object.geometry failed (2 tests failed)
test_base_resolution from ./raster/r.in.pdal failed
r_forestfrag_trivial from ./raster/r.forestfrag failed (1 test failed)
r_forestfrag_xy from ./raster/r.forestfrag failed (1 test failed)
test_append from ./raster/r.learn.ml2 failed

Does anyone know why it fails in the releasebranch_7_8 (Python 3.6)
pipeline but works in releasebranch_7_8 (Python 3.8)?

thanks
Markus

On Mon, Mar 22, 2021 at 7:39 AM Markus Neteler <neteler@osgeo.org> wrote:

Executing <python3 -m grass.gunittest.main --grassdata /home/runner
–location nc_spm_full_v2alpha2 --location-type nc --min-success 60>

generate_manpage_images from ./vector/v.stream.order failed
test_stream_order from ./vector/v.stream.order failed (2 tests failed)
test_pygbif_import from ./vector/v.in.pygbif failed (3 tests failed)
test_v_db_pyupdate from ./vector/v.db.pyupdate failed (2 tests failed)
r3_forestfrag_trivial from ./raster3d/r3.forestfrag failed (1 test failed)
test_whatcsv from ./temporal/t.rast.whatcsv failed (1 test failed)
test_whataggr from ./temporal/t.rast.what.aggr failed (6 tests failed)
test_gpot from ./raster/r.green/r.green.gshp/libgshp failed
test_numbers from ./raster/r.sample.category failed
r_extract_test from ./raster/r.extract failed (2 tests failed)
r_object_geometry_test from ./raster/r.object.geometry failed (2 tests failed)
test_base_resolution from ./raster/r.in.pdal failed
r_forestfrag_trivial from ./raster/r.forestfrag failed (1 test failed)
r_forestfrag_xy from ./raster/r.forestfrag failed (1 test failed)
test_append from ./raster/r.learn.ml2 failed

Looking at all commits [1], it fails only sometimes although it seems really only in the releasebranch_7_8 (Python 3.6) combo. By comparing the lists of failing tests in Meld, I identified that the fluctuating test is r_forestfrag_xy from ./raster/r.forestfrag failed. Resolving that, or fixing any other failing test (test file to be exact) will stabilize the success rate above 60% and resolve this issue without lowering the rate or introducing separate success rates for each combo at this point.

Does anyone know why it fails in the releasebranch_7_8 (Python 3.6)
pipeline but works in releasebranch_7_8 (Python 3.8)?

This would be good of course good to investigate…

[1] https://github.com/OSGeo/grass-addons/commits/master

See also: https://github.com/OSGeo/grass-addons/pull/488

That does not really solve the issue at hand. Unstable tests seem very unfortunate to me. But the PR contributes to improved addon testing and increases success rate…

Cheers

Stefan

···

On Mon, Mar 22, 2021 at 7:39 AM Markus Neteler <neteler@osgeo.org> wrote:

Executing <python3 -m grass.gunittest.main --grassdata /home/runner
–location nc_spm_full_v2alpha2 --location-type nc --min-success 60>

generate_manpage_images from ./vector/v.stream.order failed
test_stream_order from ./vector/v.stream.order failed (2 tests failed)
test_pygbif_import from ./vector/v.in.pygbif failed (3 tests failed)
test_v_db_pyupdate from ./vector/v.db.pyupdate failed (2 tests failed)
r3_forestfrag_trivial from ./raster3d/r3.forestfrag failed (1 test failed)
test_whatcsv from ./temporal/t.rast.whatcsv failed (1 test failed)
test_whataggr from ./temporal/t.rast.what.aggr failed (6 tests failed)
test_gpot from ./raster/r.green/r.green.gshp/libgshp failed
test_numbers from ./raster/r.sample.category failed
r_extract_test from ./raster/r.extract failed (2 tests failed)
r_object_geometry_test from ./raster/r.object.geometry failed (2 tests failed)
test_base_resolution from ./raster/r.in.pdal failed
r_forestfrag_trivial from ./raster/r.forestfrag failed (1 test failed)
r_forestfrag_xy from ./raster/r.forestfrag failed (1 test failed)
test_append from ./raster/r.learn.ml2 failed

Looking at all commits [1], it fails only sometimes although it seems really only in the releasebranch_7_8 (Python 3.6) combo. By comparing the lists of failing tests in Meld, I identified that the fluctuating test is r_forestfrag_xy from ./raster/r.forestfrag failed. Resolving that, or fixing any other failing test (test file to be exact) will stabilize the success rate above 60% and resolve this issue without lowering the rate or introducing separate success rates for each combo at this point.

Does anyone know why it fails in the releasebranch_7_8 (Python 3.6)
pipeline but works in releasebranch_7_8 (Python 3.8)?

This would be good of course good to investigate…

[1] https://github.com/OSGeo/grass-addons/commits/master

On Wed, Mar 24, 2021 at 5:59 AM Stefan Blumentrath <Stefan.Blumentrath@nina.no> wrote:

Unstable tests seem very unfortunate to me.

Yes, should not happen. We have a problem with that from the beginning. Running tests many, many times seems necessary to even discover such a thing. Why that happens is still an open question. It was never really a visible issue which is changing now.

Hi,

I spent some time on this - installed Ubuntu 18.04 from docker and
installed GRASS GIS 7.8 and addons inside to run the tests. It comes
with Python 3.6:

Here what came up (would be easier to see this in GH Action directly
since it took me almost an hour to set up the container... ):

1)
------------------------------------------------
GRASS nc_spm_08_grass7/user1:testsuite > python test_pygbif_import.py

Traceback (most recent call last):
  File "/root/install/grass78/scripts/v.in.pygbif", line 841, in <module>
    sys.exit(main())
  File "/root/install/grass78/scripts/v.in.pygbif", line 600, in main
    print("alternatives={}".format(",".join(alternatives)))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe5' in
position 340: ordinal not in range(128)

F

FAIL: test_poa_taxon_match (__main__.TestPyGBIFImport)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/install/grass78/etc/python/grass/gunittest/case.py",
line 1149, in assertModule
    module.run()
  File "/root/install/grass78/etc/python/grass/pygrass/modules/interface/module.py",
line 771, in run
    self.wait()
grass.exceptions.CalledModuleError: Module run v.in.pygbif v.in.pygbif
taxa=Poa basisofrecord=ALL rank=species output=gbif_poa1 -g --v ended
with error
Process ended with non-zero return code 1. See the following errors:
b'Traceback (most recent call last):\n File
"/root/install/grass78/scripts/v.in.pygbif", line 841, in <module>\n
sys.exit(main())\n File "/root/install/grass78/scripts/v.in.pygbif",
line 600, in main\n
print("alternatives={}".format(",".join(alternatives)))\nUnicodeEncodeError:
\'ascii\' codec can\'t encode character \'\\xe5\' in position 340:
ordinal not in range(128)\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_pygbif_import.py", line 35, in test_poa_taxon_match
    self.assertModule(v_in_pygbif_match)
  File "/root/install/grass78/etc/python/grass/gunittest/case.py",
line 1164, in assertModule
    self.fail(self._formatMessage(msg, stdmsg))
AssertionError: Running <v.in.pygbif> module ended with non-zero return code (1)
Called: v.in_pygbif(output='gbif_poa1', taxa='Poa',
basisofrecord='ALL', rank='species', flags='g', verbose=True)
See the following errors:
Traceback (most recent call last):
  File "/root/install/grass78/scripts/v.in.pygbif", line 841, in <module>
    sys.exit(main())
  File "/root/install/grass78/scripts/v.in.pygbif", line 600, in main
    print("alternatives={}".format(",".join(alternatives)))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe5' in
position 340: ordinal not in range(128)
----------------------------------------------------------------------
Ran 3 tests in 5.094s
FAILED (failures=1)

2)
----------------------------------------------------------------------
GRASS nc_spm_08_grass7/user1:testsuite > python r3_forestfrag_trivial.py

Step 1: Computing Pf values...
100%
Step 2: Computing Pff values...
Step 3: Computing fragmentation index...
The following layers were created
The fragmentation index: r3ff_test_forest_frag_trivial
The proportion forested (Pf): r3ff_test_forest_frag_trivial_pf
The proportion forested pixel pairs (Pff):
r3ff_test_forest_frag_trivial_pff
F

FAIL: test_simple_window (__main__.TestForestFragTrivial)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mneteler/software/grass-addons/grass7/raster3d/r3.forestfrag/testsuite/r3_forestfrag_trivial.py",
line 183, in test_simple_window
    self.assertRasters3dNoDifference(actual=pf,
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/case.py",
line 989, in assertRasters3dNoDifference
    self.assertModuleKeyValue(
  File "/home/mneteler/software/grass_master/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/case.py",
line 282, in assertModuleKeyValue
    self.fail(self._formatMessage(msg, stdMsg))
AssertionError: r3.info map=tmp_520660a0f348483e86a206d016f4c7d5 -r difference:
mismatch values (key, reference, actual): [('max', 0.001, -0.00037)]
command: r3.info map=tmp_520660a0f348483e86a206d016f4c7d5 -r {'map':
'tmp_520660a0f348483e86a206d016f4c7d5', 'flags': 'r'}
----------------------------------------------------------------------
Ran 1 test in 3.075s
FAILED (failures=1)

3)
----------------------------------------------------------------------
GRASS :testsuite > python test_whatcsv.py
ERROR: Raster map <dummy> not found

--> reported with detail as https://github.com/OSGeo/grass-addons/issues/490

4)
----------------------------------------------------------------------
GRASS :testsuite > python test_whataggr.py
1|2001-04-01|nan
2|2001-04-01|nan
3|2001-04-01|nan
/home/mneteler/.grass7/addons/scripts/t.rast.what.aggr:125:
RuntimeWarning: Mean of empty slice.
  return vals.mean()
/usr/lib64/python3.9/site-packages/numpy/core/_methods.py:170:
RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
F
WARNING: Attribute table of vector points will be updated...
Adding column <A_average> to the table
/home/mneteler/.grass7/addons/scripts/t.rast.what.aggr:125:
RuntimeWarning: Mean of empty slice.
  return vals.mean()
/usr/lib64/python3.9/site-packages/numpy/core/_methods.py:170:
RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
SQL: "UPDATE points SET A_average=nan WHERE cat=1"
DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such column: nan
DBMI-SQLite driver error:
Error in sqlite3_prepare():
no such column: nan
ERROR: Error while executing: 'UPDATE points SET A_average=nan WHERE cat=1'
[...]

5)
----------------------------------------------------------------------
v.stream.order testsuite: update tests

See new https://github.com/OSGeo/grass-addons/pull/491

----------------------------------------------------------------------
Untested for today...:

test_whataggr from ./temporal/t.rast.what.aggr failed (6 tests failed)
test_gpot from ./raster/r.green/r.green.gshp/libgshp failed
test_numbers from ./raster/r.sample.category failed
r_extract_test from ./raster/r.extract failed (2 tests failed)
r_object_geometry_test from ./raster/r.object.geometry failed (2 tests failed)
r_forestfrag_trivial from ./raster/r.forestfrag failed (1 test failed)
r_forestfrag_xy from ./raster/r.forestfrag failed (1 test failed)
test_append from ./raster/r.learn.ml2 failed

Shell test:
test_base_resolution from ./raster/r.in.pdal failed --> test is a
shell script and should be Python

Seems the Addon tests needs some love...

Markus

On Wed, Mar 24, 2021 at 6:45 PM Markus Neteler <neteler@osgeo.org> wrote:

I spent some time on this - installed Ubuntu 18.04 from docker and
installed GRASS GIS 7.8 and addons inside to run the tests. It comes
with Python 3.6:

Here what came up (would be easier to see this in GH Action directly
since it took me almost an hour to set up the container… ):

All what you have sent so far can/should be seen in the GH Actions running with the master branch of core grass repo. But since you have set it up, you can run the r_forestfrag_xy.py from ./raster/r.forestfrag which seems to be the one failing in 7.8 with Python 3.6.

Seems the Addon tests needs some love…

…so do the ones in core. Help needed.

On Thu, Mar 25, 2021 at 1:55 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Wed, Mar 24, 2021 at 6:45 PM Markus Neteler <neteler@osgeo.org> wrote:
>
> I spent some time on this - installed Ubuntu 18.04 from docker and
> installed GRASS GIS 7.8 and addons inside to run the tests. It comes
> with Python 3.6:
>
> Here what came up (would be easier to see this in GH Action directly
> since it took me almost an hour to set up the container... ):

All what you have sent so far can/should be seen in the GH Actions running with the master branch of core grass repo. But since you have set it up,

Due to "battery empty" on laptop I lost the Ubuntu 18.04 based
container, however...

you can run the r_forestfrag_xy.py from ./raster/r.forestfrag which seems to be the one failing in 7.8 with Python 3.6.

we have a Centos VM with Python 3.6 which I used now.

- r_forestfrag_ncspm.py: OK
- r_forestfrag_xy.py: OK
- r_forestfrag_trivial.py: Failing, see below

GRASS 7.8.5 (nc_spm_08_grass7):/tmp/testsuite > python3 r_forestfrag_trivial.py
WARNING: The window option is deprecated, use the option size instead
Step 1: Computing Pf values...
Step 2: Computing Pff values...
Step 3: Computing fragmentation index...
The following layers were created
The fragmentation index: rff_test_forest_frag_trivial
The proportion forested (Pf): rff_test_forest_frag_trivial_pf
The proportion forested pixel pairs (Pff): rff_test_forest_frag_trivial_pff

F

FAIL: test_riitters (__main__.TestForestFragTrivial)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "r_forestfrag_trivial.py", line 158, in test_riitters
    self.forest_frag_general(FOREST_RIITTERS, 3, FRAG_RIITTERS)
  File "r_forestfrag_trivial.py", line 146, in forest_frag_general
    precision=0.01)
  File "/usr/lib64/grass78/etc/python/grass/gunittest/case.py", line
771, in assertRastersNoDifference
    reference=statistics, msg=msg)
  File "/usr/lib64/grass78/etc/python/grass/gunittest/case.py", line
254, in assertModuleKeyValue
    self.fail(self._formatMessage(msg, stdMsg))
AssertionError: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r difference:
mismatch values (key, reference, actual): [('min', -0.01, 0.00333331346511845)]
command: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r {'map':
'tmp_0ed1ea80157c438eb79d720a781542bf', 'flags': 'r'}

----------------------------------------------------------------------
Ran 1 test in 3.133s
FAILED (failures=1)

Markus

On Thu, Mar 25, 2021 at 7:49 PM Markus Neteler <neteler@osgeo.org> wrote:

On Thu, Mar 25, 2021 at 1:55 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

you can run the r_forestfrag_xy.py from ./raster/r.forestfrag which seems to be the one failing in 7.8 with Python 3.6.

we have a Centos VM with Python 3.6 which I used now.

  • r_forestfrag_ncspm.py: OK
  • r_forestfrag_xy.py: OK
  • r_forestfrag_trivial.py: Failing, see below

GRASS 7.8.5 (nc_spm_08_grass7):/tmp/testsuite > python3 r_forestfrag_trivial.py

self.fail(self._formatMessage(msg, stdMsg))
AssertionError: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r difference:
mismatch values (key, reference, actual): [(‘min’, -0.01, 0.00333331346511845)]
command: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r {‘map’:
‘tmp_0ed1ea80157c438eb79d720a781542bf’, ‘flags’: ‘r’}

Thanks, but that one you can see failing with output in the master - Python 3.8 combo. See e.g. here:

https://github.com/OSGeo/grass-addons/runs/2165122098?check_suite_focus=true

Nevermind, we will get that eventually.

Not an immediate issue after the dependency related PR 488 is merged, but for actually fixing the failing tests, it is worth noting that now also the 7.8 runs can be examined in core and in addons through artifacts. For master branch (in both core and addons), it just depends what you find more convenient, HTML from artifact, text output in the Checks view, or the same text output but through View raw logs.

https://github.com/OSGeo/grass-addons/pull/488

https://github.com/OSGeo/grass-addons/pull/494
https://github.com/OSGeo/grass/pull/1484

On Thu, Mar 25, 2021 at 9:26 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Thu, Mar 25, 2021 at 7:49 PM Markus Neteler <neteler@osgeo.org> wrote:

On Thu, Mar 25, 2021 at 1:55 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

you can run the r_forestfrag_xy.py from ./raster/r.forestfrag which seems to be the one failing in 7.8 with Python 3.6.

we have a Centos VM with Python 3.6 which I used now.

  • r_forestfrag_ncspm.py: OK
  • r_forestfrag_xy.py: OK
  • r_forestfrag_trivial.py: Failing, see below

GRASS 7.8.5 (nc_spm_08_grass7):/tmp/testsuite > python3 r_forestfrag_trivial.py

self.fail(self._formatMessage(msg, stdMsg))
AssertionError: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r difference:
mismatch values (key, reference, actual): [(‘min’, -0.01, 0.00333331346511845)]
command: r.info map=tmp_0ed1ea80157c438eb79d720a781542bf -r {‘map’:
‘tmp_0ed1ea80157c438eb79d720a781542bf’, ‘flags’: ‘r’}

Thanks, but that one you can see failing with output in the master - Python 3.8 combo. See e.g. here:

https://github.com/OSGeo/grass-addons/runs/2165122098?check_suite_focus=true

Nevermind, we will get that eventually.

On Sun, Mar 28, 2021 at 4:55 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

Not an immediate issue after the dependency related PR 488 is merged, but for actually fixing the failing tests, it is worth noting that now also the 7.8 runs can be examined in core and in addons through artifacts. For master branch (in both core and addons), it just depends what you find more convenient, HTML from artifact, text output in the Checks view, or the same text output but through View raw logs.

This is a great help - thanks for that!

Markus