[GRASS-dev] Python assertion error in GRASS GIS script

GRASS GRASS developer
1

Dears,

I try to understand why the following assertion fails to work, as
expected, in a Python GRASS GIS script?

The following fragment of code

fraction_categories = grass.parse_command('r.category',
                map=fractions,
                delimiter='\t')

fractions_sum = sum([float(x)
    if not math.isnan(float(x))
    else 0
    for x
    in fraction_categories.values()])
msg = "Fractions: {f}".format(f=fraction_categories)
g.message(_(msg))
g.message(_("Sum: {s}".format(s=fractions_sum)))

gives

Fractions: {u'24': u'nan', u'25': u'0.0159235666814', u'26':
u'0.0891719747445', u'27': u'nan', u'20': u'0.0238853509712', u'21':
u'0.0286624211654', u'23': u'0.143312100132', u'29': u'nan', u'40': u'nan',
u'41': u'0.0318471333627', u'1': u'nan', u'3': u'nan', u'2':
u'0.0159235669187', u'5': u'nan', u'4': u'nan', u'7': u'nan', u'6': u'nan',
u'9': u'nan', u'8': u'nan', u'11': u'0.00159235669186', u'10': u'nan',
u'12': u'0.162420386604', u'15': u'0.366242033672', u'16': u'nan', u'18':
u'0.0573248423308', u'31': u'nan', u'36': u'nan', u'35':
u'0.0636942667255', u'32': u'nan'}
Sum: 1.0

Yet the following assertion does not work as expected

assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

and returns

Traceback (most recent call last):
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 3979, in <module>
    sys.exit(main())
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 3964, in main
    **supply_parameters)
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 2864, in compute_supply
    assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"
AssertionError: Sum of fractions is > 1

The following _does_ work, in Python (inside a GRASS GIS session),

import os, sys, subprocess
import datetime, time
import csv
import math
import atexit
import grass.script as grass
from grass.exceptions import CalledModuleError
from grass.pygrass.modules.shortcuts import general as g
from grass.pygrass.modules.shortcuts import raster as r
from grass.pygrass.modules.shortcuts import vector as v
fractions_sum = 1.
assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

Nikos

2

On Mon, Oct 8, 2018 at 6:43 PM Nikos Alexandris <nik@nikosalexandris.net> wrote:

Dears,

I try to understand why the following assertion fails to work, as
expected, in a Python GRASS GIS script?

The following fragment of code

fraction_categories = grass.parse_command('r.category',
map=fractions,
delimiter='\t')

fractions_sum = sum([float(x)
if not math.isnan(float(x))
else 0
for x
in fraction_categories.values()])
msg = "Fractions: {f}".format(f=fraction_categories)
g.message(_(msg))
g.message(_("Sum: {s}".format(s=fractions_sum)))

gives

Fractions: {u'24': u'nan', u'25': u'0.0159235666814', u'26':
u'0.0891719747445', u'27': u'nan', u'20': u'0.0238853509712', u'21':
u'0.0286624211654', u'23': u'0.143312100132', u'29': u'nan', u'40': u'nan',
u'41': u'0.0318471333627', u'1': u'nan', u'3': u'nan', u'2':
u'0.0159235669187', u'5': u'nan', u'4': u'nan', u'7': u'nan', u'6': u'nan',
u'9': u'nan', u'8': u'nan', u'11': u'0.00159235669186', u'10': u'nan',
u'12': u'0.162420386604', u'15': u'0.366242033672', u'16': u'nan', u'18':
u'0.0573248423308', u'31': u'nan', u'36': u'nan', u'35':
u'0.0636942667255', u'32': u'nan'}
Sum: 1.0

Yet the following assertion does not work as expected

assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

and returns

Traceback (most recent call last):
File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 3979, in <module>
sys.exit(main())
File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 3964, in main
**supply_parameters)
File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 2864, in compute_supply
assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"
AssertionError: Sum of fractions is > 1

What is the sum when printed out with %.17g?

Why do you use 1.000000000000001 and not e.g. 1.000001 as threshold? 1.000000000000001 is most probably a too small difference to 1 considering floating point precision limits.

Markus M

The following does work, in Python (inside a GRASS GIS session),

import os, sys, subprocess
import datetime, time
import csv
import math
import atexit
import grass.script as grass
from grass.exceptions import CalledModuleError
from grass.pygrass.modules.shortcuts import general as g
from grass.pygrass.modules.shortcuts import raster as r
from grass.pygrass.modules.shortcuts import vector as v
fractions_sum = 1.
assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

Nikos

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

3

Nikos wrote:

The following fragment of code

fraction_categories = grass.parse_command('r.category',
                map=fractions,
                delimiter='\t')

fractions_sum = sum([float(x)
    if not math.isnan(float(x))
    else 0
    for x
    in fraction_categories.values()])
msg = "Fractions: {f}".format(f=fraction_categories)
g.message(_(msg))
g.message(_("Sum: {s}".format(s=fractions_sum)))

gives

Fractions: {u'24': u'nan', u'25': u'0.0159235666814', u'26':
u'0.0891719747445', u'27': u'nan', u'20': u'0.0238853509712', u'21':
u'0.0286624211654', u'23': u'0.143312100132', u'29': u'nan', u'40':

u’nan’,

u’41’: u’0.0318471333627’, u’1’: u’nan’, u’3’: u’nan’, u’2’:
u’0.0159235669187’, u’5’: u’nan’, u’4’: u’nan’, u’7’: u’nan’, u’6’:

u’nan’,

u’9’: u’nan’, u’8’: u’nan’, u’11’: u’0.00159235669186’, u’10’: u’nan’,
u’12’: u’0.162420386604’, u’15’: u’0.366242033672’, u’16’: u’nan’, u’18’:
u’0.0573248423308’, u’31’: u’nan’, u’36’: u’nan’, u’35’:
u’0.0636942667255’, u’32’: u’nan’}
Sum: 1.0


Yet the following assertion does not work as expected

assert fractions_sum < 1.000000000000001, “Sum of fractions is > 1”

and returns

Traceback (most recent call last):
File “/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap”,

line 3979, in 

sys.exit(main())

File “/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap”,

line 3964, in main

**supply_parameters)

File “/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap”,

line 2864, in compute_supply

assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

AssertionError: Sum of fractions is > 1

Markus Metz:

What is the sum when printed out with %.17g?

Bingo!

Following is from a script run, not from the fractions posted in this
thread.

Sum: 1.0000000000003
Traceback (most recent call last):
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 4207, in <module>
    sys.exit(main())
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 4192, in main
    **supply_parameters)
  File "/osgeo/grasstrunk/dist.x86_64-pc-linux-gnu/scripts/r.estimap", line 3019, in compute_supply
    assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"
AssertionError: Sum of fractions is > 1

Why do you use 1.000000000000001 and not e.g. 1.000001 as threshold?

Because of this (wrong I understand now) quick trial and error:

In [5]: 1.00000000000001 > 1
Out[5]: True

In [6]: 1.000000000000001 > 1
Out[6]: True

In [7]: 1.0000000000000001 > 1
Out[7]: False

1.000000000000001 is most probably a too small difference to 1 considering
floating point precision limits.

It is. But then again, how do I decide how "loose" it can be?

1.01?
1.001?
1.0001?

N

4

On Fri, Oct 12, 2018 at 11:22 PM Nikos Alexandris <nik@nikosalexandris.net> wrote:

Nikos wrote:

[…]

1.000000000000001 is most probably a too small difference to 1 considering
floating point precision limits.

It is. But then again, how do I decide how “loose” it can be?

1.01?
1.001?
1.0001?

From experience I recommend 1 + 1e-6 which is a bit lower than the minimum single precision floating point limit.

Instead of

assert fractions_sum < 1.000000000000001, “Sum of fractions is > 1”

try

assert abs(fractions_sum - 1) < 1e-6, “Sum of fractions is != 1”

You should not go lower than 1e-15 because here you are hitting the double precision floating point limit.

Markus M

5

Markus wrote:

>1.000000000000001 is most probably a too small difference to 1

considering

>floating point precision limits.

Nikos:

It is. But then again, how do I decide how "loose" it can be?

1.01?
1.001?
1.0001?

Markus:

From experience I recommend 1 + 1e-6 which is a bit lower than the minimum
single precision floating point limit.

Instead of

assert fractions_sum < 1.000000000000001, "Sum of fractions is > 1"

try

assert abs(fractions_sum - 1) < 1e-6, "Sum of fractions is != 1"

That's really smart. The fractions are fragments from the same "sum",
so this assertion refers exactly to this: their "sum" being close to 1.

This is what I need!

You should not go lower than 1e-15 because here you are hitting the double
precision floating point limit.

Kudos, Nikos