[GRASS-dev] r.quantile strange output at small percentiles intervals

GRASS GIS GRASS GIS developer
1

Hi,

When running r.quantile with percentiles set at 0-100 at steps of 0.25, all runs fine, the output would look like e.g (output in the recode rules format - only first and last three shown) .,

2.879961:6.660391:1
6.660391:6.791101:2
6.791101:8.757845:3
.
.
.
9.144313:9.145126:397
9.145126:12.874843:398
12.874843:12.874843:399

When setting at smaller steps, e.g., 0.2, the output becomes strange (first, middle and last three shown):

2.879961:6.469324:1
6.469324:9.219839:2
9.219839:9.219842:3
.
.
.
9.216583:0.000000:407
0.000000:0.000000:408
0.000000:0.000000:409
.
.
.
0.000000:0.000000:498
0.000000:12.874843:499
12.874843:12.874843:500

Any ideas why this is?

Paulo

2

Paulo van Breugel wrote:

When running r.quantile with percentiles set at 0-100 at steps of 0.25, all
runs fine

When setting at smaller steps, e.g., 0.2, the output becomes strange

9.216583:0.000000:407
0.000000:0.000000:408
0.000000:0.000000:409

Any ideas why this is?

It uses an 8-bit value to store the bin number for each slot, which
imposes a limit of 255 bins. Exceeding this number causes earlier bins
to be "recycled" while later bins aren't filled (leaving them
containing zeroes).

In r53951, I've enlarged this to 16 bits and added an error if there
are more than 65535 quantiles (the number of bins will never exceed
the number of quantiles).

--
Glynn Clements <glynn@gclements.plus.com>

3

Great, thanks for the quick fix / improvement!

On Wed 21 Nov 2012 04:36:10 PM CET, Glynn Clements wrote:

Paulo van Breugel wrote:

When running r.quantile with percentiles set at 0-100 at steps of 0.25, all
runs fine

When setting at smaller steps, e.g., 0.2, the output becomes strange

9.216583:0.000000:407
0.000000:0.000000:408
0.000000:0.000000:409

Any ideas why this is?

It uses an 8-bit value to store the bin number for each slot, which
imposes a limit of 255 bins. Exceeding this number causes earlier bins
to be "recycled" while later bins aren't filled (leaving them
containing zeroes).

In r53951, I've enlarged this to 16 bits and added an error if there
are more than 65535 quantiles (the number of bins will never exceed
the number of quantiles).