[GRASS-dev] RFC: variations of statistics in r.neighbors (and the stats lib)

GRASS GRASS developer
1

Hi,

for some specific needs of a research project, I had to make a little
change to r.neighbors (the target version was 7.8.5 but that's not essential).

Essentially, the idea behind is computing first order statistics on partial
populations as identified by selected quantiles (samples >= or <= of a
threshold value of quantile).
  For that, I introduced average_ge_quantile and average_le_quantile operators modes.

https://github.com/fpl/grass/commit/6b83795b037c6645a32d6a525cfdee3cc65d521c

(the html file is still not updated)

I'm not persuaded this is the most elegant way of doing this kind of things,
maybe it would be better using an option (as in the case of -w for weighted
operations) to compute average and possibly other statistics. Even, one could
think in general to other multiple ways of selecting population on the base of
quantiles/percentiles of population in a window.

Any hint/opinion/alternative/critic about that?

All this in the remote hypothesis that a pull request could have sense
for such a kind of features.

Thanks

--
Francesco P. Lovergine

2

Hi Francesco,

the proposed change to r.neighbors is interesting, but maybe too specific: you have introduced two new functions, and many more functions would be needed to e.g. get the filtered standard deviation or median.

Therefore I suggest adding some filtering option to r.neighbors consisting of a filtering function and a comparison operator. The filtering function would be any of the currently supported neighborhood functions returning some value. The comparison operator would be one of gt, ge, le, lt (>, >=, <=, <). r.neighbors would then first get the result value of the filtering function, then set all values in the neighborhood to NULL that do not fulfil the condition “value <result value”>, then call the actual neighborhood operation with the filtered values. This would be more flexible, because the user can freely combine a neighborhood filter function with a neighborhood operation.

Makes sense?

On Wed, Jan 25, 2023 at 10:55 AM Francesco P. Lovergine <frankie@debian.org> wrote:

Hi,

for some specific needs of a research project, I had to make a little
change to r.neighbors (the target version was 7.8.5 but that’s not essential).

Essentially, the idea behind is computing first order statistics on partial
populations as identified by selected quantiles (samples >= or <= of a
threshold value of quantile).

For that, I introduced average_ge_quantile and average_le_quantile operators modes.

https://github.com/fpl/grass/commit/6b83795b037c6645a32d6a525cfdee3cc65d521c

(the html file is still not updated)

I’m not persuaded this is the most elegant way of doing this kind of things,
maybe it would be better using an option (as in the case of -w for weighted
operations) to compute average and possibly other statistics. Even, one could
think in general to other multiple ways of selecting population on the base of
quantiles/percentiles of population in a window.

Any hint/opinion/alternative/critic about that?

All this in the remote hypothesis that a pull request could have sense
for such a kind of features.

Thanks

–
Francesco P. Lovergine

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

3

Hello Francesco, Markus

I find the possibility of getting statistics for only a certain fraction of a neighbourhood really interesting and I like Markus’ idea as it is more general and allows for more use cases :nerd_face: Count me in for testing :wink:

Vero

El mar, 31 ene 2023 a las 15:48, Markus Metz (<markus.metz.giswork@gmail.com>) escribió:

Hi Francesco,

the proposed change to r.neighbors is interesting, but maybe too specific: you have introduced two new functions, and many more functions would be needed to e.g. get the filtered standard deviation or median.

Therefore I suggest adding some filtering option to r.neighbors consisting of a filtering function and a comparison operator. The filtering function would be any of the currently supported neighborhood functions returning some value. The comparison operator would be one of gt, ge, le, lt (>, >=, <=, <). r.neighbors would then first get the result value of the filtering function, then set all values in the neighborhood to NULL that do not fulfil the condition “value <result value”>, then call the actual neighborhood operation with the filtered values. This would be more flexible, because the user can freely combine a neighborhood filter function with a neighborhood operation.

Makes sense?

On Wed, Jan 25, 2023 at 10:55 AM Francesco P. Lovergine <frankie@debian.org> wrote:

Hi,

for some specific needs of a research project, I had to make a little
change to r.neighbors (the target version was 7.8.5 but that’s not essential).

Essentially, the idea behind is computing first order statistics on partial
populations as identified by selected quantiles (samples >= or <= of a
threshold value of quantile).

For that, I introduced average_ge_quantile and average_le_quantile operators modes.

https://github.com/fpl/grass/commit/6b83795b037c6645a32d6a525cfdee3cc65d521c

(the html file is still not updated)

I’m not persuaded this is the most elegant way of doing this kind of things,
maybe it would be better using an option (as in the case of -w for weighted
operations) to compute average and possibly other statistics. Even, one could
think in general to other multiple ways of selecting population on the base of
quantiles/percentiles of population in a window.

Any hint/opinion/alternative/critic about that?

All this in the remote hypothesis that a pull request could have sense
for such a kind of features.

Thanks

–
Francesco P. Lovergine

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

4

On Tue, Jan 31, 2023 at 07:47:49PM +0100, Markus Metz wrote:

Hi Francesco,

Hi Markus,

the proposed change to r.neighbors is interesting, but maybe too specific:
you have introduced two new functions, and many more functions would be
needed to e.g. get the filtered standard deviation or median.

Indeed, that was the reason of my doubt on that approach. The changes
introduced were simple enough and workerd properly for my specific case,
but it is neither elegant nor general at all.

Therefore I suggest adding some filtering option to r.neighbors consisting
of a filtering function and a comparison operator. The filtering function
would be any of the currently supported neighborhood functions returning
some value. The comparison operator would be one of gt, ge, le, lt (>, >=,
<=, <). r.neighbors would then first get the result value of the filtering
function, then set all values in the neighborhood to NULL that do not
fulfil the condition "value <comparison operator> <result value">, then
call the actual neighborhood operation with the filtered values. This would
be more flexible, because the user can freely combine a neighborhood filter
function with a neighborhood operation.

This seems reasonable to me. The same approach could be used for at least
a pair of different population selection filter, i.e. by-quantile and by-quartile.

filter ::= by-quantile | by-quartile
filter-cmp-op ::= ge | le | gt | lt | range

with target value taken from quantile or quartile arguments. Eventually, one
could use a range too and specify a pair of new cmd line arguments:

quantile_low | quantile_high quartile_low | quartile_high

or change quantile and quartile into a list of 1..2 comma separated values.

Much better, isn't it?

On Wed, Jan 25, 2023 at 10:55 AM Francesco P. Lovergine <frankie@debian.org>
wrote:

Hi,

for some specific needs of a research project, I had to make a little
change to r.neighbors (the target version was 7.8.5 but that's not
essential).

Essentially, the idea behind is computing first order statistics on partial
populations as identified by selected quantiles (samples >= or <= of a
threshold value of quantile).

For that, I introduced average_ge_quantile and average_le_quantile
operators modes.

https://github.com/fpl/grass/commit/6b83795b037c6645a32d6a525cfdee3cc65d521c

(the html file is still not updated)

I'm not persuaded this is the most elegant way of doing this kind of
things,
maybe it would be better using an option (as in the case of -w for weighted
operations) to compute average and possibly other statistics. Even, one
could
think in general to other multiple ways of selecting population on the
base of
quantiles/percentiles of population in a window.

Any hint/opinion/alternative/critic about that?

All this in the remote hypothesis that a pull request could have sense
for such a kind of features.

Thanks

--
Francesco P. Lovergine
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Francesco P. Lovergine

5

Hi Francesco,

On Wed, 1 Feb 2023 at 07:09, Francesco Paolo Lovergine <frankie@debian.org> wrote:

…or change quantile and quartile into a list of 1…2 comma separated values.

Much better, isn’t it?

Maybe, but explicit named arguments are nice, too.

Do you plan to open a PR? A more experimental code could also go to the grass-addons repo.

Vaclav