[GRASS-dev] dealing with null values in rasters only within mask

GRASS GRASS developer
1

Hi to all,

As a followup of some recent discussions concerning the handling of null values, I have a fundamental issue, and would like some feedback:

I am working with irregular tiles of orthoimages. Each tile corresponds to the part of a large image mosaic that comes from a particular take.

Some of these tiles contain pixels with null values (generally due to sensor saturation).

I would like to be able to do the following:

- detect whether there are any null values in the tile

- replace the pixels values with a given value (or possibly use something like r.fill.stats)

For this, I am confronted to some issues:

- r.univar does not make a difference between null values in the data and those that are due to a mask (https://trac.osgeo.org/grass/ticket/3696). This means that if I create a mask to correspond to the irregular tile, I cannot use r.univar to detect whether there are any null values within the actual image tile.

- If I decide to just replace all null values with a given value, r.null also does not take into account the mask. IOW, it will fill all null pixels in the entire region outside the mask with that value.

- The same happens if I use r.mapcalc to replace null values with a given value.

Is there any way to solve this issues with existing tools, or would this demand a change in the code, in order to differentiate null values that are null because of a mask from other null values ? Any reflections on this ?

Moritz

2

Hi Moritz,

Did you consider r.stats -N for checking for NULL values within the masked area, like:
r.stats -N input=tile,MASK
if there is *|1 in the output you have NULL cells in the tile...

For filling NULLs with a given value you could do:
r.mapcalc expression=tile_filled=if(MASK,if(isnull(tile),1,tile),null())

Just an idea...

Cheers
Stefan

-----Original Message-----
From: grass-dev <grass-dev-bounces@lists.osgeo.org> On Behalf Of Moritz Lennert
Sent: fredag 28. juni 2019 10:57
To: grass-dev@lists.osgeo.org
Subject: [GRASS-dev] dealing with null values in rasters only within mask

Hi to all,

As a followup of some recent discussions concerning the handling of null values, I have a fundamental issue, and would like some feedback:

I am working with irregular tiles of orthoimages. Each tile corresponds to the part of a large image mosaic that comes from a particular take.

Some of these tiles contain pixels with null values (generally due to sensor saturation).

I would like to be able to do the following:

- detect whether there are any null values in the tile

- replace the pixels values with a given value (or possibly use something like r.fill.stats)

For this, I am confronted to some issues:

- r.univar does not make a difference between null values in the data and those that are due to a mask (https://trac.osgeo.org/grass/ticket/3696). This means that if I create a mask to correspond to the irregular tile, I cannot use r.univar to detect whether there are any null values within the actual image tile.

- If I decide to just replace all null values with a given value, r.null also does not take into account the mask. IOW, it will fill all null pixels in the entire region outside the mask with that value.

- The same happens if I use r.mapcalc to replace null values with a given value.

Is there any way to solve this issues with existing tools, or would this demand a change in the code, in order to differentiate null values that are null because of a mask from other null values ? Any reflections on this ?

Moritz
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

3

Hi Moritz,

More generally from Stefan's answer:

1) MASK is a raster that you can use is your r.mapcalc expressions

2) A two-step process where everything is filled, and then the parts outside the mask are re-nulled?

  -k.

4

On 28/06/19 11:10, Stefan Blumentrath wrote:

Hi Moritz,

Did you consider r.stats -N for checking for NULL values within the masked area, like:
r.stats -N input=tile,MASK
if there is *|1 in the output you have NULL cells in the tile...

Nice trick, didn't actually know about the -N flag...

For filling NULLs with a given value you could do:
r.mapcalc expression=tile_filled=if(MASK,if(isnull(tile),1,tile),null())

Duh, now that I see this, it seems quite obvious.

Thanks Stefan !

(And thanks Ken for your ideas)

Moritz

5

Hi Moritz,

On Fri, Jun 28, 2019 at 11:56 AM Moritz Lennert <mlennert@club.worldonline.be> wrote:

On 28/06/19 11:10, Stefan Blumentrath wrote:

Hi Moritz,

Did you consider r.stats -N for checking for NULL values within the masked area, like:
r.stats -N input=tile,MASK
if there is *|1 in the output you have NULL cells in the tile…

the fastest way to get the total number of cells and the number of non-NULL cells, ignoring any MASK, is r.info -s, e.g.
cells=4917486 ← total number of cells
n=4873149 ← number of non-NULL cells

these numbers are stored in metadata and independent of the current region

Nice trick, didn’t actually know about the -N flag…

For filling NULLs with a given value you could do:
r.mapcalc expression=tile_filled=if(MASK,if(isnull(tile),1,tile),null())

in this case, I recommend to create a custom mask that is not named MASK but e.g. mymask, and the r.mapcalc expression probably needs some modification regarding NULL values in mymask:

r.mapcalc expression=“tile_filled = if(isnull(mymask) != 0 and mymask != 0, if(isnull(tile), 1, tile), null())”

The reason is that if(MASK, …) should return NULL if MASK is NULL which is not what you want, you want " if MASK is not NULL"

Markus M

Duh, now that I see this, it seems quite obvious.

Thanks Stefan !

(And thanks Ken for your ideas)

Moritz

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

6

Hi Markus,

On 28/06/19 22:08, Markus Metz wrote:

Hi Moritz,

On Fri, Jun 28, 2019 at 11:56 AM Moritz Lennert <mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be>> wrote:
>
> On 28/06/19 11:10, Stefan Blumentrath wrote:
> > Hi Moritz,
> >
> > Did you consider r.stats -N for checking for NULL values within the masked area, like:
> > r.stats -N input=tile,MASK
> > if there is *|1 in the output you have NULL cells in the tile...

the fastest way to get the total number of cells and the number of non-NULL cells, ignoring any MASK, is r.info <http://r.info> -s, e.g.
cells=4917486 <-- total number of cells
n=4873149 <-- number of non-NULL cells

these numbers are stored in metadata and independent of the current region

But so this does not help in my problem which is to determine the number of NULL pixels within a given MASK.

> > For filling NULLs with a given value you could do:
> > r.mapcalc expression=tile_filled=if(MASK,if(isnull(tile),1,tile),null())

in this case, I recommend to create a custom mask that is not named MASK but e.g. mymask, and the r.mapcalc expression probably needs some modification regarding NULL values in mymask:
r.mapcalc expression="tile_filled = if(isnull(mymask) != 0 and mymask != 0, if(isnull(tile), 1, tile), null())"

The reason is that if(MASK, ...) should return NULL if MASK is NULL which is not what you want, you want " if MASK is not NULL"

Right, thanks for the hint !

Moritz