[GRASS-dev] GRASS-dev] On i.histo.match (Re: On (Landsat) imagery naming patterns)

GRASS GIS GRASS GIS developer
1

Nikos,

What about just using r.rescale to rescale this?

Michael

C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Arizona State University

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671 (SHESC), 480-727-0709 (CSDC)

www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu

On Jul 31, 2013, at 8:07 PM, <grass-dev-request@lists.osgeo.org>
wrote:

From: Nikos Alexandris <nik@nikosalexandris.net>

Subject: Re: [GRASS-dev] On i.histo.match (Re: On (Landsat) imagery naming patterns)

Date: July 31, 2013 3:38:24 PM MST

To: Hamish <hamish_b@yahoo.com>

Cc: <grass-dev@lists.osgeo.org>, Markus Metz <markus.metz.giswork@gmail.com>

Nikos wrote:

Just FYI, results look nice! I even convert back to 0-1.0 via

r.mapcalc --o “${HistoMatchedMap} = ${HistoMatchedMap} / 1000.0”
ps- I wonder if *10000 is better for higher precision?

By the way, that was Landsat5!

depending on the sensor’s 8-bitnesss or not, you can probably calculate by
hand now many significant digits are useful. A little bit extra probably
doesn’t hurt.

1 / 2^8 = 0.00390625

I didn’t pay too much attention back then as I was under enormous time
pressure. But, I think I didn’t miss much of the precision with respect to
the final product’s scope. Except if the last 4 (or 3, can’t remember) digits
make up a great deal when histo-matching.

Back to present. QuickBird is an 11-bit sensor [1] and data are delivered as
either 8-bit or 16-bit. In this case we have

1 / 2^11 = 0.0004882812

Does that say anything about the significant digits? I want to rescale
QuickBird bands in order to use’em with i.pansharpen. I am thinking to go
from Reflectance (double precision) to integer (as above, only this time I’d
multiply with a number as big as it takes to keep all decimals) and then
rescale to [0, 255].

Makes non-/sense? Any idea how many decimals should be preserved for
analysis? I might be hunting “fine digits” for nothing…

In any case, conversions from DNs to Reflectance should be done in a 32-bit
level [2] (that corresponds to: 1 / 2.328306e-10). The same is done with
Landsat imagery though both L5 and L7 data are delivered as 8-bit [3].

Nikos

[1] also mentioned in the document
<http://www.digitalglobe.com/downloads/QuickBird_technote_raduse_v1.pdf>, page
7

[2] same document, page 8: “…conversion equations are to be
performed on all pixels in a given band of a QuickBird image and should use
32-bit floating point calculations.”

[3] http://landsat.usgs.gov/how_is_radiance_calculated.php

2

Michael Barton wrote:

Nikos,

What about just using r.rescale to rescale this?

Already tried (in the past) and I don't think it works from DCELL > 8-bit. It
seems to chew-up (silently, as Moritz mentioned I think among the lines in
ticket #2^11) values.

It seems that "integerising" manually, in this case, is the best approach.
With the essential question remaining on "how many fine digits should be
preserved?".

Thank you Michael, Nikos

3

On 01/08/13 11:04, Nikos Alexandris wrote:

Michael Barton wrote:

Nikos,

What about just using r.rescale to rescale this?

Already tried (in the past) and I don't think it works from DCELL> 8-bit. It
seems to chew-up (silently, as Moritz mentioned I think among the lines in
ticket #2^11) values.

It seems that "integerising" manually, in this case, is the best approach.
With the essential question remaining on "how many fine digits should be
preserved?".

r.rescale is just a frontend to r.reclass. and as such is meant for CELL maps. It should'nt make a difference whether it is 8-bit or more, though.

For DCELL you can try to use r.recode.

Moritz

4

Michael:

>> What about just using r.rescale to rescale this?

Nikos:

> Already tried (in the past) and I don't think it works from DCELL> 8-bit.
> It seems to chew-up (silently, as Moritz mentioned I think among the
> lines in ticket #2^11) values.
>
> It seems that "integerising" manually, in this case, is the best approach.
> With the essential question remaining on "how many fine digits should be
> preserved?".

Moritz:

r.rescale is just a frontend to r.reclass. and as such is meant for CELL
maps. It should'nt make a difference whether it is 8-bit or more, though.

For DCELL you can try to use r.recode.

Didn't work also (tried the previous days) -- I can try again.

Nikos

5

On 01/08/13 11:40, Nikos Alexandris wrote:

Michael:

What about just using r.rescale to rescale this?

Nikos:

Already tried (in the past) and I don't think it works from DCELL> 8-bit.
It seems to chew-up (silently, as Moritz mentioned I think among the
lines in ticket #2^11) values.

It seems that "integerising" manually, in this case, is the best approach.
With the essential question remaining on "how many fine digits should be
preserved?".

Moritz:

r.rescale is just a frontend to r.reclass. and as such is meant for CELL
maps. It should'nt make a difference whether it is 8-bit or more, though.

For DCELL you can try to use r.recode.

Didn't work also (tried the previous days) -- I can try again.

Please be more precise than "didn't work"...

Moritz

6

Michael:

>>>> What about just using r.rescale to rescale this?

Nikos:

>>> Already tried (in the past) and I don't think it works from DCELL>
>>> 8-bit. It seems to chew-up (silently, as Moritz mentioned I think among
>>> the lines in ticket #2^11) values.

>>> It seems that "integerising" manually, in this case, is the best
>>> approach. With the essential question remaining on "how many fine digits
>>> should be preserved?".

Moritz:

>> r.rescale is just a frontend to r.reclass. and as such is meant for CELL
>> maps. It should'nt make a difference whether it is 8-bit or more, though.
>> For DCELL you can try to use r.recode.

Nikos:

> Didn't work also (tried the previous days) -- I can try again.

Moritz:

Please be more precise than "didn't work"...

Right, "be more precise" is the key to "freedom" :D. Indeed, I used to say
(either in a "rules" file, or directly using "...<< EOF"

0.0:1.0:0:255

This did not work. Both stats and histogram of the recoded raster map, e.g. a
Red-Reflectance image ranging in

r.info Red_ToAR -r

min=0
max=0.774115699104528

were kinda flattened out

r.stats Red_ToAR_recoded_255
100%

0
255

Looking at the image I want to recode

r.stats Red_ToAR | head
100%

0-0.003036
0.02125-0.024286
0.024286-0.027322
0.027322-0.030357
0.030357-0.033393
0.033393-0.036429
0.036429-0.039465
0.039465-0.0425
0.0425-0.045536
0.045536-0.048572

I altered the rules file like

0.001:1.0:0:255

This works-out! Now, the recoded image is

r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
r.stats Red_ToAR_recoded_255
100%
5
6
7
.
..
... \
.... > Many values in-between
... /
..
.
195
196
197
*

And the histogram looks "nice" as well. I didn't grasp that -- from where
should I? In the manual there is only an example from int to float (however,
indeed, instructing 0.1 as the "target" min value).

Thanks, Nikos

7

On 01/08/13 13:43, Nikos Alexandris wrote:

Michael:

What about just using r.rescale to rescale this?

Nikos:

Already tried (in the past) and I don't think it works from DCELL>
8-bit. It seems to chew-up (silently, as Moritz mentioned I think among
the lines in ticket #2^11) values.

It seems that "integerising" manually, in this case, is the best
approach. With the essential question remaining on "how many fine digits
should be preserved?".

Moritz:

r.rescale is just a frontend to r.reclass. and as such is meant for CELL
maps. It should'nt make a difference whether it is 8-bit or more, though.
For DCELL you can try to use r.recode.

Nikos:

Didn't work also (tried the previous days) -- I can try again.

Moritz:

Please be more precise than "didn't work"...

Right, "be more precise" is the key to "freedom" :D. Indeed, I used to say
(either in a "rules" file, or directly using "...<< EOF"

0.0:1.0:0:255

This did not work. Both stats and histogram of the recoded raster map, e.g. a
Red-Reflectance image ranging in

r.info Red_ToAR -r

min=0
max=0.774115699104528

were kinda flattened out

r.stats Red_ToAR_recoded_255
  100%

0
255

Looking at the image I want to recode

r.stats Red_ToAR | head
  100%

0-0.003036
0.02125-0.024286
0.024286-0.027322
0.027322-0.030357
0.030357-0.033393
0.033393-0.036429
0.036429-0.039465
0.039465-0.0425
0.0425-0.045536
0.045536-0.048572

I altered the rules file like

0.001:1.0:0:255

This works-out! Now, the recoded image is

r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
r.stats Red_ToAR_recoded_255
  100%
5
6
7
.
..
... \
....> Many values in-between
... /
..
.
195
196
197
*

And the histogram looks "nice" as well. I didn't grasp that -- from where
should I? In the manual there is only an example from int to float (however,
indeed, instructing 0.1 as the "target" min value).

There does seem to be a bug in r.recode. Your first rule set should work if you set the -d flag. Maybe you can file a ticket ?

Moritz

8

On Thursday 01 of August 2013 13:42:03 Moritz Lennert wrote:

On 01/08/13 13:43, Nikos Alexandris wrote:
> Michael:
>>>>>> What about just using r.rescale to rescale this?
>
> Nikos:
>>>>> Already tried (in the past) and I don't think it works from DCELL>
>>>>> 8-bit. It seems to chew-up (silently, as Moritz mentioned I think
>>>>> among
>>>>> the lines in ticket #2^11) values.
>>>>>
>>>>> It seems that "integerising" manually, in this case, is the best
>>>>> approach. With the essential question remaining on "how many fine
>>>>> digits
>>>>> should be preserved?".
>
> Moritz:
>>>> r.rescale is just a frontend to r.reclass. and as such is meant for
>>>> CELL
>>>> maps. It should'nt make a difference whether it is 8-bit or more,
>>>> though.
>>>> For DCELL you can try to use r.recode.
>
> Nikos:
>>> Didn't work also (tried the previous days) -- I can try again.
>
> Moritz:
>> Please be more precise than "didn't work"...
>
> Right, "be more precise" is the key to "freedom" :D. Indeed, I used to
> say
> (either in a "rules" file, or directly using "...<< EOF"
>
> 0.0:1.0:0:255
>
> This did not work. Both stats and histogram of the recoded raster map,
> e.g. a Red-Reflectance image ranging in
>
> r.info Red_ToAR -r
>
> min=0
> max=0.774115699104528
>
> were kinda flattened out
>
> r.stats Red_ToAR_recoded_255
>
> 100%
>
> 0
> 255
>
> Looking at the image I want to recode
>
> r.stats Red_ToAR | head
>
> 100%
>
> 0-0.003036
> 0.02125-0.024286
> 0.024286-0.027322
> 0.027322-0.030357
> 0.030357-0.033393
> 0.033393-0.036429
> 0.036429-0.039465
> 0.039465-0.0425
> 0.0425-0.045536
> 0.045536-0.048572
>
> I altered the rules file like
>
> 0.001:1.0:0:255
>
>
> This works-out! Now, the recoded image is
>
> r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
> r.stats Red_ToAR_recoded_255
>
> 100%
>
> 5
> 6
> 7
> .
> ..
> ... \
> ....> Many values in-between
> ... /
> ..
> .
> 195
> 196
> 197
> *
>
> And the histogram looks "nice" as well. I didn't grasp that -- from where
> should I? In the manual there is only an example from int to float
> (however, indeed, instructing 0.1 as the "target" min value).

There does seem to be a bug in r.recode. Your first rule set should work
if you set the -d flag. Maybe you can file a ticket ?

Oh man... I need to rest. The "-d" flag... that's it :-p

Nikos (learning every day new stuff :-)).

9

Moritz:

> >>>> r.rescale is just a frontend to r.reclass. and as such is meant for
> >>>> CELLmaps. It should'nt make a difference whether it is 8-bit or more,
> >>>> though. For DCELL you can try to use r.recode.

Nikos:

> >>> Didn't work also (tried the previous days) -- I can try again.

Moritz:

> >> Please be more precise than "didn't work"...

Nikos:

> > Right, "be more precise" is the key to "freedom" :D. Indeed, I used to
> > say (either in a "rules" file, or directly using "...<< EOF"

[..]

> > I altered the rules file like

> > 0.001:1.0:0:255

> > This works-out! Now, the recoded image is

> > r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
> > r.stats Red_ToAR_recoded_255
> >
> > 100%
> >
> > 5
> > 6
> > 7
> > .
> > ..
> > ... \
> > ....> Many values in-between
> > ... /
> > ..
> > .
> > 195
> > 196
> > 197
> > *
> >
> > And the histogram looks "nice" as well. I didn't grasp that -- from
> > where
> > should I? In the manual there is only an example from int to float
> > (however, indeed, instructing 0.1 as the "target" min value).

Moritz:

> There does seem to be a bug in r.recode. Your first rule set should work
> if you set the -d flag. Maybe you can file a ticket ?

Nikos:

Oh man... I need to rest. The "-d" flag... that's it :-p

On the other hand, even if I would read it (because I did not read carefully
the --help before), it reads:

-d Force output to 'double' raster map type (DCELL)

In this case I have a DCELL Input, recoding in to CELL Output. How is this
related?

N

10

On 01/08/13 14:01, Nikos Alexandris wrote:

Moritz:

r.rescale is just a frontend to r.reclass. and as such is meant for
CELLmaps. It should'nt make a difference whether it is 8-bit or more,
though. For DCELL you can try to use r.recode.

Nikos:

Didn't work also (tried the previous days) -- I can try again.

Moritz:

Please be more precise than "didn't work"...

Nikos:

Right, "be more precise" is the key to "freedom" :D. Indeed, I used to
say (either in a "rules" file, or directly using "...<< EOF"

[..]

I altered the rules file like

0.001:1.0:0:255

This works-out! Now, the recoded image is

r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
r.stats Red_ToAR_recoded_255

   100%

5
6
7
.
..
... \
....> Many values in-between
... /
..
.
195
196
197
*

And the histogram looks "nice" as well. I didn't grasp that -- from
where
should I? In the manual there is only an example from int to float
(however, indeed, instructing 0.1 as the "target" min value).

Moritz:

There does seem to be a bug in r.recode. Your first rule set should work
if you set the -d flag. Maybe you can file a ticket ?

Nikos:

Oh man... I need to rest. The "-d" flag... that's it :-p

On the other hand, even if I would read it (because I did not read carefully
the --help before), it reads:

-d Force output to 'double' raster map type (DCELL)

In this case I have a DCELL Input, recoding in to CELL Output. How is this
related?

Sorry, you're right. Got mixed up with another problem I had when I tried to recode a 0:2047 image to 0.0:1.0. And even if -d solved my problem there AFAICT it shouldn't be necessary.

Trying to do too many things at the same time in the last hours before leaving on vacation !!

Moritz

11

Sorry, I meant r.recode.

r.recode infile outfile min:max:0:255

should do it.

MIchael
______________________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671(SHESC), 480-727-0709 (CSDC)
www: http://csdc.asu.edu, http://shesc.asu.edu
    http://www.public.asu.edu/~cmbarton

On Aug 1, 2013, at 2:28 AM, Moritz Lennert <mlennert@club.worldonline.be>
wrote:

On 01/08/13 11:04, Nikos Alexandris wrote:

Michael Barton wrote:

Nikos,

What about just using r.rescale to rescale this?

Already tried (in the past) and I don't think it works from DCELL> 8-bit. It
seems to chew-up (silently, as Moritz mentioned I think among the lines in
ticket #2^11) values.

It seems that "integerising" manually, in this case, is the best approach.
With the essential question remaining on "how many fine digits should be
preserved?".

r.rescale is just a frontend to r.reclass. and as such is meant for CELL maps. It should'nt make a difference whether it is 8-bit or more, though.

For DCELL you can try to use r.recode.

Moritz

12

So r.recode works, although it seems to have a bug.

Maybe that is the key to having i.pansharpen work better with a wider range of input values. The user would still need to input the min and max possible range, but it could default to 0 and 255 for 8 bit data.

This changes original values of course. But pan sharpening is not intended to keep the original values. It is to create a visual output. Of course if the visual output looks crummy, that's not good either.

Michael
______________________________
C. Michael Barton
Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262 (SHESC), 480-965-8130/727-9746 (CSDC)
fax: 480-965-7671(SHESC), 480-727-0709 (CSDC)
www: http://csdc.asu.edu, http://shesc.asu.edu
    http://www.public.asu.edu/~cmbarton

On Aug 1, 2013, at 6:15 AM, Moritz Lennert <mlennert@club.worldonline.be>
wrote:

On 01/08/13 14:01, Nikos Alexandris wrote:

Moritz:

r.rescale is just a frontend to r.reclass. and as such is meant for
CELLmaps. It should'nt make a difference whether it is 8-bit or more,
though. For DCELL you can try to use r.recode.

Nikos:

Didn't work also (tried the previous days) -- I can try again.

Moritz:

Please be more precise than "didn't work"...

Nikos:

Right, "be more precise" is the key to "freedom" :D. Indeed, I used to
say (either in a "rules" file, or directly using "...<< EOF"

[..]

I altered the rules file like

0.001:1.0:0:255

This works-out! Now, the recoded image is

r.recode in=Red_ToAR out=Red_ToAR_recoded_255 rules=recode_rules --o
r.stats Red_ToAR_recoded_255

  100%

5
6
7
.
..
... \
....> Many values in-between
... /
..
.
195
196
197
*

And the histogram looks "nice" as well. I didn't grasp that -- from
where
should I? In the manual there is only an example from int to float
(however, indeed, instructing 0.1 as the "target" min value).

Moritz:

There does seem to be a bug in r.recode. Your first rule set should work
if you set the -d flag. Maybe you can file a ticket ?

Nikos:

Oh man... I need to rest. The "-d" flag... that's it :-p

On the other hand, even if I would read it (because I did not read carefully
the --help before), it reads:

-d Force output to 'double' raster map type (DCELL)

In this case I have a DCELL Input, recoding in to CELL Output. How is this
related?

Sorry, you're right. Got mixed up with another problem I had when I tried to recode a 0:2047 image to 0.0:1.0. And even if -d solved my problem there AFAICT it shouldn't be necessary.

Trying to do too many things at the same time in the last hours before leaving on vacation !!

Moritz