[GRASS-dev] r.colors "-e" not going to full range

Hi,

I just created a v.surf.rst elevation surface from some point data which
has a few negative values in it. After creating the surface I set the
colors with 'r.colors -e color=bcyr'. It looks really nice but there are
white holes in the d.rast display.

r.info -r shows:
min=-1.510814
max=146.157700

The colr/ file doesn't go to one integer beyond the full range:
----------------------------------
% -1 145
nv:255
*:255
-1:0:0:255 0:0:94:255
...
144:255:1:0 145:255:1:0
----------------------------------

??

I also noticed for another map 'r.colors -e' made a massive and slow
colr/ file (1 for each cat) when the raster values were +-250000. That
was a diff map centered on zero and I was using r.colors.stddev (wiki
addons) to set the colors. While it had those huge outliers 95% of the
points were +-5 (hence using the -e flag to get useful colors). It took
impossibly long to render of course, so I looked in the colr/ file and
discovered it was huge.

Ok, maybe that's a feature of the -e method. But what I saw that was
interesting was that between -265000 and -120 all the R:G:B values were
identical, ie all those thousands and thousands of rules could have been
condensed into a single "-265000:R:G:B -120:R:G:B". And that a little
program to do that would probably be trivial. I am a bit unsure if that
condensing should happen as a addon script which reprocesses the file,
when the colr/ file is created by 'r.colors -e', or when the color rules
are read in upon d.rast. (or some combination of the above) It would seem like a nice enhancement to have that happen though.

I notice that even in my current -1.5 thru 146m map '-e' produces some redundant entries.

Hamish

Hamish wrote:

I just created a v.surf.rst elevation surface from some point data which
has a few negative values in it. After creating the surface I set the
colors with 'r.colors -e color=bcyr'. It looks really nice but there are
white holes in the d.rast display.

r.info -r shows:
min=-1.510814
max=146.157700

The colr/ file doesn't go to one integer beyond the full range:
----------------------------------
% -1 145
nv:255
*:255
-1:0:0:255 0:0:94:255
...
144:255:1:0 145:255:1:0
----------------------------------

??

I didn't realise that "r.colors -e" even worked on FP data;
"r.colors color=grey.eq" doesn't.

Ultimately, the code is dependent upon the categories returned by
G_next_cell_stat(). If it only returns up to category 145, then the
colour table will only extend to that value.

I also noticed for another map 'r.colors -e' made a massive and slow
colr/ file (1 for each cat) when the raster values were +-250000. That
was a diff map centered on zero and I was using r.colors.stddev (wiki
addons) to set the colors. While it had those huge outliers 95% of the
points were +-5 (hence using the -e flag to get useful colors). It took
impossibly long to render of course, so I looked in the colr/ file and
discovered it was huge.

Ok, maybe that's a feature of the -e method. But what I saw that was
interesting was that between -265000 and -120 all the R:G:B values were
identical, ie all those thousands and thousands of rules could have been
condensed into a single "-265000:R:G:B -120:R:G:B". And that a little
program to do that would probably be trivial. I am a bit unsure if that
condensing should happen as a addon script which reprocesses the file,
when the colr/ file is created by 'r.colors -e', or when the color rules
are read in upon d.rast. (or some combination of the above) It would seem like a nice enhancement to have that happen though.

I notice that even in my current -1.5 thru 146m map '-e' produces some redundant entries.

Can you test the attached patch?

--
Glynn Clements <glynn@gclements.plus.com>

(attachments)

color_xform.c.diff (1.35 KB)

Hamish wrote:
> I just created a v.surf.rst elevation surface from some point data
> which has a few negative values in it. After creating the surface I
> set the colors with 'r.colors -e color=bcyr'. It looks really nice
> but there are white holes in the d.rast display.
>
> r.info -r shows:
> min=-1.510814
> max=146.157700
>
> The colr/ file doesn't go to one integer beyond the full range:
> ----------------------------------
> % -1 145
> nv:255
> *:255
> -1:0:0:255 0:0:94:255
> ...
> 144:255:1:0 145:255:1:0
> ----------------------------------
>
> ??

Glynn:

I didn't realise that "r.colors -e" even worked on FP data;
"r.colors color=grey.eq" doesn't.

Ultimately, the code is dependent upon the categories returned by
G_next_cell_stat(). If it only returns up to category 145, then the
colour table will only extend to that value.

> min=-1.510814
> max=146.157700

..

> % -1 145

I could assume the -1.51 -> -1 problem was due to a cast to CELL when
using a FP map with "*_cell_stat". The 146.15 vs 145 is a bit weirder,
cast + missing the last iteration? No idea how to make it extend to be
"% -2 147" for FP maps with G_next_cell_stat().

> I also noticed for another map 'r.colors -e' made a massive and slow
> colr/ file (1 for each cat) when the raster values were +-250000. That
> was a diff map centered on zero and I was using r.colors.stddev (wiki
> addons) to set the colors. While it had those huge outliers 95% of the
> points were +-5 (hence using the -e flag to get useful colors). It
> took impossibly long to render of course, so I looked in the colr/
> file and discovered it was huge.
>
> Ok, maybe that's a feature of the -e method. But what I saw that was
> interesting was that between -265000 and -120 all the R:G:B values
> were identical, ie all those thousands and thousands of rules could
> have been condensed into a single "-265000:R:G:B -120:R:G:B". And
> that a little program to do that would probably be trivial. I am a
> bit unsure if that condensing should happen as a addon script which
> reprocesses the file, when the colr/ file is created by 'r.colors
> -e', or when the color rules are read in upon d.rast. (or some
> combination of the above) It would seem like a nice enhancement to
> have that happen though.
>
> I notice that even in my current -1.5 thru 146m map '-e' produces
> some redundant entries.

Can you test the attached patch?

for a random lidar test map, the colr/ file is 21349 lines long unpatched
and 1497 lines long patched. Nice. There is still room for improvement
though:

[color_xform.c unpatched]
% -14900 1462400
nv:255
*:255
-14900:0:0:255 -14100:0:0:255
-14100:0:0:255 -14000:0:0:255
-14000:0:0:255 -13800:0:0:255
-13800:0:0:255 -13700:0:0:255
-13700:0:0:255 -13600:0:0:255
-13600:0:0:255 -13400:0:0:255
-13400:0:0:255 -13300:0:0:255
-13300:0:0:255 -13200:0:0:255
-13200:0:0:255 -13100:0:0:255
-13100:0:0:255 -13000:0:0:255
-13000:0:0:255 -12900:0:0:255
...

[color_xform.c patched]
% -14900 1462400
nv:255
*:25
-14900:0:0:255 -10500:0:0:255
-10500:0:0:255 -10400:0:1:255
-10400:0:1:255 -9800:0:1:255
-9800:0:1:255 -9750:0:2:255
-9750:0:2:255 -9350:0:2:255
-9350:0:2:255 -9300:0:3:255
-9300:0:3:255 -9000:0:3:255
-9000:0:3:255 -8950:0:4:255
-8950:0:4:255 -8700:0:4:255
-8700:0:4:255 -8650:0:5:255
-8650:0:5:255 -8450:0:5:255
-8450:0:5:255 -8400:0:6:255
-8400:0:6:255 -8300:0:6:255
-8300:0:6:255 -8200:0:7:255
-8200:0:7:255 -8050:0:7:255
-8050:0:7:255 -8000:0:8:255
-8000:0:8:255 -7900:0:8:255
-7900:0:8:255 -7850:0:9:255
-7850:0:9:255 -7700:0:9:255
...

it seems that every second line is redundant.

aside: I hadn't noticed the *:color [undefined, does this correspond to
"default" in G_parse_color_rule()?], invert, and shift:n colr/ rules
abilities before. invert is now covered by the r.colors -n flag using
another method, is 'shift' of any use?

Hamish

Hamish wrote:
['r.colors -e color=bcyr' for a FP map]

[color_xform.c patched]
% -14900 1462400
nv:255
*:25
-14900:0:0:255 -10500:0:0:255
-10500:0:0:255 -10400:0:1:255
-10400:0:1:255 -9800:0:1:255
-9800:0:1:255 -9750:0:2:255
-9750:0:2:255 -9350:0:2:255
-9350:0:2:255 -9300:0:3:255
-9300:0:3:255 -9000:0:3:255
-9000:0:3:255 -8950:0:4:255
-8950:0:4:255 -8700:0:4:255
-8700:0:4:255 -8650:0:5:255
-8650:0:5:255 -8450:0:5:255
-8450:0:5:255 -8400:0:6:255
-8400:0:6:255 -8300:0:6:255
-8300:0:6:255 -8200:0:7:255
-8200:0:7:255 -8050:0:7:255
-8050:0:7:255 -8000:0:8:255
-8000:0:8:255 -7900:0:8:255
-7900:0:8:255 -7850:0:9:255
-7850:0:9:255 -7700:0:9:255
...

it seems that every second line is redundant.

actually, it has made long series of steady color followed by the color
transition all occurring in a sudden step?

(I said this was LIDAR elevation data, actually it is elev*10000. as
r.surf.idw,.idw2,... only output CELL maps, I was r.mapcalc'ing the
result back to FP)

Hamish

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

Hamish wrote:

> > I just created a v.surf.rst elevation surface from some point data
> > which has a few negative values in it. After creating the surface I
> > set the colors with 'r.colors -e color=bcyr'. It looks really nice
> > but there are white holes in the d.rast display.
> >
> > r.info -r shows:
> > min=-1.510814
> > max=146.157700
> >
> > The colr/ file doesn't go to one integer beyond the full range:
> > ----------------------------------
> > % -1 145
> > nv:255
> > *:255
> > -1:0:0:255 0:0:94:255
> > ...
> > 144:255:1:0 145:255:1:0
> > ----------------------------------
> >
> > ??

Glynn:
> I didn't realise that "r.colors -e" even worked on FP data;
> "r.colors color=grey.eq" doesn't.
>
> Ultimately, the code is dependent upon the categories returned by
> G_next_cell_stat(). If it only returns up to category 145, then the
> colour table will only extend to that value.

> > min=-1.510814
> > max=146.157700
..
> > % -1 145

I could assume the -1.51 -> -1 problem was due to a cast to CELL when
using a FP map with "*_cell_stat". The 146.15 vs 145 is a bit weirder,
cast + missing the last iteration? No idea how to make it extend to be
"% -2 147" for FP maps with G_next_cell_stat().

It's more complex than a simple cast. For computing statistics,
r.colors reads the map as CELL (as that's all that Cell_stats
supports). The map will be converted using G_quant_get_cell_value()
(lib/gis/quant.c).

> > I also noticed for another map 'r.colors -e' made a massive and slow
> > colr/ file (1 for each cat) when the raster values were +-250000. That
> > was a diff map centered on zero and I was using r.colors.stddev (wiki
> > addons) to set the colors. While it had those huge outliers 95% of the
> > points were +-5 (hence using the -e flag to get useful colors). It
> > took impossibly long to render of course, so I looked in the colr/
> > file and discovered it was huge.
> >
> > Ok, maybe that's a feature of the -e method. But what I saw that was
> > interesting was that between -265000 and -120 all the R:G:B values
> > were identical, ie all those thousands and thousands of rules could
> > have been condensed into a single "-265000:R:G:B -120:R:G:B". And
> > that a little program to do that would probably be trivial. I am a
> > bit unsure if that condensing should happen as a addon script which
> > reprocesses the file, when the colr/ file is created by 'r.colors
> > -e', or when the color rules are read in upon d.rast. (or some
> > combination of the above) It would seem like a nice enhancement to
> > have that happen though.
> >
> > I notice that even in my current -1.5 thru 146m map '-e' produces
> > some redundant entries.
>
> Can you test the attached patch?

for a random lidar test map, the colr/ file is 21349 lines long unpatched
and 1497 lines long patched. Nice. There is still room for improvement
though:

[color_xform.c unpatched]
% -14900 1462400
nv:255
*:255
-14900:0:0:255 -14100:0:0:255
-14100:0:0:255 -14000:0:0:255
-14000:0:0:255 -13800:0:0:255
-13800:0:0:255 -13700:0:0:255
-13700:0:0:255 -13600:0:0:255
-13600:0:0:255 -13400:0:0:255
-13400:0:0:255 -13300:0:0:255
-13300:0:0:255 -13200:0:0:255
-13200:0:0:255 -13100:0:0:255
-13100:0:0:255 -13000:0:0:255
-13000:0:0:255 -12900:0:0:255
...

[color_xform.c patched]
% -14900 1462400
nv:255
*:25
-14900:0:0:255 -10500:0:0:255
-10500:0:0:255 -10400:0:1:255
-10400:0:1:255 -9800:0:1:255
-9800:0:1:255 -9750:0:2:255
-9750:0:2:255 -9350:0:2:255
-9350:0:2:255 -9300:0:3:255
-9300:0:3:255 -9000:0:3:255
-9000:0:3:255 -8950:0:4:255
-8950:0:4:255 -8700:0:4:255
-8700:0:4:255 -8650:0:5:255
-8650:0:5:255 -8450:0:5:255
-8450:0:5:255 -8400:0:6:255
-8400:0:6:255 -8300:0:6:255
-8300:0:6:255 -8200:0:7:255
-8200:0:7:255 -8050:0:7:255
-8050:0:7:255 -8000:0:8:255
-8000:0:8:255 -7900:0:8:255
-7900:0:8:255 -7850:0:9:255
-7850:0:9:255 -7700:0:9:255
...

it seems that every second line is redundant.

In this case, yes, as only one component changes at each step, and
only changes by one. If the colour table has any discontinuities,
eliminating the flat sections would make a significant difference.

Ultimately, this should still be an improvement over the previous
output. For now, I'm more concerned with it being right than getting
further efficiency.

aside: I hadn't noticed the *:color [undefined, does this correspond to
"default" in G_parse_color_rule()?]

Yes.

invert, and shift:n colr/ rules abilities before. invert is now
covered by the r.colors -n flag using another method,

This is what r.colors -n uses. It calls G_invert_colors(), which
inverts the "invert" field in the Colors structure, which corresponds
to the "invert" keyword in the file.

is 'shift' of any use?

I don't know. If it's present, all values are shifted by this amount
and (if necessary) wrapped into the range of the colour table before
they are looked up.

--
Glynn Clements <glynn@gclements.plus.com>