[GRASS-dev] Re: [GRASS-user] low radiance values after i.atcor and i.topo.corr

Glynn_Clements1 · June 18, 2008, 9:41pm

Glynn Clements wrote:

> Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
> chance adding floating points (FCELL) to i.cluster?

I had a brief look at the code[1], and cannot see any obvious reason
why the values would need to be integers, so I'm assuming that it's
just a legacy of the days before FP support was added.

[1] Most of the code is actually in the lib/imagery/c_*.c files. That
should either be made part of i.cluster (nothing else uses it), or at
least split into a separate library.

I have committed these changes (splitting the cluster code off to a
separate library, and changing it to use DCELL instead of CELL) to the
SVN trunk.

I would appreciate it if someone who understands i.cluster could test
the current version.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · June 19, 2008, 10:12am

Nikos Alexandris wrote:

> I have committed these changes (splitting the cluster code off to a
> separate library, and changing it to use DCELL instead of CELL) to the
> SVN trunk.
>
> I would appreciate it if someone who understands i.cluster could test
> the current version.

I've compiled and run it yesterday. How can I tell that the changes are
effective? Should i.cluster NOT work before the changes with FP maps?

Well, the most important point is the absence of regressions, i.e.
anything which used to work should continue to work.

Beyond that, the old i.cluster (and underlying library code) only
handled integer values, so any FP maps were rounded to integers.

If you take an existing set of maps and scale down the values to the
point that rounding will introduce significant error, you should see a
significant difference between the old and new versions (i.e. the new
version should continue to work, the old one will produce bogus
results).

--
Glynn Clements <glynn@gclements.plus.com>

maning_sambale · June 19, 2008, 10:16am

On 6/19/08, Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

On Wed, 2008-06-18 at 22:41 +0100, Glynn Clements wrote:

Glynn Clements wrote:

> > Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
> > chance adding floating points (FCELL) to i.cluster?
>
> I had a brief look at the code[1], and cannot see any obvious reason
> why the values would need to be integers, so I'm assuming that it's
> just a legacy of the days before FP support was added.
>
> [1] Most of the code is actually in the lib/imagery/c_*.c files. That
> should either be made part of i.cluster (nothing else uses it), or at
> least split into a separate library.

I have committed these changes (splitting the cluster code off to a
separate library, and changing it to use DCELL instead of CELL) to the
SVN trunk.

I would appreciate it if someone who understands i.cluster could test
the current version.

I've compiled and run it yesterday. How can I tell that the changes are
effective? Should i.cluster NOT work before the changes with FP maps?
From my previous run of i.cluster

Very low values like this:
r.info -r tcor.lsat7_2000.toar.1
min=0.039443
max=0.442334

Gives out low separability:
means and standard deviations for 3 bands

means 0.00 0.00 0.00
stddev 0.06 0.00 0.00

initial means for each band

class 1 -0.05 0.00 0.00
class 2 -0.05 0.00 0.00
class 3 -0.05 0.00 0.00
class 4 -0.05 0.00 0.00
class 5 -0.04 0.00 0.00
class 6 -0.04 0.00 0.00
class 7 -0.04 0.00 0.00
class 8 -0.04 0.00 0.00
class 9 -0.03 0.00 0.00
class 10 -0.03 0.00 0.00

...
1 classes (convergence=100.0%)

class separability matrix

1

1 0

class means/stddev for each band

class 1 (2161)
means 0.00 0.00 0.00
stddev 0.00 0.00 0.00

Whereas recalculating using the same layers multiplied by 1000 with
r.mapcalc gives better results:
means and standard deviations for 3 bands

means 316.19 82.61 -78.36
stddev 63.13 54.24 41.51

initial means for each band

class 1 253.06 28.38 -119.87
class 2 255.64 30.59 -118.18
class 3 258.21 32.80 -116.49
class 4 260.79 35.02 -114.79
class 5 263.37 37.23 -113.10
class 6 265.94 39.44 -111.40

....

lass distribution
        172 15 20 21 41
         22 20 27 27 28
         29 36 27 37 45
         35 35 44 53 53
         52 50 57 54 45
         56 48 58 61 47
         58 59 65 50 47
         52 35 34 35 41
         41 34 27 25 27
         24 29 10 15 145

######## iteration 1 ###########
50 classes, 19.37% points stable
class distribution
         60 45 19 98 10
         18 71 13 8 112
        176 8 10 15 7
          6 9 23 124 22
          8 53 13 125 134
         10 14 10 52 8
        101 12 19 39 45
         12 4 178 119 11
          8 11 29 77 18

Can you report back your results? I hope I dont have to do r.mapcalc
and run i.cluster from the original layer values.

cheers,
maning
_______________________________________________

grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

--
|---------|----------------------------------------------------------|
| __.-._ |"Ohhh. Great warrior. Wars not make one great." -Yoda |
| '-._"7' |"Freedom is still the most radical idea of all" -N.Branden|
| /'.-c |Linux registered user #402901, http://counter.li.org/ |
| | /T |http://esambale.wikispaces.com|
| _)_/LI
|---------|----------------------------------------------------------|

neteler · June 21, 2008, 9:32am

On Wed, Jun 18, 2008 at 11:41 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Glynn Clements wrote:

> Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
> chance adding floating points (FCELL) to i.cluster?

I had a brief look at the code[1], and cannot see any obvious reason
why the values would need to be integers, so I'm assuming that it's
just a legacy of the days before FP support was added.

[1] Most of the code is actually in the lib/imagery/c_*.c files. That
should either be made part of i.cluster (nothing else uses it), or at
least split into a separate library.

I have committed these changes (splitting the cluster code off to a
separate library, and changing it to use DCELL instead of CELL) to the
SVN trunk.

I would appreciate it if someone who understands i.cluster could test
the current version.

(I have backported the changes to 6.4.svn for easier testing for me).

The i.cluster continues to work for CELL maps.

Here a test with the NC data set for FP maps:
  r.mapcalc b10=lsat7_2000_10/10000.
  r.mapcalc b20=lsat7_2000_20/10000.
  r.mapcalc b30=lsat7_2000_30/10000.
  r.mapcalc b40=lsat7_2000_40/10000.
  r.mapcalc b50=lsat7_2000_50/10000.
  r.mapcalc b70=lsat7_2000_70/10000.
  r.mapcalc b80=lsat7_2000_80/10000.
  i.group bgroup sub=bgroup in=b10,b20,b30,b40,b50,b70,b80
  i.cluster bgroup sub=bgroup sig=bsig class=10
Reading image ...
100%
Iteration 1: % Convergence: 77.47 (0s elapsed, 0s left)
Iteration 2: % Convergence: 78.21 (0s elapsed, 0s left)
Iteration 3: % Convergence: 83.12 (0s elapsed, 0s left)
Iteration 4: % Convergence: 91.00 (0s elapsed, 0s left)
Iteration 5: % Convergence: 94.15 (0s elapsed, 0s left)
Iteration 6: % Convergence: 95.67 (0s elapsed, 0s left)
Iteration 7: % Convergence: 96.44 (0s elapsed, 0s left)
Iteration 8: % Convergence: 96.54 (0s elapsed, 0s left)
Iteration 9: % Convergence: 96.54 (0s elapsed, 0s left)
Iteration 10: % Convergence: 96.57 (0s elapsed, 0s left)
Iteration 11: % Convergence: 97.01 (0s elapsed, 0s left)
Iteration 12: % Convergence: 97.04 (0s elapsed, 0s left)
Iteration 13: % Convergence: 97.18 (0s elapsed, 0s left)
Iteration 14: % Convergence: 97.35 (1s elapsed, 1s left)
Iteration 15: % Convergence: 97.72 (1s elapsed, 1s left)
Iteration 16: % Convergence: 97.93 (1s elapsed, 1s left)
Iteration 17: % Convergence: 98.12 (1s elapsed, 1s left)

i.maxlik bgroup sub=bgroup sig=bsig class=bclass
WARNING: signature 1 is not valid (ill-conditioned) - ignored.
WARNING: signature 2 is not valid (ill-conditioned) - ignored.
WARNING: signature 3 is not valid (ill-conditioned) - ignored.
WARNING: signature 4 is not valid (ill-conditioned) - ignored.
WARNING: signature 5 is not valid (ill-conditioned) - ignored.
WARNING: signature 6 is not valid (ill-conditioned) - ignored.
WARNING: signature 7 is not valid (ill-conditioned) - ignored.
WARNING: signature 8 is not valid (ill-conditioned) - ignored.
WARNING: signature 10 is not valid (ill-conditioned) - ignored.
i.maxlik ... 100%

d.rast bclass
-> all NULL

cat nc_spm_07/neteler/group/bgroup/subgroup/bgroup/sig/bsig
#produced by i.cluster
#Class 1
235
0.006890 0.005014 0.004109 0.001857 0.001746 0.001417 0.002585
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000001
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000001
#Class 2
2296
0.007022 0.005345 0.004651 0.006115 0.006222 0.003614 0.004977
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
-0.000000 -0.000000 -0.000000 0.000001
-0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 -0.000000 0.000000 0.000000
-0.000000 -0.000000 -0.000000 0.000000 0.000000 -0.000000 0.000001
#Class 3
2870
0.007302 0.005776 0.005326 0.006659 0.007960 0.004741 0.005350
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
0.000000 0.000000 -0.000000 0.000001
-0.000000 -0.000000 0.000000 -0.000000 0.000000
0.000000 0.000000 0.000000 -0.000000 0.000000 0.000000
0.000000 0.000000 -0.000000 0.000000 -0.000000 -0.000000 0.000001
#Class 4
1329
0.008375 0.006843 0.006841 0.006228 0.007832 0.005562 0.005556
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
-0.000000 -0.000000 -0.000000 0.000001
-0.000000 -0.000000 -0.000000 0.000000 0.000001
0.000000 0.000000 0.000000 -0.000000 0.000000 0.000000
-0.000000 0.000000 -0.000000 0.000000 0.000000 -0.000000 0.000001
#Class 5
848
0.007498 0.006350 0.005569 0.009792 0.009360 0.005067 0.007111
0.000000
0.000000 0.000000
0.000000 0.000000 0.000001
-0.000000 -0.000000 -0.000000 0.000002
0.000000 0.000000 0.000000 -0.000000 0.000001
0.000000 0.000000 0.000000 -0.000000 0.000001 0.000001
-0.000000 0.000000 0.000000 0.000000 0.000000 -0.000000 0.000001
#Class 6
1997
0.007739 0.006318 0.006402 0.006896 0.010058 0.006285 0.005598
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000001
-0.000000 -0.000000 0.000000 -0.000000 0.000001
0.000000 0.000000 0.000000 -0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 -0.000000 -0.000000 0.000001
#Class 7
990
0.008410 0.007342 0.007924 0.007967 0.012359 0.007995 0.006678
0.000000
0.000000 0.000000
0.000000 0.000000 0.000001
0.000000 0.000000 -0.000000 0.000001
-0.000000 -0.000000 0.000000 -0.000000 0.000001
0.000000 0.000000 0.000001 -0.000000 0.000001 0.000001
0.000000 0.000000 0.000000 0.000001 -0.000000 -0.000000 0.000001
#Class 8
1117
0.009603 0.008255 0.008899 0.006452 0.009602 0.007522 0.006170
0.000001
0.000000 0.000000
0.000001 0.000001 0.000001
-0.000000 -0.000000 -0.000000 0.000001
-0.000000 0.000000 0.000000 0.000000 0.000001
0.000000 0.000000 0.000000 -0.000000 0.000000 0.000001
-0.000000 0.000000 0.000000 0.000000 0.000000 -0.000000 0.000001
#Class 9
620
0.011070 0.010236 0.012072 0.007513 0.013108 0.010484 0.007491
0.000002
0.000002 0.000002
0.000001 0.000002 0.000002
-0.000000 0.000000 0.000000 0.000001
-0.000001 -0.000000 0.000001 0.000001 0.000003
0.000001 0.000001 0.000001 0.000000 0.000002 0.000003
0.000000 0.000000 0.000000 0.000000 0.000000 -0.000000 0.000003
#Class 10
88
0.017001 0.016265 0.018767 0.010181 0.018973 0.016527 0.010624
0.000014
0.000014 0.000014
0.000012 0.000013 0.000013
0.000007 0.000008 0.000007 0.000005
0.000004 0.000005 0.000005 0.000003 0.000010
-0.000000 -0.000000 0.000001 -0.000001 0.000011 0.000018
0.000004 0.000005 0.000004 0.000003 0.000002 -0.000002 0.000014

Maybe my example is nonsense?

Markus

Glynn_Clements1 · June 21, 2008, 11:47am

Markus Neteler wrote:

>> > Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
>> > chance adding floating points (FCELL) to i.cluster?
>>
>> I had a brief look at the code[1], and cannot see any obvious reason
>> why the values would need to be integers, so I'm assuming that it's
>> just a legacy of the days before FP support was added.
>>
>> [1] Most of the code is actually in the lib/imagery/c_*.c files. That
>> should either be made part of i.cluster (nothing else uses it), or at
>> least split into a separate library.
>
> I have committed these changes (splitting the cluster code off to a
> separate library, and changing it to use DCELL instead of CELL) to the
> SVN trunk.
>
> I would appreciate it if someone who understands i.cluster could test
> the current version.

(I have backported the changes to 6.4.svn for easier testing for me).

The i.cluster continues to work for CELL maps.

Here a test with the NC data set for FP maps:

Maybe my example is nonsense?

Well, I don't actually understand what i.cluster does, so I couldn't
tell "sensible" output from garbage.

However:

1. The behaviour for integer maps should remain unchanged.
2. Simply converting an integer map to FP shouldn't affect the
results.

If the above aren't true, then I've introduced a bug somewhere.

Beyond that, is i.cluster linear? IOW, if you scale all inputs by a
constant factor, should you get "equivalent" results?

If so, then the new i.cluster should behave as expected while the old
one would produce bogus results if the values are scaled down such
that the error introduced by rounding becomes significant.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · June 21, 2008, 12:24pm

On Sat, Jun 21, 2008 at 1:47 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

>> > Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
>> > chance adding floating points (FCELL) to i.cluster?
>>
>> I had a brief look at the code[1], and cannot see any obvious reason
>> why the values would need to be integers, so I'm assuming that it's
>> just a legacy of the days before FP support was added.
>>
>> [1] Most of the code is actually in the lib/imagery/c_*.c files. That
>> should either be made part of i.cluster (nothing else uses it), or at
>> least split into a separate library.
>
> I have committed these changes (splitting the cluster code off to a
> separate library, and changing it to use DCELL instead of CELL) to the
> SVN trunk.
>
> I would appreciate it if someone who understands i.cluster could test
> the current version.

(I have backported the changes to 6.4.svn for easier testing for me).

The i.cluster continues to work for CELL maps.

Here a test with the NC data set for FP maps:

Maybe my example is nonsense?

Well, I don't actually understand what i.cluster does, so I couldn't
tell "sensible" output from garbage.

It does something like this:
http://en.wikipedia.org/wiki/Cluster_analysis

However:

1. The behaviour for integer maps should remain unchanged.

Confirmed.

2. Simply converting an integer map to FP shouldn't affect the
results.

I suppose so.

If the above aren't true, then I've introduced a bug somewhere.

No bug in the upper part. No bug at all, but missing feature (see below).

Beyond that, is i.cluster linear? IOW, if you scale all inputs by a
constant factor, should you get "equivalent" results?

Indeed: results are simply divided by 10000. in this case.
Since the precision doesn't suffice, it fails as all gets 0.00000x.

First class of integer map:
#produced by i.cluster
#Class 1
235
68.897872 50.136170 41.085106 18.565957 17.463830 14.174468 25.851064
9.425423
10.432770 18.562575
14.645481 23.813148 37.283324
8.686252 12.072177 21.374704 45.503110
11.581760 12.654519 23.242408 48.065430 74.796763
9.086270 8.638534 15.972268 27.007674 44.474286 32.136097
8.065921 9.900709 18.880251 28.033370 33.176214 19.970540 55.734133

First class of FP map (integer map divided by 10000. as in my example):
#produced by i.cluster
#Class 1
235
0.006890 0.005014 0.004109 0.001857 0.001746 0.001417 0.002585
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000001
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000001

What's needed since we cannot change the format is a normalization
related to the range (AFAIK in i.cluster/print2.c).

Markus

mlennert · June 21, 2008, 1:11pm

On 21/06/08 14:24, Markus Neteler wrote:

On Sat, Jun 21, 2008 at 1:47 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

Thanks! Multiplying by 1000 with r.mapcalc gives better results. Any
chance adding floating points (FCELL) to i.cluster?

I had a brief look at the code[1], and cannot see any obvious reason
why the values would need to be integers, so I'm assuming that it's
just a legacy of the days before FP support was added.

[1] Most of the code is actually in the lib/imagery/c_*.c files. That
should either be made part of i.cluster (nothing else uses it), or at
least split into a separate library.

I have committed these changes (splitting the cluster code off to a
separate library, and changing it to use DCELL instead of CELL) to the
SVN trunk.

I would appreciate it if someone who understands i.cluster could test
the current version.

(I have backported the changes to 6.4.svn for easier testing for me).

The i.cluster continues to work for CELL maps.

Here a test with the NC data set for FP maps:
Maybe my example is nonsense?

Well, I don't actually understand what i.cluster does, so I couldn't
tell "sensible" output from garbage.

It does something like this:
http://en.wikipedia.org/wiki/Cluster_analysis

More precisely it does a k-means style of cluster analysis, AFAICT. It is used in image classification to reduce the number of elements to class from the total of individual pixels to a more limited number of pixel clusters.

What's needed since we cannot change the format is a normalization
related to the range (AFAIK in i.cluster/print2.c).

After a rapid look: isn't the problem rather in lib/imagery/sig.c, function I_write_signatures, where the fprintf statements use a simple %f ?

Moritz

Glynn_Clements1 · June 21, 2008, 4:39pm

Markus Neteler wrote:

>> > I would appreciate it if someone who understands i.cluster could test
>> > the current version.
>>
>> (I have backported the changes to 6.4.svn for easier testing for me).
>>
>> The i.cluster continues to work for CELL maps.
>>
>> Here a test with the NC data set for FP maps:
>
>> Maybe my example is nonsense?
>
> Well, I don't actually understand what i.cluster does, so I couldn't
> tell "sensible" output from garbage.

It does something like this:
http://en.wikipedia.org/wiki/Cluster_analysis

Oh, I'm familiar with the general concept, k-means, etc. I'm just not
familiar with the specifics of i.cluster (e.g. whether it's finding
clusters in geographic space or colour space, what the output
signifies, etc).

> However:
>
> 1. The behaviour for integer maps should remain unchanged.

Confirmed.

> 2. Simply converting an integer map to FP shouldn't affect the
> results.

I suppose so.

> If the above aren't true, then I've introduced a bug somewhere.

No bug in the upper part. No bug at all, but missing feature (see below).

> Beyond that, is i.cluster linear? IOW, if you scale all inputs by a
> constant factor, should you get "equivalent" results?

Indeed: results are simply divided by 10000. in this case.
Since the precision doesn't suffice, it fails as all gets 0.00000x.

First class of integer map:
#produced by i.cluster
#Class 1
235
68.897872 50.136170 41.085106 18.565957 17.463830 14.174468 25.851064
9.425423
10.432770 18.562575
14.645481 23.813148 37.283324
8.686252 12.072177 21.374704 45.503110
11.581760 12.654519 23.242408 48.065430 74.796763
9.086270 8.638534 15.972268 27.007674 44.474286 32.136097
8.065921 9.900709 18.880251 28.033370 33.176214 19.970540 55.734133

First class of FP map (integer map divided by 10000. as in my example):
#produced by i.cluster
#Class 1
235
0.006890 0.005014 0.004109 0.001857 0.001746 0.001417 0.002585
0.000000
0.000000 0.000000
0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000001
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000001

What's needed since we cannot change the format

No? If the format was changed to %g (i.e. like %f for "typical"
values, like %e for particularly large or small values), it would
remain compatible with anything using scanf("%f"), atof(), strtod()
etc, as those functions all understand exponential notation.

[About the only plausible situation where switching from %f to %g or
%e can cause problems is when code tries to match FP literals against
a regexp such as "[0-9]+\.[0-9]*".]

is a normalization related to the range (AFAIK in
i.cluster/print2.c).

You would need to add an explicit scale= (e.g.) option. As there's no
way to pass the scale factor out to the user, the user would have to
pass the scale factor in.

OTOH, a scale= option and switching to %g aren't mutually exclusive;
but %g would largely eliminate the need for scale=, and would probably
be more useful.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · June 21, 2008, 6:24pm

On Sat, Jun 21, 2008 at 6:39 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:
I'm just not familiar with the specifics of i.cluster (e.g. whether it's finding
clusters in geographic space or colour space, what the output
signifies, etc).

It takes several maps and builds up a multi-dimensional feature space
from the pixel (spectral, ...) values. Those are then clustered.
...

What's needed since we cannot change the format

No? If the format was changed to %g (i.e. like %f for "typical"
values, like %e for particularly large or small values), it would
remain compatible with anything using scanf("%f"), atof(), strtod()
etc, as those functions all understand exponential notation.

Ah, sounds good.

[About the only plausible situation where switching from %f to %g or
%e can cause problems is when code tries to match FP literals against
a regexp such as "[0-9]+\.[0-9]*".]

You would need to add an explicit scale= (e.g.) option. As there's no
way to pass the scale factor out to the user, the user would have to
pass the scale factor in.

OTOH, a scale= option and switching to %g aren't mutually exclusive;
but %g would largely eliminate the need for scale=, and would probably
be more useful.

If there is no scanning problem, I would appreciate to just increase
the precision and don't introduce a scale parameter. I guess it then
just works.

Markus

neteler · June 21, 2008, 7:06pm

On Sat, Jun 21, 2008 at 6:39 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

....

It does something like this:
http://en.wikipedia.org/wiki/Cluster_analysis

Oh, I'm familiar with the general concept, k-means, etc. I'm just not
familiar with the specifics of i.cluster (e.g. whether it's finding
clusters in geographic space or colour space, what the output
signifies, etc).

I have added a illustrative figure to imagery/i.cluster/. this may
explain the idea.

Markus

Glynn_Clements1 · June 21, 2008, 9:16pm

Markus Neteler wrote:

> I'm just not familiar with the specifics of i.cluster (e.g. whether it's finding
> clusters in geographic space or colour space, what the output
> signifies, etc).

It takes several maps and builds up a multi-dimensional feature space
from the pixel (spectral, ...) values. Those are then clustered.

Right. So the geographic position of each cell is ignored; a group of
rasters is just a "set" of points in "colour space" (i.e. each
coordinate is the intensity of a particular band)?

>> What's needed since we cannot change the format
>
> No? If the format was changed to %g (i.e. like %f for "typical"
> values, like %e for particularly large or small values), it would
> remain compatible with anything using scanf("%f"), atof(), strtod()
> etc, as those functions all understand exponential notation.

Ah, sounds good.

> [About the only plausible situation where switching from %f to %g or
> %e can cause problems is when code tries to match FP literals against
> a regexp such as "[0-9]+\.[0-9]*".]
>
> You would need to add an explicit scale= (e.g.) option. As there's no
> way to pass the scale factor out to the user, the user would have to
> pass the scale factor in.
>
> OTOH, a scale= option and switching to %g aren't mutually exclusive;
> but %g would largely eliminate the need for scale=, and would probably
> be more useful.

If there is no scanning problem, I would appreciate to just increase
the precision and don't introduce a scale parameter. I guess it then
just works.

It shouldn't be necessary to increase the precision. For %g, the
precision (default = 6) is the number of significant digits, not the
number of decimal places, even when exponential form isn't used:

printf("%f\n", 0.000100001);
=> 0.000100

printf("%f\n", 0.000099999);
=> 0.000100

printf("%g\n", 0.000100001);
=> 0.000100001

printf("%g\n", 0.000099999);
=> 9.9999e-05

So the default precision should be fine unless you actually get images
with intensity values accurate to 1PPM or better.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · June 21, 2008, 9:22pm

On Sat, Jun 21, 2008 at 11:16 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

> I'm just not familiar with the specifics of i.cluster (e.g. whether it's finding
> clusters in geographic space or colour space, what the output
> signifies, etc).

It takes several maps and builds up a multi-dimensional feature space
from the pixel (spectral, ...) values. Those are then clustered.

Right. So the geographic position of each cell is ignored; a group of
rasters is just a "set" of points in "colour space" (i.e. each
coordinate is the intensity of a particular band)?

Yes.
The i.gensiset/i.smap approach is different and also includes the spatial
position (hence it performs usually better). See also my other mail.

>> What's needed since we cannot change the format
>
> No? If the format was changed to %g (i.e. like %f for "typical"
> values, like %e for particularly large or small values), it would
> remain compatible with anything using scanf("%f"), atof(), strtod()
> etc, as those functions all understand exponential notation.

Ah, sounds good.

...

printf("%g\n", 0.000100001);
=> 0.000100001

printf("%g\n", 0.000099999);
=> 9.9999e-05

So the default precision should be fine unless you actually get images
with intensity values accurate to 1PPM or better.

Sounds very promising. If you could indicate where to change, I can
run again my test examples.

Markus

Glynn_Clements1 · June 22, 2008, 1:46pm

Markus Neteler wrote:

> printf("%g\n", 0.000100001);
> => 0.000100001
>
> printf("%g\n", 0.000099999);
> => 9.9999e-05
>
> So the default precision should be fine unless you actually get images
> with intensity values accurate to 1PPM or better.

Sounds very promising. If you could indicate where to change, I can
run again my test examples.

To change the format of the signature files, replace %f with %g in
I_write_signatures(), in lib/imagery/sig.c.

To change the format of the report file, replace occurrences of %6.2f
with e.g. %g in the various imagery/i.cluster/print*.c files.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · June 22, 2008, 3:35pm

On Sun, Jun 22, 2008 at 3:46 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:...

To change the format of the signature files, replace %f with %g in
I_write_signatures(), in lib/imagery/sig.c.

I have tried that, of course it works.
But apparently i.maxlik needs a change, too, to read that properly (same
sig.c fie, I suppose). At least i.maxlik doesn't produce any better result
with the precision improved signatures.

To change the format of the report file, replace occurrences of %6.2f
with e.g. %g in the various imagery/i.cluster/print*.c files.

ok, thanks,
markus

Glynn_Clements1 · June 22, 2008, 6:28pm

Markus Neteler wrote:

> To change the format of the signature files, replace %f with %g in
> I_write_signatures(), in lib/imagery/sig.c.

I have tried that, of course it works.
But apparently i.maxlik needs a change, too, to read that properly (same
sig.c fie, I suppose). At least i.maxlik doesn't produce any better result
with the precision improved signatures.

i.maxlik just calls I_read_signatures(). That just uses fscanf("%f"),
which will read anything which was generated by %f, %e or %g (for the
*scanf family, %f, %e and %g are synonyms).

--
Glynn Clements <glynn@gclements.plus.com>

neteler · June 22, 2008, 7:02pm

On Sun, Jun 22, 2008 at 8:28 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

> To change the format of the signature files, replace %f with %g in
> I_write_signatures(), in lib/imagery/sig.c.

I have tried that, of course it works.

Submitted.

But apparently i.maxlik needs a change, too, to read that properly (same
sig.c fie, I suppose). At least i.maxlik doesn't produce any better result
with the precision improved signatures.

i.maxlik just calls I_read_signatures(). That just uses fscanf("%f"),
which will read anything which was generated by %f, %e or %g (for the
*scanf family, %f, %e and %g are synonyms).

Right. I found the problem: i.maxlik reads in as CELL. This needs a rewrite
to permit all common raster2D types.
Hacking in DCELL instead of CELL for reading my FP simulated maps,
i.maxlik works.

Are you willing to update i.maxlik for INT/FP support for the input maps?

Markus

neteler · June 22, 2008, 7:27pm

On Sun, Jun 22, 2008 at 9:02 PM, Markus Neteler <neteler@osgeo.org> wrote:

I found the problem: i.maxlik reads in as CELL. This needs a rewrite
to permit all common raster2D types.
Hacking in DCELL instead of CELL for reading my FP simulated maps,
i.maxlik works.

Are you willing to update i.maxlik for INT/FP support for the input maps?

The same applies unfortunately to
i.gensig
i.gensiset
i.smap/bouman/read_block.c
i.class

which also use G_get_c_raster_row().
Then the entire classification suite would be FP compliant. I ave already
updated the signature write function of i.gensigset.

Markus

Glynn_Clements1 · June 22, 2008, 11:14pm

Markus Neteler wrote:

> I found the problem: i.maxlik reads in as CELL. This needs a rewrite
> to permit all common raster2D types.
> Hacking in DCELL instead of CELL for reading my FP simulated maps,
> i.maxlik works.
>
> Are you willing to update i.maxlik for INT/FP support for the input maps?

The same applies unfortunately to
i.gensig
i.gensiset
i.smap/bouman/read_block.c
i.class

Changing i.maxlik looks straightforward enough, but the others are
quite a bit more complex.

i.class is a libraster-based GUI program (R_get_location_*), so its
life expectancy extends to roughly 5 minutes after 7.x gets
"indent"ed, so I don't particularly want to waste time there.

And i.smap is gibberish; e.g. write_img() calls G_is_c_null_value() on
"char"s, so that module probably hasn't worked since 4.x.

--
Glynn Clements <glynn@gclements.plus.com>

maning_sambale · June 23, 2008, 12:48am

Nikos,

thank you fo testing seems it's working OK now. I will test it myself later.

maning
On 6/20/08, Nikos Alexandris <nikos.alexandris@felis.uni-freiburg.de> wrote:

Not sure if this helps... (!)
---

r.univar change_pca_250m_06.239_07.242.2_test

100%
total null and non-null cells: 247720
total null cells: 162432

Of the non-null cells:
----------------------
n: 85288
minimum: -0.315021
maximum: 0.260252
range: 0.575273
mean: -0.000584902
mean of absolute values: 0.0129577
standard deviation: 0.0191773
variance: 0.00036777
variation coefficient: -3278.73 %
sum: -49.8850918202

r.univar change_pca_250m_06.239_07.242.1_test

100%
total null and non-null cells: 247720
total null cells: 162432

Of the non-null cells:
----------------------
n: 85288
minimum: -0.513804
maximum: 0.394813
range: 0.908617
mean: 0.0211992
mean of absolute values: 0.032389
standard deviation: 0.042701
variance: 0.00182337
variation coefficient: 201.428 %
sum: 1808.0350767233

cat sig_cluster_test

#produced by i.cluster
#Class 1
23
0.014645 -0.107214
0.000473
0.000026 0.000349
#Class 2
37
-0.014961 -0.076135
0.000157
0.000027 0.000139
#Class 3
52
0.002926 -0.056125
0.000124
-0.000012 0.000061
#Class 4
35
-0.028285 -0.040243
0.000049
0.000003 0.000105
#Class 5
73
-0.004751 -0.035473
0.000021
-0.000002 0.000035

[...]

--
|---------|----------------------------------------------------------|
| __.-._ |"Ohhh. Great warrior. Wars not make one great." -Yoda |
| '-._"7' |"Freedom is still the most radical idea of all" -N.Branden|
| /'.-c |Linux registered user #402901, http://counter.li.org/ |
| | /T |http://esambale.wikispaces.com|
| _)_/LI
|---------|----------------------------------------------------------|

neteler · June 23, 2008, 7:05am

On Mon, Jun 23, 2008 at 1:14 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

> I found the problem: i.maxlik reads in as CELL. This needs a rewrite
> to permit all common raster2D types.
> Hacking in DCELL instead of CELL for reading my FP simulated maps,
> i.maxlik works.
>
> Are you willing to update i.maxlik for INT/FP support for the input maps?

The same applies unfortunately to
i.gensig
i.gensiset
i.smap/bouman/read_block.c
i.class

Changing i.maxlik looks straightforward enough, but the others are
quite a bit more complex.

Funny, my i.maxlik was identical. I expected something more complex
but fine as is.
i.cluster/i.maxlik now work with FP data, I just made a classification
of slope/aspect maps and it does something apparently reasonable.

i.class is a libraster-based GUI program (R_get_location_*), so its
life expectancy extends to roughly 5 minutes after 7.x gets
"indent"ed, so I don't particularly want to waste time there.

ok, so we ignore that.

And i.smap is gibberish; e.g. write_img() calls G_is_c_null_value() on
"char"s, so that module probably hasn't worked since 4.x.

i.smap works, I have tested it recently and also others use it regularly.

Any opinions on
i.gensig
i.gensiset
? Both are used for supervised classification where you have training
areas instead of just looking at the pixels as does i.cluster.

Markus