[GRASS5] [bug #1502] (grass) r.mapcalc error

Justin_Hickey_via_RT · January 9, 2003, 8:21am

this bug's URL: http://intevation.de/rt/webrt?serial_num=1502
-------------------------------------------------------------------------

Subject: r.mapcalc error

Platform: GNU/Linux/i386
grass obtained from: Trento Italy site
grass binary for platform: Compiled from Sources

GRASS 5.0.1(cvs:2003/1/3) compiled under Redhat Linux 7.2

I found the following bugs(?):

Error Message: Segmentation Error
Command : r.mapcalc "crange=@cmax-@cmin"
cmax(raster) : created by r.statistics command
                 base(integer) : include 12737 categoriws
                 cover(integer): DEM
                 method : max
cmin(raster) : created by r.statistics command
                 base(integer) : include 12737 categoriws(same as cmax)
                 cover(integer): DEM
                 method : min
All raster data have 2400x3600 cells with 0.5m resolution under UTM corrdinates.

Kazukiyo Yamamoto:
Graduate School of Bioagricultural Sciences
School of agricultural Sciences
Nagoya University

-------------------------------------------- Managed by Request Tracker

Markus_Neteler1 · January 10, 2003, 5:31pm

On Thu, Jan 09, 2003 at 09:21:17AM +0100, Request Tracker wrote:

this bug's URL: http://intevation.de/rt/webrt?serial_num=1502
-------------------------------------------------------------------------

Subject: r.mapcalc error

Platform: GNU/Linux/i386
grass obtained from: Trento Italy site
grass binary for platform: Compiled from Sources

GRASS 5.0.1(cvs:2003/1/3) compiled under Redhat Linux 7.2

I found the following bugs(?):

Error Message: Segmentation Error
Command : r.mapcalc "crange=@cmax-@cmin"
cmax(raster) : created by r.statistics command
                 base(integer) : include 12737 categoriws
                 cover(integer): DEM
                 method : max
cmin(raster) : created by r.statistics command
                 base(integer) : include 12737 categoriws(same as cmax)
                 cover(integer): DEM
                 method : min
All raster data have 2400x3600 cells with 0.5m resolution under UTM corrdinates.

Kazukiyo Yamamoto:
Graduate School of Bioagricultural Sciences
School of agricultural Sciences
Nagoya University

The problem appears also for me.

r.mapcalc "test=@map"
Segmentation fault

A fix were great

Markus

Glynn_Clements · January 10, 2003, 11:16pm

Markus Neteler wrote:

> I found the following bugs(?):
>
> Error Message: Segmentation Error
> Command : r.mapcalc "crange=@cmax-@cmin"

The problem appears also for me.

r.mapcalc "test=@map"
Segmentation fault

Not for me, unfortunately. Can you reproduce the problem with a map
from one of the sample locations (spearfish, global, etopo5 etc)? If
not, can you provide a *small* location (in terms of file size) which
demonstrates the problem?

Failing that, can you get a backtrace? Either enable core files then
use "gdb -c core" on the resulting core file, or run r.mapcalc under
gdb, e.g.

  $ gdb /usr/local/grass5/etc/bin/cmd/r.mapcalc
  ...
  > run test=@map
  ...
  > where

--
Glynn Clements <glynn.clements@virgin.net>

Markus_Neteler1 · January 11, 2003, 4:31pm

On Fri, Jan 10, 2003 at 11:16:39PM +0000, Glynn Clements wrote:

Markus Neteler wrote:

> > I found the following bugs(?):
> >
> > Error Message: Segmentation Error
> > Command : r.mapcalc "crange=@cmax-@cmin"

> The problem appears also for me.
>
> r.mapcalc "test=@map"
> Segmentation fault

Not for me, unfortunately. Can you reproduce the problem with a map
from one of the sample locations (spearfish, global, etopo5 etc)? If
not, can you provide a *small* location (in terms of file size) which
demonstrates the problem?

Failing that, can you get a backtrace? Either enable core files then
use "gdb -c core" on the resulting core file, or run r.mapcalc under
gdb, e.g.

  $ gdb /usr/local/grass5/etc/bin/cmd/r.mapcalc
  ...
  > run test=@map
  ...
  > where

I can try that on Monday only. The map "map" was generated with r.in.shape
and contained floating point values as category labels (the category numbers
were the polygon IDs).

Markus

Markus_Neteler1 · January 13, 2003, 2:22pm

On Fri, Jan 10, 2003 at 11:16:39PM +0000, Glynn Clements wrote:
[...]

> r.mapcalc "test=@map"
> Segmentation fault

Not for me, unfortunately. Can you reproduce the problem with a map
from one of the sample locations (spearfish, global, etopo5 etc)? If
not, can you provide a *small* location (in terms of file size) which
demonstrates the problem?

Failing that, can you get a backtrace? Either enable core files then
use "gdb -c core" on the resulting core file, or run r.mapcalc under
gdb, e.g.

Here we are:

Program received signal SIGSEGV, Segmentation fault.
G_quant_get_cell_value (q=0x80a9d28, dcellVal=0) at quant.c:700
/hardmnt/thuille1/ssi/cvsgrass_exp/src/libes/gis/quant.c:700:19329:beg:0x80641ad
(gdb)

I have stored the location (320kb) here:

http://mpa.itc.it/markus/tmp/rmapcalcUTM.tar.gz

The command crashing is:

r.mapcalc "test=@census1994dryUTM34S"
6%
Segmentation fault

Thanks for looking into this. The map was imported with r.in.shape.

Markus

Glynn_Clements · January 14, 2003, 1:59am

Markus Neteler wrote:

The command crashing is:

r.mapcalc "test=@census1994dryUTM34S"
6%
Segmentation fault

Thanks for looking into this. The map was imported with r.in.shape.

OK. The segfault occurs when G_get_cat() is called for an invalid
category (in this case 0; the cats table starts at 1). However, the
problem actually occurs when G__quant_get_rule_for_d_raster_val()
returns NULL (no quantisation rule matches the argument), and
G_quant_get_cell_value() tries to dereference the NULL pointer.

The problem with r.mapcalc is due to the algorithm which is used for
caching the results of G_get_cat() (in translate_from_cats()). Results
are stored in a btree; however, rather than storing individual values,
blocks of 64 values are stored. When the code is called for a value in
the range 1-63, it attempts to retrieve the values for the range 0-63,
resulting in an error.

The algorithm was copied verbatim from the original r.mapcalc, and I
would guess that it would have the same problem.

There are a variety of possible fixes.

1. Change G_quant_get_cell_value() to handle the case where
G__quant_get_rule_for_d_raster_val() returns NULL. I'm not sure what
it should return in this case (although it shouldn't segfault). In
this particular case (r.mapcalc) it shouldn't actually matter what was
returned, as the value should never actually be used.

2. Change translate_from_cats() to cache individual values in the
btree, rather than blocks. However, this may incur a substantial
penalty in terms of speed or memory usage or both. Discarding the
btree altogether would eliminate any memory issues, but the speed
penalty would probably be horrendous (the quantisation table lookup is
a linear search).

3. Change translate_from_cats() to trap invalid categories which would
cause a segfault. However, it isn't entirely clear what the test
should be; there's a fair amount of code between the call to
G_get_cat() and the quantisation table lookup.

--
Glynn Clements <glynn.clements@virgin.net>

Markus_Neteler1 · January 14, 2003, 9:04am

On Tue, Jan 14, 2003 at 01:59:49AM +0000, Glynn Clements wrote:

Markus Neteler wrote:

> The command crashing is:
>
> r.mapcalc "test=@census1994dryUTM34S"
> 6%
> Segmentation fault
>
> Thanks for looking into this. The map was imported with r.in.shape.

OK. The segfault occurs when G_get_cat() is called for an invalid
category (in this case 0; the cats table starts at 1).

Excuse me, but this is not clear to me. The cats file starts with
1:0.0000
Do you mean that 0.0000 is invalid?

However, the
problem actually occurs when G__quant_get_rule_for_d_raster_val()
returns NULL (no quantisation rule matches the argument), and
G_quant_get_cell_value() tries to dereference the NULL pointer.

The problem with r.mapcalc is due to the algorithm which is used for
caching the results of G_get_cat() (in translate_from_cats()). Results
are stored in a btree; however, rather than storing individual values,
blocks of 64 values are stored. When the code is called for a value in
the range 1-63, it attempts to retrieve the values for the range 0-63,
resulting in an error.

Ah, now I see the problem.

The algorithm was copied verbatim from the original r.mapcalc, and I
would guess that it would have the same problem.

Correct, I have just tested the old r.mapcalc which also fails.

There are a variety of possible fixes.

1. Change G_quant_get_cell_value() to handle the case where
G__quant_get_rule_for_d_raster_val() returns NULL. I'm not sure what
it should return in this case (although it shouldn't segfault). In
this particular case (r.mapcalc) it shouldn't actually matter what was
returned, as the value should never actually be used.

2. Change translate_from_cats() to cache individual values in the
btree, rather than blocks. However, this may incur a substantial
penalty in terms of speed or memory usage or both. Discarding the
btree altogether would eliminate any memory issues, but the speed
penalty would probably be horrendous (the quantisation table lookup is
a linear search).

My maps are rather big

3. Change translate_from_cats() to trap invalid categories which would
cause a segfault. However, it isn't entirely clear what the test
should be; there's a fair amount of code between the call to
G_get_cat() and the quantisation table lookup.

It is surprising that this problem was not yet already fixed some
years ago... Hope we can find a solution!

Markus

Glynn_Clements · January 14, 2003, 1:55pm

Markus Neteler wrote:

> > The command crashing is:
> >
> > r.mapcalc "test=@census1994dryUTM34S"
> > 6%
> > Segmentation fault
> >
> > Thanks for looking into this. The map was imported with r.in.shape.
>
> OK. The segfault occurs when G_get_cat() is called for an invalid
> category (in this case 0; the cats table starts at 1).

Excuse me, but this is not clear to me. The cats file starts with
1:0.0000
Do you mean that 0.0000 is invalid?

No; requesting the label for category zero is invalid.

> There are a variety of possible fixes.
>
> 1. Change G_quant_get_cell_value() to handle the case where
> G__quant_get_rule_for_d_raster_val() returns NULL. I'm not sure what
> it should return in this case (although it shouldn't segfault). In
> this particular case (r.mapcalc) it shouldn't actually matter what was
> returned, as the value should never actually be used.

Actually, returning null should be OK. This would seem to be the
simplest (and most efficient) solution.

--
Glynn Clements <glynn.clements@virgin.net>