[GRASS-dev] r.univar, ERROR G_realloc:

Pietro2 · December 6, 2013, 1:40am

Dear developers,

I tried to use r,univar in grass7 (r58395 on linux 64bit with 24Gb of
ram and 8Gb of swap), before to run the command I did a distclean, and
I configured with:

CFLAGS="-ggdb -Wall -Werror-implicit-function-declaration" ./configure ...

when I run:

r.univar map=Combabula_Nearmap.red zones=seg005_64_zones
percentile=90. output=red.csv -e --o

The module termanate with:

D1/1: G_find_raster(): name=MASK mapset=pietro
Current region rows: 28545, cols: 27645
ERROR: G_realloc: unable to allocate 68000 bytes of memory at
r.univar_main.c:327

Any idea?

Maybe is a stupid idea but since I had some problems also with
v.build, do you think that could be possible that the problem is due
not to the GRASS code, but to my physical memory that maybe is damaged
in some sector?
A damaged ram can lead to this result?

However, using gdb, I got:

G70> gdb `which r.univar`

GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>\.\.\.
Reading symbols from
/home/pietro/docdat/src/gis/grass/grass_trunk/dist.x86_64-unknown-linux-gnu/bin/r.univar...done.

(gdb) run map=Combabula_Nearmap.red zones=seg005_64_zones
percentile=90. output=red.csv -e --o

Starting program:
/home/pietro/docdat/src/gis/grass/grass_trunk/dist.x86_64-unknown-linux-gnu/bin/r.univar
map=Combabula_Nearmap.red zones=seg005_64_zones percentile=90.
output=red.csv -e --o
warning: no loadable sections found in added symbol-file
system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
D1/1: G_find_raster2(): name=seg005_64_zones mapset=
D1/1: G_find_raster2(): name=seg005_64_zones mapset=
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster(): name=MASK mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster(): name=MASK mapset=pietro
Current region rows: 28545, cols: 27645
ERROR: G_realloc: unable to allocate 68000 bytes of memory at
r.univar_main.c:327

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72ed3d9 in raise () from /usr/lib/libc.so.6

(gdb) bt full

#0 0x00007ffff72ed3d9 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1 0x00007ffff798603d in G_fatal_error (
    msg=0x7ffff799f668 "G_realloc: unable to allocate %lu bytes of
memory at %s:%d") at error.c:169
        busy = 1
        ap = {{gp_offset = 32, fp_offset = 48,
            overflow_arg_area = 0x7fffffffd7e0,
            reg_save_area = 0x7fffffffd720}}
#2 0x00007ffff7982180 in G__realloc (file=0x405f4e "r.univar_main.c",
    line=327, buf=0x0, n=68000) at alloc.c:135
        window = {format = 0, compressed = -1, rows = 28545, rows3 = 28545,
          cols = 27645, cols3 = 27645, depths = 1, proj = 1, zone = -55,
          ew_res = 0.52619081800000067, ew_res3 = 0.52619081999999995,
          ns_res = 0.52619081799999301, ns_res3 = 0.52619081999999995,
          tb_res = 1, north = 7104206.8011954101, south = 7089186.6842956003,
          east = 759827.49500820006, west = 745280.94984459004, top = 1,
          bottom = 0}
#3 0x00000000004023cf in process_raster (stats=0x7fffe17c4010, fd=1, fdz=0,
    region=0x7fffffffd970) at r.univar_main.c:327
        msize = 68000
        val = 129
        zone = 2400960
        ptr = 0x5b5b374
        zptr = 0x5b76374
        col = 24149
        rows = 28545
        cols = 27645
        map_type = 0
        value_sz = 4
        row = 24693
        raster_row = 0x5b43a20
        zoneraster_row = 0x5b5ea20
        n_zones = 2774561
#4 0x0000000000401e3f in main (argc=7, argv=0x7fffffffdb38)
    at r.univar_main.c:190
        rasters = 1
        region = {format = 0, compressed = -1, rows = 28545, rows3 = 28545,
          cols = 27645, cols3 = 27645, depths = 1, proj = 1, zone = -55,
          ew_res = 0.52619081800000067, ew_res3 = 0.52619081999999995,
          ns_res = 0.52619081799999301, ns_res3 = 0.52619081999999995,
          tb_res = 1, north = 7104206.8011954101, south = 7089186.6842956003,
          east = 759827.49500820006, west = 745280.94984459004, top = 1,
          bottom = 0}
        module = 0x7ffff7babd08 <state+40>
        stats = 0x7fffe17c4010
        p = 0x609b80
        z = 0x609ac0 "seg005_64_zones"
        fd = 1
        fdz = 0
        cell_type = 0
        min = 1
        max = 2774561
        zone_range = {min = 1, max = 2774561, first_time = 0}
        mapset = 0x60a320 "pietro"
        name = 0x609b60 "red.csv"
        map_type = 0
        __PRETTY_FUNCTION__ = "main"

As suggested by MarkusM in another thread, I set:

G70> export GRASS_VECTOR_LOWMEM=1

and then again with gdb:

G70> gdb `which r.univar`

GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>\.\.\.
Reading symbols from
/home/pietro/docdat/src/gis/grass/grass_trunk/dist.x86_64-unknown-linux-gnu/bin/r.univar...done.

(gdb) run map=Combabula_Nearmap.red zones=seg005_64_zones
percentile=90. output=red.csv -e --o

Starting program:
/home/pietro/docdat/src/gis/grass/grass_trunk/dist.x86_64-unknown-linux-gnu/bin/r.univar
map=Combabula_Nearmap.red zones=seg005_64_zones percentile=90.
output=red.csv -e --o
warning: no loadable sections found in added symbol-file
system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
D1/1: G_find_raster2(): name=seg005_64_zones mapset=
D1/1: G_find_raster2(): name=seg005_64_zones mapset=
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster(): name=MASK mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=seg005_64_zones mapset=pietro
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster2(): name=Combabula_Nearmap.red mapset=PERMANENT
D1/1: G_find_raster(): name=MASK mapset=pietro
Current region rows: 28545, cols: 27645
ERROR: G_realloc: unable to allocate 68000 bytes of memory at
r.univar_main.c:327

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72ed3d9 in raise () from /usr/lib/libc.so.6

(gdb) bt full

#0 0x00007ffff72ed3d9 in raise () from /usr/lib/libc.so.6
No symbol table info available.
#1 0x00007ffff798603d in G_fatal_error (
    msg=0x7ffff799f668 "G_realloc: unable to allocate %lu bytes of
memory at %s:%d") at error.c:169
        busy = 1
        ap = {{gp_offset = 32, fp_offset = 48,
            overflow_arg_area = 0x7fffffffd7d0,
            reg_save_area = 0x7fffffffd710}}
#2 0x00007ffff7982180 in G__realloc (file=0x405f4e "r.univar_main.c",
    line=327, buf=0x0, n=68000) at alloc.c:135
        window = {format = 0, compressed = -1, rows = 28545, rows3 = 28545,
          cols = 27645, cols3 = 27645, depths = 1, proj = 1, zone = -55,
          ew_res = 0.52619081800000067, ew_res3 = 0.52619081999999995,
          ns_res = 0.52619081799999301, ns_res3 = 0.52619081999999995,
          tb_res = 1, north = 7104206.8011954101, south = 7089186.6842956003,
          east = 759827.49500820006, west = 745280.94984459004, top = 1,
          bottom = 0}
#3 0x00000000004023cf in process_raster (stats=0x7fffe17c4010, fd=1, fdz=0,
    region=0x7fffffffd960) at r.univar_main.c:327
        msize = 68000
        val = 129
        zone = 2400960
        ptr = 0x5b5b374
        zptr = 0x5b76374
        col = 24149
        rows = 28545
        cols = 27645
        map_type = 0
        value_sz = 4
        row = 24693
        raster_row = 0x5b43a20
        zoneraster_row = 0x5b5ea20
        n_zones = 2774561
#4 0x0000000000401e3f in main (argc=7, argv=0x7fffffffdb28)
    at r.univar_main.c:190
        rasters = 1
        region = {format = 0, compressed = -1, rows = 28545, rows3 = 28545,
          cols = 27645, cols3 = 27645, depths = 1, proj = 1, zone = -55,
          ew_res = 0.52619081800000067, ew_res3 = 0.52619081999999995,
          ns_res = 0.52619081799999301, ns_res3 = 0.52619081999999995,
          tb_res = 1, north = 7104206.8011954101, south = 7089186.6842956003,
          east = 759827.49500820006, west = 745280.94984459004, top = 1,
          bottom = 0}
        module = 0x7ffff7babd08 <state+40>
        stats = 0x7fffe17c4010
        p = 0x609b80
        z = 0x609ac0 "seg005_64_zones"
        fd = 1
        fdz = 0
        cell_type = 0
        min = 1
        max = 2774561
        zone_range = {min = 1, max = 2774561, first_time = 0}
        mapset = 0x60a320 "pietro"
        name = 0x609b60 "red.csv"
        map_type = 0
        __PRETTY_FUNCTION__ = "main"

Glynn_Clements1 · December 6, 2013, 11:58pm

Pietro wrote:

I tried to use r,univar in grass7 (r58395 on linux 64bit with 24Gb of
ram and 8Gb of swap), before to run the command I did a distclean, and
I configured with:

CFLAGS="-ggdb -Wall -Werror-implicit-function-declaration" ./configure ...

when I run:

r.univar map=Combabula_Nearmap.red zones=seg005_64_zones
percentile=90. output=red.csv -e --o

The module termanate with:

D1/1: G_find_raster(): name=MASK mapset=pietro
Current region rows: 28545, cols: 27645
ERROR: G_realloc: unable to allocate 68000 bytes of memory at
r.univar_main.c:327

Don't use "r.univar -e", particularly for large maps. r.quantile and
r.statistics3 are far more efficient.

Maybe is a stupid idea but since I had some problems also with
v.build, do you think that could be possible that the problem is due
not to the GRASS code, but to my physical memory that maybe is damaged
in some sector?

That's unlikely. Is the OS recognising that you have 24 GiB of RAM?
Are you allowed to use all (or most) of it for a single process (check
the output from "ulimit -a")?

--
Glynn Clements <glynn@gclements.plus.com>

Pietro2 · December 7, 2013, 8:47am

On Fri, Dec 6, 2013 at 11:58 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

D1/1: G_find_raster(): name=MASK mapset=pietro
Current region rows: 28545, cols: 27645
ERROR: G_realloc: unable to allocate 68000 bytes of memory at
r.univar_main.c:327

Don't use "r.univar -e", particularly for large maps. r.quantile and
r.statistics3 are far more efficient.

ok, thanks I will look into it!

Maybe is a stupid idea but since I had some problems also with
v.build, do you think that could be possible that the problem is due
not to the GRASS code, but to my physical memory that maybe is damaged
in some sector?

That's unlikely. Is the OS recognising that you have 24 GiB of RAM?

yes it is.

Are you allowed to use all (or most) of it for a single process (check
the output from "ulimit -a")?

I think so:

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 30
file size (blocks, -f) unlimited
pending signals (-i) 192592
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 99
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 192592
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Actually running Memtest86, to test the memory bank I found hundred of
errors so I strongly believe that the problem is due to some physical
problem.

neteler · December 8, 2013, 9:37pm

On Sat, Dec 7, 2013 at 12:58 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:
...

Don't use "r.univar -e", particularly for large maps. r.quantile and
r.statistics3 are far more efficient.

It might be nice to define a threshold for "large maps" and issue in
case a G_message()/G_warning() to inform the user about the other
modules...

Markus

Pietro2 · December 8, 2013, 11:51pm

On Sunday 08 Dec 2013 22:37:34 Markus Neteler wrote:

On Sat, Dec 7, 2013 at 12:58 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:
...

> Don't use "r.univar -e", particularly for large maps. r.quantile
> and r.statistics3 are far more efficient.

It might be nice to define a threshold for "large maps" and issue in
case a G_message()/G_warning() to inform the user about the other
modules...

Today I tried to reduce the memory footprint of the module and I added
also: skewness, kurtosis and mode, and reduced some capabilities, I
will test it in the next days.

However r.statistics/2/3/r.quantile generate different results and are
not directly substitutes of r.univar (IMHO).

Pietro2 · December 10, 2013, 10:38pm

On Sunday 08 Dec 2013 23:51:22 you wrote:

> It might be nice to define a threshold for "large maps" and issue
> in case a G_message()/G_warning() to inform the user about the
> other modules...

Today I tried to reduce the memory footprint of the module and I
added also: skewness, kurtosis and mode, and reduced some
capabilities, I will test it in the next days.

OK, I pushed in grass-addons, at the moment it works only with zones
and with the flag -e (for extended statistics).

On my test case r.univar consume more than 15 GB before to crash, now
the module consume less than GB, and compute also some extra values.

Pietro

neteler · February 5, 2015, 1:48pm

Hi Pietro,

(back to an older email)

On Tue, Dec 10, 2013 at 11:38 PM, Pietro Zambelli <peter.zamb@gmail.com> wrote:

On Sunday 08 Dec 2013 23:51:22 you wrote:

> It might be nice to define a threshold for "large maps" and issue
> in case a G_message()/G_warning() to inform the user about the
> other modules...

Today I tried to reduce the memory footprint of the module and I
added also: skewness, kurtosis and mode, and reduced some
capabilities, I will test it in the next days.

we would be interested to have the "mode" calculation in trunk.

OK, I pushed in grass-addons, at the moment it works only with zones
and with the flag -e (for extended statistics).

On my test case r.univar consume more than 15 GB before to crash, now
the module consume less than GB, and compute also some extra values.

Sounds very good

Do you see a chance to get the changes into trunk?

thanks
Markus

Pietro2 · February 5, 2015, 2:06pm

On Thu, Feb 5, 2015 at 2:48 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Pietro,

(back to an older email)

On Tue, Dec 10, 2013 at 11:38 PM, Pietro Zambelli <peter.zamb@gmail.com> wrote:

On Sunday 08 Dec 2013 23:51:22 you wrote:

> It might be nice to define a threshold for "large maps" and issue
> in case a G_message()/G_warning() to inform the user about the
> other modules...

Today I tried to reduce the memory footprint of the module and I
added also: skewness, kurtosis and mode, and reduced some
capabilities, I will test it in the next days.

we would be interested to have the "mode" calculation in trunk.

OK, I pushed in grass-addons, at the moment it works only with zones
and with the flag -e (for extended statistics).

On my test case r.univar consume more than 15 GB before to crash, now
the module consume less than GB, and compute also some extra values.

Sounds very good

Do you see a chance to get the changes into trunk?

the module it is already in the grass-addons: r.univar2 [0].

if compared with r.univar in trunk, zones is a required parameter, it
should be not too difficult to modify r.univar2 to use just one zone
if the zones parameter is not provided by the user...

But at the moment I'm quite buisy with my thesis and work... so I'm
not sure to have time to look into it soon (with soon I mean before
April).

Let me know if it is ok for you.

Best regards

Pietro

[0] http://trac.osgeo.org/grass/browser/grass-addons/grass7/raster/r.univar2