[GRASS5] RE: [GRASSLIST:2288] Re: s.sample gets killed

GRASS GRASS developer
1

Hamish, Glynn,
as you predicted, the memory use grows until the swap space is full and then the program crashes.
I added the lines to sites.c and recompiled the whole thing, but unfortunately the result is the same. Instead of 'killed' I now get a 'Bus Error' message. More ideas?

Thank you for your help
Christoph

-----Original Message-----
From: Glynn Clements [mailto:glynn.clements@virgin.net]
Sent: Tue 1/20/2004 7:46 AM
To: Hamish
Cc: Deuchler, Christoph; GRASSLIST@baylor.edu; grass5@grass.itc.it
Subject: Re: [GRASSLIST:2288] Re: s.sample gets killed

Hamish wrote:

> I need to extract raster cell values for a large number (approx. 2.5
> million) of sites. Unfortunately s.sample gets killed after probing a
> few thousand sites without apparent reason. No error message is given,
> just 'killed'. Has anybody encountered the same problem and found a
> solution or workaround?

Sounds like you are running out of memory.

can you run "top" to watch the memory use while s.sample runs?
hit "M" while top is running to sort processes by memory use.

Maybe it is a memory leak -- does the memory use steadily grow as the
program runs or is it all allocated at once? How big does it get? Please
post results so we can fix it if need be.

It's almost certainly due to a memory leak in libgis.

s.sample calls G_readsites_xyz() in a loop; G_readsites_xyz() calls
G_site_new_struct() to allocate a Site structure and
G_site_free_struct() to free it.

G_site_new_struct() allocates the dim, str_att and dbl_att arrays, and
also allocates a buffer for each entry in the str_att array.
G_site_free_struct() frees these arrays, but *doesn't* free the
buffers in the str_att array.

The attached patch fixes that; however, it should probably be tested
before being commited. AFAICT, the following programs may be affected:

d.extend d.pan d.site.labels d.site.pg d.sites d.sites.qual
d.vect.labels d.what.sites d.zoom g.region m.in.e00 p.map p.map.new
ps.map r.cost r.random r.to.sites r.volume s.delaunay s.hull
s.in.ascii s.in.dbf s.info s.in.shape s.mask s.out.ascii s.out.e00
s.perturb s.proj s.qcount s.random s.sample s.surf.idw s.surf.rst
s.territory s.to.rast s.to.vect s.vol.rst s.voronoi s.what s.windavg
v.circle v.in.tig.lndmk v.surf.rst v.to.sites v.out.moss NVIZ

--
Glynn Clements <glynn.clements@virgin.net>

2

If it crashes because it cannot allocate more memory, then running 'unlimit' in the shell before running grass may help.
-Laura

On Wednesday, January 21, 2004, at 07:28 AM, Deuchler, Christoph wrote:

Hamish, Glynn,
as you predicted, the memory use grows until the swap space is full and then the program crashes.
I added the lines to sites.c and recompiled the whole thing, but unfortunately the result is the same. Instead of 'killed' I now get a 'Bus Error' message. More ideas?

Thank you for your help
Christoph

-----Original Message-----
From: Glynn Clements [mailto:glynn.clements@virgin.net]
Sent: Tue 1/20/2004 7:46 AM
To: Hamish
Cc: Deuchler, Christoph; GRASSLIST@baylor.edu; grass5@grass.itc.it
Subject: Re: [GRASSLIST:2288] Re: s.sample gets killed

Hamish wrote:

I need to extract raster cell values for a large number (approx. 2.5
million) of sites. Unfortunately s.sample gets killed after probing a
few thousand sites without apparent reason. No error message is given,
just 'killed'. Has anybody encountered the same problem and found a
solution or workaround?

Sounds like you are running out of memory.

can you run "top" to watch the memory use while s.sample runs?
hit "M" while top is running to sort processes by memory use.

Maybe it is a memory leak -- does the memory use steadily grow as the
program runs or is it all allocated at once? How big does it get? Please
post results so we can fix it if need be.

It's almost certainly due to a memory leak in libgis.

s.sample calls G_readsites_xyz() in a loop; G_readsites_xyz() calls
G_site_new_struct() to allocate a Site structure and
G_site_free_struct() to free it.

G_site_new_struct() allocates the dim, str_att and dbl_att arrays, and
also allocates a buffer for each entry in the str_att array.
G_site_free_struct() frees these arrays, but *doesn't* free the
buffers in the str_att array.

The attached patch fixes that; however, it should probably be tested
before being commited. AFAICT, the following programs may be affected:

d.extend d.pan d.site.labels d.site.pg d.sites d.sites.qual
d.vect.labels d.what.sites d.zoom g.region m.in.e00 p.map p.map.new
ps.map r.cost r.random r.to.sites r.volume s.delaunay s.hull
s.in.ascii s.in.dbf s.info s.in.shape s.mask s.out.ascii s.out.e00
s.perturb s.proj s.qcount s.random s.sample s.surf.idw s.surf.rst
s.territory s.to.rast s.to.vect s.vol.rst s.voronoi s.what s.windavg
v.circle v.in.tig.lndmk v.surf.rst v.to.sites v.out.moss NVIZ

--
Glynn Clements <glynn.clements@virgin.net>

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

3

Deuchler, Christoph wrote:

I added the lines to sites.c and recompiled the whole thing, but
unfortunately the result is the same. Instead of 'killed' I now get a
'Bus Error' message. More ideas?

Unfortunately not.

Memory exceptions (i.e. "Segmentation Fault" or "Bus Error") are
almost impossible to track down without a suitable test case (and
sending me a 2.5-million entry sites file or an equally large raster
are out of the question).

Can you reproduce the error with one of the sample data sets (e.g.
spearfish), either alone or with a small additional sites file? Or
with data which can be generated? If so, post the details and I'll try
to reproduce it.

--
Glynn Clements <glynn.clements@virgin.net>

4

> I added the lines to sites.c and recompiled the whole thing, but
> unfortunately the result is the same. Instead of 'killed' I now get
> a'Bus Error' message. More ideas?

Unfortunately not.

Memory exceptions (i.e. "Segmentation Fault" or "Bus Error") are
almost impossible to track down without a suitable test case (and
sending me a 2.5-million entry sites file or an equally large raster
are out of the question).

Can you reproduce the error with one of the sample data sets (e.g.
spearfish), either alone or with a small additional sites file? Or
with data which can be generated? If so, post the details and I'll try
to reproduce it.

use s.random, but you have to modify it to allow n to be bigger than
32k. I made "int n" into "long int n" and commented out parm.nsites->options

(Maybe worth doing properly in CVS.. 32k is pretty small)

Spearfish:

g.region rast=elevation.dted
s.random sites=random_sites n=2500000 # (122mb)
s.sample input=random_sites rast=elevation.dted > sample_sites

and watch it go..

I think 3-4gig of virtual/swap memory will make it through the 2.5M points.

Hamish

5

Hamish wrote:

> > I added the lines to sites.c and recompiled the whole thing, but
> > unfortunately the result is the same. Instead of 'killed' I now get
> > a'Bus Error' message. More ideas?
>
> Unfortunately not.
>
> Memory exceptions (i.e. "Segmentation Fault" or "Bus Error") are
> almost impossible to track down without a suitable test case (and
> sending me a 2.5-million entry sites file or an equally large raster
> are out of the question).
>
> Can you reproduce the error with one of the sample data sets (e.g.
> spearfish), either alone or with a small additional sites file? Or
> with data which can be generated? If so, post the details and I'll try
> to reproduce it.

use s.random, but you have to modify it to allow n to be bigger than
32k. I made "int n" into "long int n" and commented out parm.nsites->options

There is no point in using a long. Any platform capable of running
GRASS will have a 32-bit int, which allows for 2,147,483,647 sites.

Also, the value of the n= option is parsed using atoi(), which returns
an int; subsequently casting it to a long won't recover lost bits. If
you really need more than 32 bits, you need to use strtol() (and a
64-bit system; either that, or a platform which has "long long" and
strtoq()).

(Maybe worth doing properly in CVS.. 32k is pretty small)

Yep.

Spearfish:

g.region rast=elevation.dted
s.random sites=random_sites n=2500000 # (122mb)
s.sample input=random_sites rast=elevation.dted > sample_sites

and watch it go..

I think 3-4gig of virtual/swap memory will make it through the 2.5M points.

I was asking for a test case that generates the "Bus Error" message,
so that I can track it down.

--
Glynn Clements <glynn.clements@virgin.net>

6

What about solve the problem with "r.report -1 ", or converting to an
ascii file with r.out.ascii, then, with awk or something like that, import
the file as a site file?

On Wed, 21 Jan 2004, Deuchler, Christoph wrote:

Hamish, Glynn,
as you predicted, the memory use grows until the swap space is full and then the program crashes.
I added the lines to sites.c and recompiled the whole thing, but unfortunately the result is the same. Instead of 'killed' I now get a 'Bus Error' message. More ideas?

Thank you for your help
Christoph

-----Original Message-----
From: Glynn Clements [mailto:glynn.clements@virgin.net]
Sent: Tue 1/20/2004 7:46 AM
To: Hamish
Cc: Deuchler, Christoph; GRASSLIST@baylor.edu; grass5@grass.itc.it
Subject: Re: [GRASSLIST:2288] Re: s.sample gets killed

Hamish wrote:

> > I need to extract raster cell values for a large number (approx. 2.5
> > million) of sites. Unfortunately s.sample gets killed after probing a
> > few thousand sites without apparent reason. No error message is given,
> > just 'killed'. Has anybody encountered the same problem and found a
> > solution or workaround?
>
> Sounds like you are running out of memory.
>
> can you run "top" to watch the memory use while s.sample runs?
> hit "M" while top is running to sort processes by memory use.
>
> Maybe it is a memory leak -- does the memory use steadily grow as the
> program runs or is it all allocated at once? How big does it get? Please
> post results so we can fix it if need be.

It's almost certainly due to a memory leak in libgis.

s.sample calls G_readsites_xyz() in a loop; G_readsites_xyz() calls
G_site_new_struct() to allocate a Site structure and
G_site_free_struct() to free it.

G_site_new_struct() allocates the dim, str_att and dbl_att arrays, and
also allocates a buffer for each entry in the str_att array.
G_site_free_struct() frees these arrays, but *doesn't* free the
buffers in the str_att array.

The attached patch fixes that; however, it should probably be tested
before being commited. AFAICT, the following programs may be affected:

d.extend d.pan d.site.labels d.site.pg d.sites d.sites.qual
d.vect.labels d.what.sites d.zoom g.region m.in.e00 p.map p.map.new
ps.map r.cost r.random r.to.sites r.volume s.delaunay s.hull
s.in.ascii s.in.dbf s.info s.in.shape s.mask s.out.ascii s.out.e00
s.perturb s.proj s.qcount s.random s.sample s.surf.idw s.surf.rst
s.territory s.to.rast s.to.vect s.vol.rst s.voronoi s.what s.windavg
v.circle v.in.tig.lndmk v.surf.rst v.to.sites v.out.moss NVIZ

--
Pedro Camilo Alcántara Concepción
-----------------
camilo@pcbiol.posgrado.unam.mx
camiloalcantara@hotmail.com
-----------------
pcbiol.posgrado.unam.mx Wednesday Jan 21 2004 08:05:01 CST
-----------------
Si hubiera previsto las consecuencias me hubiera hecho relojero.
    -- Albert Einstein. (1879-1955) Físico alemán.