[GRASSLIST:7168] v.in.ascii out of memory

GRASS GRASS user
1

I'm trying to import a somewhat large file of 24.8 million LIDAR points
(866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

File format is just a list of easting, northing, and elevation in NC
state-plane.

1939340.8400,825793.8900,657.2200
1939071.9500,825987.7800,660.2200
1939035.5200,826013.9700,662.4600
1938762.4500,826210.1500,686.2800

I'm using

v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

and I get the following:

Maximum input row length: 34
Maximum number of columns: 3
Minimum number of columns: 3

Building topology ...
Registering lines: ERROR: G_realloc: out of memory

There shouldn't be any lines to register. This is just a list of points.
Any ideas?

I have imported close to 500 million points in GRASS 5.4 using
s.in.ascii, but I don't know why things are crashing in the new vector
format.

I have tested v.in.ascii upto 3 million points.

I'm running on a Fedora Core 3 Linux with a 2.6 kernel, 1GB of physical
memory and 8GB of swap.

At three million points, I noticed that I'm using over 1GB of memory. Is
it trying to build the entire topology in memory? I don't need to build
any topology for a set of points. Is there a way to turn this off?

In 5.4 I was developing a scalable version of s.surf.rst that could
interpolate surfaces on over 500 million points, but now I can't even
import that many points into a vector/site format.

Comments/Suggestions?

-Andy

2

I'm trying to import a somewhat large file of 24.8 million LIDAR
points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

Status: (all due to memory limitations [ie leaks])
  GRASS 6.0.0's v.in.ascii ~ 100k points
  GRASS 6.1-cvs v.in.ascii ~ 3M points (you)

File format is just a list of easting, northing, and elevation in NC
state-plane.

1939340.8400,825793.8900,657.2200
1939071.9500,825987.7800,660.2200
1939035.5200,826013.9700,662.4600
1938762.4500,826210.1500,686.2800

I'm using

v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

try with -t too. No need to create a table if you have no attributes!

and I get the following:

Maximum input row length: 34
Maximum number of columns: 3
Minimum number of columns: 3

Building topology ...
Registering lines: ERROR: G_realloc: out of memory

There shouldn't be any lines to register. This is just a list of
points. Any ideas?

I've seen the same thing. Not sure if it has to know everything to
build topology or if it is a memory leak. I think it's a leak.

I did some tests with valgrind back in March. Radim plugged the biggest
leak, but I think "Registering Lines.." still has one too.

see this thread:
http://article.gmane.org/gmane.comp.gis.grass.devel/7212/

I have imported close to 500 million points in GRASS 5.4 using
s.in.ascii, but I don't know why things are crashing in the new vector
format.

GRASS 5.4 stored sites in a simple ASCII file-- the limit was whatever
your filesystem/OS could handle. Vector points in GRASS 6 is a 100%
different implementation.

I have tested v.in.ascii upto 3 million points.

I'm running on a Fedora Core 3 Linux with a 2.6 kernel, 1GB of
physical memory and 8GB of swap.

Note you can add more swap memory to Linux on the fly, see:

http://grass.itc.it/pipermail/grassuser/2002-February/006072.html

At three million points, I noticed that I'm using over 1GB of memory.
Is it trying to build the entire topology in memory? I don't need to
build any topology for a set of points. Is there a way to turn this
off?

In 5.4 I was developing a scalable version of s.surf.rst that could
interpolate surfaces on over 500 million points, but now I can't even
import that many points into a vector/site format.

Comments/Suggestions?

a) Try adding more swap.
b) We need to fix the leak. (more valgrind)

let us know how you get on.

Hamish

3

On Thu, 2005-06-16 at 15:42 +1200, Hamish wrote:

> I'm trying to import a somewhat large file of 24.8 million LIDAR
> points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

Status: (all due to memory limitations [ie leaks])
  GRASS 6.0.0's v.in.ascii ~ 100k points
  GRASS 6.1-cvs v.in.ascii ~ 3M points (you)

> I'm using
>
> v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

try with -t too. No need to create a table if you have no attributes!

-t saves maybe a few KB. Good to know, but not the main culprit

> and I get the following:
>
> Building topology ...
> Registering lines: ERROR: G_realloc: out of memory
>
> There shouldn't be any lines to register. This is just a list of
> points. Any ideas?

I've seen the same thing. Not sure if it has to know everything to
build topology or if it is a memory leak. I think it's a leak.

I did some tests with valgrind back in March. Radim plugged the biggest
leak, but I think "Registering Lines.." still has one too.

see this thread:
http://article.gmane.org/gmane.comp.gis.grass.devel/7212/

> At three million points, I noticed that I'm using over 1GB of memory.
> Is it trying to build the entire topology in memory? I don't need to
> build any topology for a set of points. Is there a way to turn this
> off?
>

a) Try adding more swap.
b) We need to fix the leak. (more valgrind)

I'm not running out of swap. In fact, isn't it the case that the 32-bit
Linux machines can address more the 4GB of address space per process (or
perhaps per thread)?, so at some point, more swap won't work. Unless the
topology building is doing something where each feature, reading in a
list of points should have memory requirements that are independent on
the input size and I should only be limited by disk space.

-Andy