[GRASS5] [bug #3354] (grass) v.in.ascii crashing on large inputs

GRASS GIS GRASS GIS developer
1

this bug's URL: http://intevation.de/rt/webrt?serial_num=3354
-------------------------------------------------------------------------

Subject: v.in.ascii crashing on large inputs

Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 6.1 CVS 11 June

A followup to bug 2903. I cannot import more than about three or four million LIDAR points using v.in.ascii. It crashes with an out of memory error.

From: Hamish
In reply to: Andrew Danner
Re: [GRASSLIST:7168] v.in.ascii out of memory
Date: Thu, 16 Jun 2005 15:42:02 +1200 (05:42 CEST)

I'm trying to import a somewhat large file of 24.8 million LIDAR
points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

Status: (all due to memory limitations [ie leaks])
  GRASS 6.0.0's v.in.ascii ~ 100k points
  GRASS 6.1-cvs v.in.ascii ~ 3M points (you)

File format is just a list of easting, northing, and elevation in NC
state-plane.

1939340.8400,825793.8900,657.2200
1939071.9500,825987.7800,660.2200
1939035.5200,826013.9700,662.4600
1938762.4500,826210.1500,686.2800

I'm using

v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

try with -t too. No need to create a table if you have no attributes!

*This only offers a small improvement of < 100000 points

and I get the following:

Maximum input row length: 34
Maximum number of columns: 3
Minimum number of columns: 3

Building topology ...
Registering lines: ERROR: G_realloc: out of memory

There shouldn't be any lines to register. This is just a list of
points. Any ideas?

I've seen the same thing. Not sure if it has to know everything to
build topology or if it is a memory leak. I think it's a leak.

I did some tests with valgrind back in March. Radim plugged the biggest
leak, but I think "Registering Lines.." still has one too.

see this thread:
http://article.gmane.org/gmane.comp.gis.grass.devel/7212/

At three million points, I noticed that I'm using over 1GB of memory.
Is it trying to build the entire topology in memory? I don't need to
build any topology for a set of points. Is there a way to turn this
off?

In 5.4 I was developing a scalable version of s.surf.rst that could
interpolate surfaces on over 500 million points, but now I can't even
import that many points into a vector/site format.

Comments/Suggestions?

a) Try adding more swap.
b) We need to fix the leak. (more valgrind)

*I have 8GB of swap. There may be some Linux 2.6 oom-killer issues, and I will try to fix this, but reading in just a list of points should not require over 1GB of memory. There is no topology to build.

-------------------------------------------- Managed by Request Tracker