Does v.in.ascii have problems with VERY large files (grass 6.1 for cygwin)?
When I do a v.info on the imported vector, I get a big red X with "Cannon
open old vector tahoe_4_pansharp_stems@tahoe on level 2". The .csv file
(x,y and a number column which should be the sole attribute of the vector)
is about 280mb in size (~ 600,000 points).
On a related note, v.in.ogr fails importing a shapefile that is about 170mb
in size (dbf file is a hair under 2gb). I get an out-of-memory error (my
memory usage goes up to 1.5 gb).
--j
--
Jonathan A. Greenberg, PhD
NRC Research Associate
NASA Ames Research Center
MS 242-4
Moffett Field, CA 94035-1000
Phone: 415-794-5043
AIM: jgrn3007
MSN: jgrn3007@hotmail.com
On Thu, 2 Mar 2006 12:08:05 -0800
"Jonathan Greenberg" <jgreenberg@arc.nasa.gov> wrote:
Does v.in.ascii have problems with VERY large files (grass 6.1 for
cygwin)? When I do a v.info on the imported vector, I get a big red X
with "Cannon open old vector tahoe_4_pansharp_stems@tahoe on level
2". The .csv file (x,y and a number column which should be the sole
attribute of the vector) is about 280mb in size (~ 600,000 points).
On a related note, v.in.ogr fails importing a shapefile that is about
170mb in size (dbf file is a hair under 2gb). I get an out-of-memory
error (my memory usage goes up to 1.5 gb).
There was plenty of discussion on the Grass vs big vector files.
Problems you are experiencing is not something unusuall given the
current state of Grass vector engine.
The dbf driver is another memory hog. This you could overcome by using
other DB backends - sqlite, mysql, postgres. I have tried sqlite and in
many circumstances it is better than dbf, although it still needs to be
optimised. I was told postgres would perform better. For mysql I don't
know.
Browse grass5 and grasslist archives for related discussions.
Maciek
---------------------
Szukasz solidnej firmy reklamowej? Dobrze trafi?e?! Zapraszamy do nas! http://www.artkomp.pl , lub zadzwo?: (0-33) 816-86-01
On Fri, Mar 03, 2006 at 12:03:53AM +0100, Maciek Sieczka wrote:
On Thu, 2 Mar 2006 12:08:05 -0800
"Jonathan Greenberg" <jgreenberg@arc.nasa.gov> wrote:
> Does v.in.ascii have problems with VERY large files (grass 6.1 for
> cygwin)? When I do a v.info on the imported vector, I get a big red X
> with "Cannon open old vector tahoe_4_pansharp_stems@tahoe on level
> 2". The .csv file (x,y and a number column which should be the sole
> attribute of the vector) is about 280mb in size (~ 600,000 points).
>
> On a related note, v.in.ogr fails importing a shapefile that is about
> 170mb in size (dbf file is a hair under 2gb). I get an out-of-memory
> error (my memory usage goes up to 1.5 gb).
There was plenty of discussion on the Grass vs big vector files.
Problems you are experiencing is not something unusuall given the
current state of Grass vector engine.
Note that there is the
-b Do not build topology in points mode
flag. Helena and others imported some 500M points like this. v.surf.rst
is under modification to support disabled topology (maybe it does
already, I didn't try recently).
If you operate with points only (like LIDAR), this is the way to
overcome the problem.
Note that there is the
-b Do not build topology in points mode
flag. Helena and others imported some 500M points like this. v.surf.rst
is under modification to support disabled topology (maybe it does
already, I didn’t try recently).
If you operate with points only (like LIDAR), this is the way to
overcome the problem.
This is interesting, as I’m still using GRASS5 for {v|s}.surf.rst and large datasets. Can anyone confirm that using the -b option reads up a large number of points without problems, and that these can be used with v.surf.rst?
Does v.in.ascii have problems with VERY large files (grass 6.1 for
cygwin)? When I do a v.info on the imported vector, I get a big red X
with "Cannon open old vector tahoe_4_pansharp_stems@tahoe on level 2".
The .csv file
(x,y and a number column which should be the sole attribute of the
vector) is about 280mb in size (~ 600,000 points).
On a related note, v.in.ogr fails importing a shapefile that is about
170mb in size (dbf file is a hair under 2gb). I get an out-of-memory
error (my memory usage goes up to 1.5 gb).
600k points should be no problem for 6.1-cvs and 1gb+ RAM.
Use the -t flag to avoid building a table, this will save lots of RAM.
The -b flag is only needed if you want to load in millions and millions
of points. Only use as a last resort as topology is needed for most
modules. We are still waiting for a binning module (s.cellstats port) to
cull massive datasets into something more useful for RST input.