[GRASS-dev] Shape import : out of memory

Hi,

I try to import a 1GB Shapefile and the geometries seem to be in a really
bad condition. Building topology GRASS exits after 30 hours with an OUT OF
MEMORY! error (see below).

Can someone give me hint, what I could do, to import this file? Using -c or
-f flag is no option, because I would like to use GRASS to clean the data as far
as possible automatically and I wouldn't like to split the file into subregion,
if I don't have to.

The PC I use is a dual core 1,8MHz with 4GB RAM and 1.6GB swap. Is this still
not enough or can I change something in GRASS (extend or change memory
parameter) to finish the import? I used a svn version from yesterday.

below you see parts of the 230MB log file:

thanks
  Otto

############# part of logfile

Projection of input dataset and current location appear to match
Layer: schutzgebiete
Importiere 886619 Objekte der Karte...
WARNUNG: Area size [1.1e-05], area not imported
WARNUNG: Area size [5.0e-06], area not imported
WARNUNG: Area size [1.1e-05], area not imported
WARNUNG: Area size [8.6e-05], area not imported
WARNUNG: Area size [7.1e-05], area not imported
WARNUNG: Area size [2.5e-05], area not imported
WARNUNG: Area size [3.9e-05], area not imported
WARNUNG: Area size [6.6e-05], area not imported
[...]
-----------------------------------------------------
Building topology for vector map <schutzgebiete>...
Registriere Linien
[...]
902000 903000 904000
906094 primitives registered
Building areas:
[...]
100%
714558 areas built
625964 isles built
Füge Inselflächen hinzu:
[...]
100%
Füge Zentroide hinzu:
[...]
100%
Topology was built
Anzahl von Knoten : 745666
Anzahl von Primitives : 906094
Anzahl von Punkten : 0
Anzahl von Linien : 0
Anzahl von Boundaries : 906094
Anzahl von Zentroiden : 0
Anzahl von Flächen : 714558
Anzahl von Inseln : 625964
Anzahl inkorrekter Boundaries : 246365
Anzahl von Flächen ohne Zentroid : 714558
-----------------------------------------------------
WARNUNG: Cleaning polygons, result is not guaranteed!
Building topology for vector map <schutzgebiete>...
Topology was built
Anzahl von Knoten : 745666
Anzahl von Primitives : 906094
Anzahl von Punkten : 0
Anzahl von Linien : 0
Anzahl von Boundaries : 906094
Anzahl von Zentroiden : 0
Anzahl von Flächen : -
Anzahl von Inseln : -
-----------------------------------------------------
Break polygons:

Registering points 566154
Registering points 1023238
[...]
Registering points 10910043
Registering points 10910043
All points (vertices): 20905602
Registered points (unique coordinates): 10910043
Points marked for break: 1270535
Breaks: 0
Breaks: 217
Breaks: 285
Breaks: 295
Breaks: 351
Breaks: 372
Breaks: 385
[...]
Breaks: 3918420
Breaks: 3918432
Breaks: 3918711
Breaks: 3918732
Breaks: 3918832
-----------------------------------------------------
Remove duplicates:
Duplicates: 0
Duplicates: 1
Duplicates: 2
Duplicates: 3Duplicates: 28596
Duplicates: 28597
Duplicates: 28598
-----------------------------------------------------
Clean boundaries at nodes:
Modifications: 0
Modifications: 0
Modifications: 0
Modifications: 0
[...]
Modifications: 765495
Modifications: 765495
-----------------------------------------------------
Break boundaries:
Intersections: 0
Intersections: 0 (line 3513)
Intersections: 0 (line 4577)
Intersections: 0 (line 10159)
[...]
Intersections: 503 (line 6431268)
Intersections: 503 (line 6431505)
Intersections: 503
-----------------------------------------------------
Remove duplicates:
Duplicates: 0
Duplicates: 1
[...]
Duplicates: 2429
Duplicates: 2430
-----------------------------------------------------
Clean boundaries at nodes:
Modifications: 0
Modifications: 0
Modifications: 0
Modifications: 0
[...]
Modifications: 0
Modifications: 0
Modifications: 0
-----------------------------------------------------
Change dangles to lines:
Removed dangles: 0 removed lines: 0
Removed dangles: 0 removed lines: 1
Removed dangles: 1 removed lines: 2
Removed dangles: 2 removed lines: 3
[...]
Removed dangles: 10527 removed lines: 10528
Removed dangles: 10528 removed lines: 10529
Removed dangles: 10529 removed lines: 10529
-----------------------------------------------------
Remove bridges:
Removed bridges: 0 removed lines: 0
Removed bridges: 0 removed lines: 0
[...]
Removed bridges: 0 removed lines: 0
Removed bridges: 0 removed lines: 0
[...]
WARNUNG: Kann den FlÃ___chenzentroiden nicht berechnen.
WARNUNG: Kann den FlÃ___chenzentroiden nicht berechnen.
WARNUNG: Kann den FlÃ___chenzentroiden nicht berechnen.
[...]
WARNUNG: Kann den FlÃ___chenzentroiden nicht berechnen.
WARNUNG: Kann den FlÃ___chenzentroiden nicht berechnen.

Layer: schutzgebiete
OUT OF MEMORY!
OUT OF MEMORY!
[...]
OUT OF MEMORY!
OUT OF MEMORY!
OUT OF MEMORY!

On Feb 2, 2008 4:38 PM, Otto Dassau <otto.dassau@gmx.de> wrote:

Hi,

I try to import a 1GB Shapefile and the geometries seem to be in a really
bad condition. Building topology GRASS exits after 30 hours with an OUT OF
MEMORY! error (see below).

Can someone give me hint, what I could do, to import this file? Using -c or
-f flag is no option, because I would like to use GRASS to clean the data as far
as possible automatically and I wouldn't like to split the file into subregion,
if I don't have to.

The PC I use is a dual core 1,8MHz with 4GB RAM and 1.6GB swap. Is this still
not enough or can I change something in GRASS (extend or change memory
parameter) to finish the import? I used a svn version from yesterday.

below you see parts of the 230MB log file:

thanks
  Otto

############# part of logfile

Projection of input dataset and current location appear to match
Layer: schutzgebiete
Importiere 886619 Objekte der Karte...
WARNUNG: Area size [1.1e-05], area not imported
WARNUNG: Area size [5.0e-06], area not imported
WARNUNG: Area size [1.1e-05], area not imported
WARNUNG: Area size [8.6e-05], area not imported
WARNUNG: Area size [7.1e-05], area not imported
WARNUNG: Area size [2.5e-05], area not imported
WARNUNG: Area size [3.9e-05], area not imported
WARNUNG: Area size [6.6e-05], area not imported
[...]

Did you enable snapping? Above micro areas may indicate
gaps and slivers. Perhaps snapping helps, e.g.
snap=0.0001

Markus

On Sat, 2 Feb 2008 17:44:03 +0100
"Markus Neteler" <neteler@osgeo.org> wrote:

On Feb 2, 2008 4:38 PM, Otto Dassau <otto.dassau@gmx.de> wrote:
> Hi,
>
> I try to import a 1GB Shapefile and the geometries seem to be in a really
> bad condition. Building topology GRASS exits after 30 hours with an OUT OF
> MEMORY! error (see below).
>
> Can someone give me hint, what I could do, to import this file? Using -c or
> -f flag is no option, because I would like to use GRASS to clean the data as
> far
> as possible automatically and I wouldn't like to split the file into
> subregion,
> if I don't have to.
>
> The PC I use is a dual core 1,8MHz with 4GB RAM and 1.6GB swap. Is this
> still
> not enough or can I change something in GRASS (extend or change memory
> parameter) to finish the import? I used a svn version from yesterday.
>
> below you see parts of the 230MB log file:
>
> thanks
> Otto
>
> ############# part of logfile
>
> Projection of input dataset and current location appear to match
> Layer: schutzgebiete
> Importiere 886619 Objekte der Karte...
> WARNUNG: Area size [1.1e-05], area not imported
> WARNUNG: Area size [5.0e-06], area not imported
> WARNUNG: Area size [1.1e-05], area not imported
> WARNUNG: Area size [8.6e-05], area not imported
> WARNUNG: Area size [7.1e-05], area not imported
> WARNUNG: Area size [2.5e-05], area not imported
> WARNUNG: Area size [3.9e-05], area not imported
> WARNUNG: Area size [6.6e-05], area not imported
> [...]

Did you enable snapping? Above micro areas may indicate
gaps and slivers. Perhaps snapping helps, e.g.
snap=0.0001

playing around with the snapping parameter was the solution to import the data.
I guess you were right about gaps and slivers.

thanks a lot
Otto

Markus
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev