[GRASS-user] d.vect fails for large dataset

Dear all,

I am importing a fairly large polygon dataset (10 086 419 land cover polygons covering the area of Finland) from PostGIS into GRASS 7.2.1 (Linux) using v.import:

v.import --o input=PG:dbname=gisdata layer=zofie_cimburova.clip_finland output=clip_finland3 (see outputs of v.import, and v.info at the end of the mail).

The import finishes successfully, but when visualizing the layer (i.e. Zoom to selected maps), I get a following error:

Failed to run command ‘d.vect map=clip_finland3@u_zofie.cimburova fcolor=200:200:200 color=black width=0 size=5 type=point,line,area,face icon=basic/x’. Details: d.vect: spindex_rw.c:1362: rtree_get_node: Assertion `t->nb[level][which].n.level == level’ failed.

Do you by any chance know what the problem might be? I thought it is caused by overlaps in the dataset, but when I import a small subset of the original dataset, the error does not occur. Is the problem in the amount of data? If so, how do I deal with it?

Thanks a lot,

Zofie

v.info map=clip_finland3:

(…)

|----------------------------------------------------------------------------|

| Map format: native |

|----------------------------------------------------------------------------|

| Type of map: vector (level: 2) |

| |

| Number of points: 0 Number of centroids: 12653604 |

| Number of lines: 0 Number of boundaries: 19187847 |

| Number of areas: 12728699 Number of islands: 8610980 |

| |

| Map is 3D: No |

| Number of dblinks: 1 |

| |

| Projection: UTM (zone 33) |

| |

| N: 7828443.03 S: 6602796.5606247 |

| E: 1335791.79465162 W: 721051.43 |

| |

| Digitization threshold: 0 |

| Comment: |

| |

±---------------------------------------------------------------------------+

v.import:

Check if OGR layer <zofie_cimburova.clip_finland> contains polygons…

Importing 10086419 features (OGR layer <zofie_cimburova.clip_finland>)…


Registering primitives…

19371733 primitives registered

519601629 vertices registered

Number of nodes: 12123276

Number of primitives: 19371733

Number of points: 0

Number of lines: 0

Number of boundaries: 19371733

Number of centroids: 0

Number of areas: -

Number of isles: -


Cleaning polygons


Snapping boundaries (threshold = 1.000e-13)…

Reading features…

Snap vertices Pass 1: select points

Snap vertices Pass 2: assign anchor vertices

Snap vertices Pass 3: snap to assigned points


Breaking polygons…

Breaking polygons (pass 1: select break points)…

Breaking polygons (pass 2: break at selected points)…


Removing duplicates…


Breaking boundaries…


Removing duplicates…


Cleaning boundaries at nodes…


(…)


Merging boundaries…


Removing dangles…


Building areas…

WARNING: Area of size = 0.0 ignored

(…)

12728699 areas built

8610980 isles built

Attaching islands…

Number of nodes: 15022766

Number of primitives: 19187847

Number of points: 0

Number of lines: 0

Number of boundaries: 19187847

Number of centroids: 0

Number of areas: 12728699

Number of isles: 8610980

WARNING: Number of incorrect boundaries: 52223

WARNING: Vect_get_point_in_poly_isl(): collapsed area

WARNING: Unable to calculate area centroid

(…)


Finding centroids for OGR layer <zofie_cimburova.clip_finland>…


Writing centroids…

WARNING: 2258413 areas represent more (overlapping) features, because polygons overlap in input layer(s). Such areas are linked to more than 1 row in attribute table. The number of features for those areas is stored as category in layer 2


11478002 input polygons

Total area: 3.93306E+11 (12728699 areas)

Overlapping area: 4.43546E+09 (2258413 areas)

Area without category: 0.1304 (6448 areas)


Copying features…

Building topology for vector map <clip_finland3@u_zofie.cimburova>…

Registering primitives…

31841451 primitives registered

292562260 vertices registered

Building areas…

WARNING: Area of size = 0.0 ignored

(…)

12728699 areas built

8610980 isles built

Attaching islands…

Attaching centroids…

Number of nodes: 15022766

Number of primitives: 31841451

Number of points: 0

Number of lines: 0

Number of boundaries: 19187847

Number of centroids: 12653604

Number of areas: 12728699

Number of isles: 8610980

WARNING: Number of incorrect boundaries: 52223


Some input polygons are overlapping each other.

If overlapping is not desired, the data need to be cleaned.

The input could be cleaned by snapping vertices to each other.

Estimated range of snapping threshold: [1e-08, 1]

Try to import again, snapping with at least 1e-08: ‘snap=1e-08’

Input PG:dbname=gisdata successfully imported without reprojection

(Tue Jun 13 00:50:36 2017) Command finished (509 min 22 sec)

On Wed, Jun 14, 2017 at 9:23 AM, Žofie Cimburová <zoficimburova@gmail.com> wrote:

Dear all,

I am importing a fairly large polygon dataset (10 086 419 land cover polygons covering the area of Finland) from PostGIS into GRASS 7.2.1 (Linux) using v.import:

v.import --o input=PG:dbname=gisdata layer=zofie_cimburova.clip_finland output=clip_finland3 (see outputs of v.import, and v.info at the end of the mail).

The import finishes successfully, but when visualizing the layer (i.e. Zoom to selected maps), I get a following error:

Failed to run command ‘d.vect map=clip_finland3@u_zofie.cimburova fcolor=200:200:200 color=black width=0 size=5 type=point,line,area,face icon=basic/x’. Details: d.vect: spindex_rw.c:1362: rtree_get_node: Assertion `t->nb[level][which].n.level == level’ failed.

Apparently something went wrong with writing the spatial index to file. Maybe some of those collapsed areas of zero size are causing problems for the spatial index (they will also cause lots of other problems when trying to analyse the map). You can try to import again with v.in.ogr using the suggested snapping threshold of 1e-8, then increasing the threshold by factor 10 until the warnings about zero-size areas disappear. This might take quite some time. Most likely there will still be overlapping areas.

Markus M

Do you by any chance know what the problem might be? I thought it is caused by overlaps in the dataset, but when I import a small subset of the original dataset, the error does not occur. Is the problem in the amount of data? If so, how do I deal with it?

Thanks a lot,

Zofie

v.info map=clip_finland3:

(…)

|----------------------------------------------------------------------------|

| Map format: native |

|----------------------------------------------------------------------------|

| Type of map: vector (level: 2) |

| |

| Number of points: 0 Number of centroids: 12653604 |

| Number of lines: 0 Number of boundaries: 19187847 |

| Number of areas: 12728699 Number of islands: 8610980 |

| |

| Map is 3D: No |

| Number of dblinks: 1 |

| |

| Projection: UTM (zone 33) |

| |

| N: 7828443.03 S: 6602796.5606247 |

| E: 1335791.79465162 W: 721051.43 |

| |

| Digitization threshold: 0 |

| Comment: |

| |

±---------------------------------------------------------------------------+

v.import:

Check if OGR layer <zofie_cimburova.clip_finland> contains polygons…

Importing 10086419 features (OGR layer <zofie_cimburova.clip_finland>)…


Registering primitives…

19371733 primitives registered

519601629 vertices registered

Number of nodes: 12123276

Number of primitives: 19371733

Number of points: 0

Number of lines: 0

Number of boundaries: 19371733

Number of centroids: 0

Number of areas: -

Number of isles: -


Cleaning polygons


Snapping boundaries (threshold = 1.000e-13)…

Reading features…

Snap vertices Pass 1: select points

Snap vertices Pass 2: assign anchor vertices

Snap vertices Pass 3: snap to assigned points


Breaking polygons…

Breaking polygons (pass 1: select break points)…

Breaking polygons (pass 2: break at selected points)…


Removing duplicates…


Breaking boundaries…


Removing duplicates…


Cleaning boundaries at nodes…


(…)


Merging boundaries…


Removing dangles…


Building areas…

WARNING: Area of size = 0.0 ignored

(…)

12728699 areas built

8610980 isles built

Attaching islands…

Number of nodes: 15022766

Number of primitives: 19187847

Number of points: 0

Number of lines: 0

Number of boundaries: 19187847

Number of centroids: 0

Number of areas: 12728699

Number of isles: 8610980

WARNING: Number of incorrect boundaries: 52223

WARNING: Vect_get_point_in_poly_isl(): collapsed area

WARNING: Unable to calculate area centroid

(…)


Finding centroids for OGR layer <zofie_cimburova.clip_finland>…


Writing centroids…

WARNING: 2258413 areas represent more (overlapping) features, because polygons overlap in input layer(s). Such areas are linked to more than 1 row in attribute table. The number of features for those areas is stored as category in layer 2


11478002 input polygons

Total area: 3.93306E+11 (12728699 areas)

Overlapping area: 4.43546E+09 (2258413 areas)

Area without category: 0.1304 (6448 areas)


Copying features…

Building topology for vector map <clip_finland3@u_zofie.cimburova>…

Registering primitives…

31841451 primitives registered

292562260 vertices registered

Building areas…

WARNING: Area of size = 0.0 ignored

(…)

12728699 areas built

8610980 isles built

Attaching islands…

Attaching centroids…

Number of nodes: 15022766

Number of primitives: 31841451

Number of points: 0

Number of lines: 0

Number of boundaries: 19187847

Number of centroids: 12653604

Number of areas: 12728699

Number of isles: 8610980

WARNING: Number of incorrect boundaries: 52223


Some input polygons are overlapping each other.

If overlapping is not desired, the data need to be cleaned.

The input could be cleaned by snapping vertices to each other.

Estimated range of snapping threshold: [1e-08, 1]

Try to import again, snapping with at least 1e-08: ‘snap=1e-08’

Input PG:dbname=gisdata successfully imported without reprojection

(Tue Jun 13 00:50:36 2017) Command finished (509 min 22 sec)


grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user