[GRASSLIST:6526] problem importing vector

Hello,

I'm new to GRASS (and to GIS). I'm having trouble importing a vector file (using v.in.ogr) of the Everglades coastline in Florida, USA. The coastline is very irregular with many small islands. My machine has been working for three days, appears to be making progress (e.g., the line number at the bottom of the output shown below keeps increasing), albeit slowly. I estimate that it will take about a week to complete this import. I'm using Grass 6.0 on Mac OSX and the output is displayed below. My specific questions are:

1. Should I wait a week for this to complete, e.g., is it likely to successfully complete this import, or is the long time required indicative of a problem?

2. Is there a faster way of accomplishing this import?

Thanks for your help,
Brian

v.in.ogr dsn=/Users/PROJECTS/ENP/Plot_Info/GIS/PaulsCD/k24bound.shp output=k24bound

A datum name nad27 (North_American_Datum_1927) was specified without transformation parameters.
WARNING: Non-interactive mode: the GRASS default for nad27 is
          towgs84=-22.000,157.000,176.000.
Projection of input dataset and current location appear to match.
Proceeding with import...
Layer: k24bound
-----------------------------------------------------
Building topology ...
18233 primitives registered
Building areas: 100%
18233 areas built
18233 isles built
Attaching islands: 100%
Attaching centroids: 100%
Topology was built.
Number of nodes : 18233
Number of primitives: 18233
Number of points : 0
Number of lines : 0
Number of boundaries: 18233
Number of centroids : 0
Number of areas : 18233
Number of isles : 18233
Number of areas without centroid : 18233
-----------------------------------------------------
WARNING: Cleaning polygons, result is not guaranteed!
Building topology ...
Topology was built.
Number of nodes : 18233
Number of primitives: 18233
Number of points : 0
Number of lines : 0
Number of boundaries: 18233
Number of centroids : 0
Number of areas : -
Number of isles : -
-----------------------------------------------------
Break polygons:
Registering points ... 745775
All points (vertices): 764031
Registered points (unique coordinates): 745775
Points marked for break: 18254
Breaks: 65
-----------------------------------------------------
Remove duplicates:
Duplicates: 0
-----------------------------------------------------
Break boundaries:
Intersections: 297999 (line 89173)

--
*********************************************************************
Brian Beckage
Department of Botany
University of Vermont
Marsh Life Science Building
Burlington, VT 05405

Phone: 802 656-0197
Fax : 802 656-0440
email: Brian.Beckage@uvm.edu
*********************************************************************

Brian Beckage wrote:

Hello,

I'm new to GRASS (and to GIS). I'm having trouble importing a vector file (using v.in.ogr) of the Everglades coastline in Florida, USA. The coastline is very irregular with many small islands.

The import of areas can be slow if some features cover big part of the map as it this case probably the costline. The reason is that v.in.ogr will try to intersect the costline with other features in bounding box, if the box is big, many features must be processed.

Hamish posted more deatails about the patch.

You can verify if this is the problem. If you have current CVS version of GRASS, try:

v.in.ogr -c
v.split
v.clean tool=break

Break boundaries:
Intersections: 297999 (line 89173)

This is strange, there should be no intersections in costline + islands. Am I wrong? Try to import with v.in.ogr -c and find where the boundaries intersect.

Is the sea and area? If so, you must somehow exclude the shape with sea (ogr2ogr -where)

Radim

At 4:16 PM +0200 4/19/05, Radim Blazek wrote:

The import of areas can be slow if some features cover big part of the map as it this case probably the costline. The reason is that v.in.ogr will try to intersect the costline with other features in bounding box, if the box is big, many features must be processed.

Hamish posted more deatails about the patch.

You can verify if this is the problem. If you have current CVS version of GRASS, try:

I have a pre-compiled binary version of GRASS (6.0.0) for OSX, not the cvs version.

v.in.ogr -c
v.split
v.clean tool=break

Is this sequence of commands equivalent to v.in.ogr?

Break boundaries:
Intersections: 297999 (line 89173)

This is strange, there should be no intersections in costline + islands. Am I wrong? Try to import with v.in.ogr -c and find where the boundaries intersect.

Well, this layer actually contains all water/land boundaries including the coastline, islands, and inland water bodies such as canals, lakes, etc.

Is the sea and area? If so, you must somehow exclude the shape with sea (ogr2ogr -where)

Brian

--
*********************************************************************
Brian Beckage
Department of Botany
University of Vermont
Marsh Life Science Building
Burlington, VT 05405

Phone: 802 656-0197
Fax : 802 656-0440
email: Brian.Beckage@uvm.edu
web : www.uvm.edu/~bbeckage
*********************************************************************

Hi Hamish

Ok, I've done this; needed to add a v.type step before and after v.split
as it doesn't support boundaries yet. Also note that it lost the DB
connection during the v.split step, but easy enough to reconnect to
original DB when done.

BEFORE:

time v.in.ogr dsn=`pwd` layer=Coast_islands out=test_pre
real 576m40.151s
user 572m45.630s
sys 2m55.560s

AFTER:

_method_
v.in.ogr -c
v.type type=boundary,line
v.split
v.type type=line,boundary
v.clean tool=break
db.copy
v.db.connect

time v.in.ogr -c dsn=`pwd` layer=Coast_islands out=test_pre
real 2m34.380s
user 1m47.290s
sys 0m42.980s

v.type in=test_pre out=test_pre2 type=boundary,line
v.split in=test_pre2 out=test_pre3 vertices=100
v.type in=test_pre3 out=test_pre4 type=line,boundary
[all fast]

time v.clean in=test_pre4 out=test_pre5 tool=break
real 0m41.982s
user 0m41.310s
sys 0m0.580s

db.copy from_table=test_pre to_table=test_pre5
v.db.connect test_pre5 table=test_pre5
[fast]

9.6 hours -> 3.25 minutes. Not bad, I think you guessed correctly Radim...
[2.8GHz P4, Linux]

Hamish

I've applied your PATCH to vector/v.in.ogr/geom.c.
All OS X users can download last cvs version (050419) with this patch at: http://wwwamb.bologna.enea.it/forgrass/downloadcvs.htm.

I have tried with some old shape that have problems in Grass with normal:
v.in.ogr -o
... and I have the same problems than before: infinite loop. I have killed v.in.ogr.

After I have followed your method (a bit complex)

v.in.ogr -c -o (-c is been used with some complex shapes)
v.type type=boundary,line
v.split
v.type type=line,boundary
v.clean tool=break
db.copy
v.db.connect

and from X11 terminal this method works in OS X and it's fast.
This is a good news but the method is not very easy.

I have retry the same commands from wish app in X11 and AQUA environment (graphic method) (v.type in graphic window is not good) but v.clean send to crash my Wish app, only wish app.

Why wish app is more unstable with some commands with some vector layer?

Bye

--
________________________________________________________________________
|| Lorenzo Moretti e-mail: lorenzo.moretti@bologna.enea.it ||/|/| ENEA prot/idr Web: http://wwwamb.bologna.enea.it/ || | via Don Fiammelli, 2 FTP: ftp://ftpamb.bologna.enea.it/ (res.)
~~~~~~ 40128 BOLOGNA - ITALY Ph: +39-0516098086 Fax: +39-0516098131
      Download GRASS for MAC OS X: http://wwwamb.bologna.enea.it/forgrass/
________________________________________________________________________

Lorenzo Moretti wrote:

After I have followed your method (a bit complex)

v.in.ogr -c -o (-c is been used with some complex shapes)
v.type type=boundary,line
v.split
v.type type=line,boundary
v.clean tool=break
db.copy
v.db.connect

and from X11 terminal this method works in OS X and it's fast.
This is a good news but the method is not very easy.

The method is NOT CORRECT! I wrote that here many times:
v.in.ogr -c + v.clean != v.in.ogr

Radim

>Hamish

I've applied your PATCH to vector/v.in.ogr/geom.c.
All OS X users can download last cvs version (050419) with this patch
at: http://wwwamb.bologna.enea.it/forgrass/downloadcvs.htm.

I have tried with some old shape that have problems in Grass with
normal: v.in.ogr -o
... and I have the same problems than before: infinite loop. I have
killed v.in.ogr.

that patch was non-functional, it still needs some work. It won't do any
harm though, as it won't do anything at all.

I used

#ifdef UNUSED
...
#endif

for commenting out a chunk of code which contains other /* comments */.

I have retry the same commands from wish app in X11 and AQUA
environment (graphic method) (v.type in graphic window is not good)

(as Radim noted, you don't actually need the v.type steps after all)

is the v.clean problem a result of you needing GUI options like:
(choose only one)

o line,boundary
o boundary,line
o point,centroid
o centroid,point

and not as now:
(choose multiple)

o line
o boundary
o centroid
o point

and then somehow know that you are only allowed to select two compatible
ones from the list and somehow specify order (impossible)?

but v.clean send to crash my Wish app, only wish app.
Why wish app is more unstable with some commands with some vector layer?

no idea. Could it be warning / error messages triggering the crash?
G_percent?

maybe debug with:
v.clean --tcltk | less
?

Hamish

Radim Blazek wrote

After I have followed your method (a bit complex)

v.in.ogr -c -o (-c is been used with some complex shapes)
v.type type=boundary,line
v.split
v.type type=line,boundary
v.clean tool=break
db.copy
v.db.connect

and from X11 terminal this method works in OS X and it's fast.
This is a good news but the method is not very easy.

The method is NOT CORRECT! I wrote that here many times:
v.in.ogr -c + v.clean != v.in.ogr

Radim

OK Radim but what is the best method if "v.in.ogr -o" does not work?

--
________________________________________________________________________
|| Lorenzo Moretti e-mail: lorenzo.moretti@bologna.enea.it ||/|/| ENEA prot/idr Web: http://wwwamb.bologna.enea.it/ || | via Don Fiammelli, 2 FTP: ftp://ftpamb.bologna.enea.it/ (ris.)
~~~~~~ 40128 BOLOGNA - ITALY Ph: +39-0516098086 Fax: +39-0516098131
________________________________________________________________________

Hi Hamish

> >Hamish

I've applied your PATCH to vector/v.in.ogr/geom.c.
All OS X users can download last cvs version (050419) with this patch
at: http://wwwamb.bologna.enea.it/forgrass/downloadcvs.htm.

I have tried with some old shape that have problems in Grass with
normal: v.in.ogr -o
... and I have the same problems than before: infinite loop. I have
killed v.in.ogr.

that patch was non-functional, it still needs some work. It won't do any
harm though, as it won't do anything at all.

I used

#ifdef UNUSED
...
#endif

for commenting out a chunk of code which contains other /* comments */.

I have retry the same commands from wish app in X11 and AQUA
environment (graphic method) (v.type in graphic window is not good)

(as Radim noted, you don't actually need the v.type steps after all)

Is this sufficient for complex shape files that don't work with normal "v.in.ogr -o"?
- v.in.ogr -c -o
- v.clean tool=break

is the v.clean problem a result of you needing GUI options like:
(choose only one)

o line,boundary
o boundary,line
o point,centroid
o centroid,point

This list is the best solution for v.type command window

and not as now:
(choose multiple)

o line
o boundary
o centroid
o point

and then somehow know that you are only allowed to select two compatible
ones from the list and somehow specify order (impossible)?

Now it's impossible to select "boundary,line" in v.type window.

but v.clean send to crash my Wish app, only wish app.
Why wish app is more unstable with some commands with some vector layer?

no idea. Could it be warning / error messages triggering the crash?
G_percent?

maybe debug with:
v.clean --tcltk | less
?

I want to test beter...

Thanx

--
________________________________________________________________________
|| Lorenzo Moretti e-mail: lorenzo.moretti@bologna.enea.it ||/|/| ENEA prot/idr Web: http://wwwamb.bologna.enea.it/ || | via Don Fiammelli, 2 FTP: ftp://ftpamb.bologna.enea.it/ (ris.)
~~~~~~ 40128 BOLOGNA - ITALY Ph: +39-0516098086 Fax: +39-0516098131
________________________________________________________________________

Lorenzo Moretti wrote:

Radim Blazek wrote

After I have followed your method (a bit complex)

v.in.ogr -c -o (-c is been used with some complex shapes)
v.type type=boundary,line
v.split
v.type type=line,boundary
v.clean tool=break
db.copy
v.db.connect

and from X11 terminal this method works in OS X and it's fast.
This is a good news but the method is not very easy.

The method is NOT CORRECT! I wrote that here many times:
v.in.ogr -c + v.clean != v.in.ogr

Radim

OK Radim but what is the best method if "v.in.ogr -o" does not work?

The best method is to fix v.in.ogr, i.e. write boundaries in smaller pieces.

Radim