[GRASS-user] problem about import of shp file

Natsuko_YOSHIFUJI · March 18, 2009, 7:55am

Dear everyone,

Hello.
I made my first challenge to import Arc-GIS shp file to GRASS.
But I have some trouble.
This may be an elementary question...,but could anyone help me, please?

What I want to do (final destination) is to
1) Import vector map (Arc-GIS shp file) into location1(UTM)
2) Reproject vector data into location2 (lat-lon).
3) Overlay on raster map in location2. I need area information in vector
map (Arc-GIS shp file) .
But I have a trouble in step 1, and area (polygon) cannot be shown...

Outline of what I did is shown below.
Details of commands and error message are shown in attached file.

1) Create UTM new location
47N, but the area was extended to eastward because the area includes a
part of 48N
This seems to succeed.

2) v.in.ogr dsn=lu2543s.shp output=lu2543s
failed.

3) v.in.ogr dsn=lu2543s.shp output=lu2543s -o
Succeeded, but had an error message.
Seems to failed to Import of area and isles...???

4) d.vect lu2543s.
Some map was shown, but had an error to show area.

5) d.what.vect
failed. Problem about topology.

6) v.category input=lu2543s option=report
failed

* I have 6 another files in addition to shp file to be imported
(lu2543s.shp), which is lu2543s.cpg, lu2543s.dbf, lu2543s.prj,
lu2543s.sbn, lu2543s.sbx, lu2543s.shx.
But I am not sure whether I should use them to import shp file correctly
or not...

How can I import shp file correctly?
I hope someone kindly advice me, please...

Thank you.

N. Yoshifuji
2009-3-18

(attachments)

log.txt (3.08 KB)

Markus_GRASS · March 18, 2009, 8:53am

Natsuko YOSHIFUJI wrote:

Dear everyone,

Hello.
I made my first challenge to import Arc-GIS shp file to GRASS.
But I have some trouble.
This may be an elementary question...,but could anyone help me, please?

What I want to do (final destination) is to
1) Import vector map (Arc-GIS shp file) into location1(UTM)
2) Reproject vector data into location2 (lat-lon).
3) Overlay on raster map in location2. I need area information in vector
map (Arc-GIS shp file) .
But I have a trouble in step 1, and area (polygon) cannot be shown...

Outline of what I did is shown below.
Details of commands and error message are shown in attached file.

1) Create UTM new location
47N, but the area was extended to eastward because the area includes a
part of 48N
This seems to succeed.

2) v.in.ogr dsn=lu2543s.shp output=lu2543s
failed.

Projections don't match. The shapefile is in tmerc (Transverse Mercator), not utm (Universal Transverse Mercator). You should create a new location with the projection of the shapefile:
       Dataset PROJ_INFO is:
       name: Transverse Mercator
       proj: tmerc
       datum: wgs84
       a: 6378137.0
       es: 0.006694379990141317
       lat_0: 0
       lon_0: 99
       k: 0.999600
       x_0: 500000
       y_0: 0
       no_defs: defined

3) v.in.ogr dsn=lu2543s.shp output=lu2543s -o
Succeeded, but had an error message.
Seems to failed to Import of area and isles...???

Looks like you ran out of memory? This message <assertion "n" failed: file "node.c", line 48> means that an attempt to allocate more memory failed.
Consequently, cleaning failed, topology was not built and the imported vector is not usable. The solution would be to install more RAM or increase swap space.
It is strongly recommended to use grass-6.4 instead of grass-6.2.3 if possible.

Markus M

Markus_GRASS · March 18, 2009, 10:31am

Natsuko YOSHIFUJI wrote:

3) v.in.ogr dsn=lu2543s.shp output=lu2543s -o
Succeeded, but had an error message.
Seems to failed to Import of area and isles...???

I forgot, it may help to close all other applications to get as much memory as possible. BTW, the procedure to break polygons should use about 20% less memory in grass65 than in grass64 and something between 20% and 40% less memory in grass7 than in grass64.

Natsuko_YOSHIFUJI · March 18, 2009, 11:35am

Dear Mr. Markus Metz ,

Thank you very much for your quick reply!
I misunderstood that tmerc is the same as UTM...

I tried again following your advice, but I have additional questions...
Could you kindly reply me, please?

Projections don't match. The shapefile is in tmerc (Transverse Mercator), not utm (Universal Transverse Mercator). You should create a new location with the projection of the shapefile:
      Dataset PROJ_INFO is:
      name: Transverse Mercator
      proj: tmerc
      datum: wgs84
      a: 6378137.0
      es: 0.006694379990141317
      lat_0: 0
      lon_0: 99
      k: 0.999600
      x_0: 500000
      y_0: 0
      no_defs: defined

3) v.in.ogr dsn=lu2543s.shp output=lu2543s -o
Succeeded, but had an error message.
Seems to failed to Import of area and isles...???


Looks like you ran out of memory? This message <assertion "n" failed: file "node.c", line 48> means that an attempt to allocate more memory failed.
Consequently, cleaning failed, topology was not built and the imported vector is not usable. The solution would be to install more RAM or increase swap space.

I created new tmerc location.
<v.in.ogr dsn=lu2543s.shp output=lu2543s > worked, but I had same error message <ssertion "n" failed: file "node.c", line 48>.
You told me that this is because of shortage of RAM.
So, I tried to use another PC (Grass6.2.2), created tmerc location, and tried to import the same shp file.
Then, I received an error message saying that projections don't match!
In this time, data set projection was reported as follows;

Dataset PROJ_INFO is:
    name: Universe Transverse Mercator
    proj: utm
    datum: wgs84
    a: 6378137.0
    es: 0.006694379990141316
    zone: 47
    no_defs: defined

The data supplier told me that the projection of this shp file is UTM.
So, the message of new PC seems to be correct, and previous message may be wrong.
Anyway, it is very confusing...
In addition, import of shp file was failed, too, probably because of shortage of RAM...

It is strongly recommended to use grass-6.4 instead of grass-6.2.3 if possible.

Can such confusing situation be resolved if I install grass-6.4?

I forgot, it may help to close all other applications to get as much memory as possible. BTW, the procedure to break polygons should use about 20% less memory in grass65 than in grass64 and something between 20% and 40% less memory in grass7 than in grass64.

In grass HP (http://wgrass.media.osaka-cu.ac.jp/grassh/download/index.php),
I can find only GRASS-6.4.
Where can I get grass-6.5 and grass7?

Now I use grass 6.2.2.
Does grass-6.4 use smaller memory than grass 6.2.2?

Thank you in advance.

Natsuko Yoshifuji

Markus_GRASS · March 18, 2009, 12:29pm

Natsuko YOSHIFUJI wrote:

I created new tmerc location.
<v.in.ogr dsn=lu2543s.shp output=lu2543s > worked, but I had same error message <ssertion "n" failed: file "node.c", line 48>.
You told me that this is because of shortage of RAM.

I guess. I hope it's not a bug in grass, but to my knowledge this error has not been reported before and I tested v.in.ogr recently (in 6.4, 6.5, and 7) and also did not get such an error.

So, I tried to use another PC (Grass6.2.2), created tmerc location, and tried to import the same shp file.
Then, I received an error message saying that projections don't match!
In this time, data set projection was reported as follows;

Dataset PROJ_INFO is:
   name: Universe Transverse Mercator
   proj: utm
   datum: wgs84
   a: 6378137.0
   es: 0.006694379990141316
   zone: 47
   no_defs: defined

The data supplier told me that the projection of this shp file is UTM.
So, the message of new PC seems to be correct, and previous message may be wrong.
Anyway, it is very confusing...

This is confusing indeed. There should be a file called lu2543s.prj . This is a plain text file with the projection information for lu2543s.shp, you can open it with any text editor, see what's in there and create a location according to this information. If v.in.ogr still complains that projections don't match, you can use v.in.ogr -o, but something may be wrong with gdal and/or proj4.

Can such confusing situation be resolved if I install grass-6.4?

grass-6.4 has lots of bug fixes and improvements over previous versions, I would always recommend to use grass-6.4 instead of earlier versions. And also update gdal and proj4.

Where can I get grass-6.5 and grass7?

grass-6.5 and grass-7 are only available as source code via svn. If you are not familiar with compiling from source, this can be a bit tricky not to say it could cost you a day or more. Information on how to download source is here: http://trac.osgeo.org/grass/wiki/DownloadSource
There is a wiki page on how to compile and install: http://grass.osgeo.org/wiki/Compile_and_Install
It is easier to install grass-6.4 binaries.

Does grass-6.4 use smaller memory than grass 6.2.2?

Not for breaking polygons.

neteler · March 18, 2009, 1:07pm

On Wed, Mar 18, 2009 at 1:29 PM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:

Natsuko YOSHIFUJI wrote:

...

Where can I get grass-6.5 and grass7?

grass-6.5 and grass-7 are only available as source code via svn.

If desired, I can set up cronjobs to generate also the grass-6.5 and
grass-7 binaries. For grass-7, there is the problem that I don't manage
to compile CAIRO >= 1.5.8 on Fedora FC4 which is used on grass.osgeo.org.
Help (i.e., 32bit RPM) welcome.

Markus

Natsuko_YOSHIFUJI · March 18, 2009, 2:22pm

Dear Mr. Markus Metz,

So, I tried to use another PC (Grass6.2.2), created tmerc location, and tried to import the same shp file.
Then, I received an error message saying that projections don't match!
In this time, data set projection was reported as follows;

Dataset PROJ_INFO is:
  name: Universe Transverse Mercator
  proj: utm
  datum: wgs84
  a: 6378137.0
  es: 0.006694379990141316
  zone: 47
  no_defs: defined

The data supplier told me that the projection of this shp file is UTM.
So, the message of new PC seems to be correct, and previous message may be wrong.
Anyway, it is very confusing...

This is confusing indeed. There should be a file called lu2543s.prj . This is a plain text file with the projection information for lu2543s.shp, you can open it with any text editor, see what's in there and create a location according to this information. If v.in.ogr still complains that projections don't match, you can use v.in.ogr -o, but something may be wrong with gdal and/or proj4.

The contents of lu2543s.prj was as follows.

PROJCS["WGS_1984_UTM_Zone_47N",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",99.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]

There are two projection information: PROJCS["WGS_1984_UTM_Zone_47N" and PROJECTION["Transverse_Mercator"].
I wonder correct projection is UTM47N, isn't it?

Where can I get grass-6.5 and grass7?

grass-6.5 and grass-7 are only available as source code via svn. If you are not familiar with compiling from source, this can be a bit tricky not to say it could cost you a day or more. Information on how to download source is here: http://trac.osgeo.org/grass/wiki/DownloadSource
There is a wiki page on how to compile and install: http://grass.osgeo.org/wiki/Compile_and_Install
It is easier to install grass-6.4 binaries.

Does grass-6.4 use smaller memory than grass 6.2.2?

Not for breaking polygons.

Do you mean that installing grass6.4 will not contribute to solve memory-shortage problem?
But, I am not familiar with compiling from source...
If I can prepare smaller shape file whose area is a part of whole area of lu2543s.shp, is there a possibility that I can avoid memory-shortage problem?

Thank you and with regards,

Natsuko YOSHIFUJI

Natsuko_YOSHIFUJI · March 18, 2009, 2:46pm

On 2009/03/18, at 22:07, Markus Neteler wrote:

On Wed, Mar 18, 2009 at 1:29 PM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:

Natsuko YOSHIFUJI wrote:

...

Where can I get grass-6.5 and grass7?

grass-6.5 and grass-7 are only available as source code via svn.

If desired, I can set up cronjobs to generate also the grass-6.5 and
grass-7 binaries. For grass-7, there is the problem that I don't manage
to compile CAIRO >= 1.5.8 on Fedora FC4 which is used on grass.osgeo.org.
Help (i.e., 32bit RPM) welcome.

Thank you very very much!!!!!
I hope binaries of grass-6.5 or grass7.

I am not so familiar with PC, so I try to tell you my PC environment anyway.

There are 3 type of PC around me.
If possible, binaries for Mac is preferable because mac has the biggest memory,
but binaries for linux and windows+cygwin is available.

PC1 (note pc)
Windows XP professional + cygwin
1.5GB RAM
Grass - 6.2.3 is already installed
* I used this pc to import shp file and failed with an error message <n failed "node.c", line48 Aborted (core dumped)>.
* Grass on this pc reported that lu2543s.shp is tmerc, but it may not be true...

PC2
Fedora Core 5
622MB RAM
Grass-6.2.2 is already installed
* import of shp file was forced to stop, probably because of shortage of memory...?

PC1
Mac OS 10.5.6
2.1GHz Intel Core 2 Duo
3GB 667 MHz DDR2 SDRAM
Grass-6.3 is already installed
* import of shp file was failed with an error message as follows:
Break polygons:
Registering points ... 17107648v.in.ogr(1388) malloc: *** mmap(size=624242688) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
ERROR: G_realloc: out of memory

With best regards,

Natsuko YOSHIFUJI

Markus_GRASS · March 18, 2009, 5:26pm

Natsuko YOSHIFUJI wrote:

The contents of lu2543s.prj was as follows.

PROJCS["WGS_1984_UTM_Zone_47N",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",99.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]

There are two projection information: PROJCS["WGS_1984_UTM_Zone_47N" and PROJECTION["Transverse_Mercator"].
I wonder correct projection is UTM47N, isn't it?

You said the data provider said it should be UTM, so I would use UTM.

Do you mean that installing grass6.4 will not contribute to solve memory-shortage problem?

Probably not, I'm afraid. From your other mail I could see that there is quite a bit of memory on the Mac, but that runs out of memory too. Can't you increase swap space on the Mac or for Windows XP?
Just curious, what is the size of lu2543s.shp? And, is it in the public domain so that you can give me a link and I can get it myself from somewhere?

If I can prepare smaller shape file whose area is a part of whole area of lu2543s.shp, is there a possibility that I can avoid memory-shortage problem?

Yes, but there is no need to prepare a smaller shape file, there are two different options for that in v.in.ogr, either -r or spatial=xmin,ymin,xmax,ymax, see manual.

Best,

Markus

PS: Please don't call me Mr. Markus Metz, Markus is really enough

neteler · March 19, 2009, 9:29am

On Wed, Mar 18, 2009 at 3:46 PM, Natsuko YOSHIFUJI
<natsuko.yoshifuji@gmail.com> wrote:

On 2009/03/18, at 22:07, Markus Neteler wrote:

Natsuko YOSHIFUJI wrote:

...

Where can I get grass-6.5 and grass7?

grass-6.5 and grass-7 are only available as source code via svn.

If desired, I can set up cronjobs to generate also the grass-6.5 and
grass-7 binaries. For grass-7, there is the problem that I don't manage
to compile CAIRO >= 1.5.8 on Fedora FC4 which is used on grass.osgeo.org.
Help (i.e., 32bit RPM) welcome.

Thank you very very much!!!!!
I hope binaries of grass-6.5 or grass7.

Here we are:

Weekly Binaries 6.4.0svn:
http://grass.osgeo.org/grass64/binary/linux/snapshot/

Weekly Binaries 6.5.svn:
http://grass.osgeo.org/grass65/binary/linux/snapshot/

Once I receive help from a community member for the CAIRO >= 1.5.8 on Fedora FC4
I can also set up weekly GRASS 7 binaries (the missing dependency breaks
the cronjob).

Markus