I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621
features) using v.in.ogr, but each attempt fails at the 'break
boundaries'' stage (at 98%!):
GRASS 7.0.svn (NZTM2000):~ > v.in.ogr
dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o
Projection of input dataset and current location appear to match
Layer: ni_nzmg
Counting polygons for 211621 features...
Importing map 211621 features...
100%
-----------------------------------------------------
Building topology for vector map <ni_lcdb2_tmp@ni>...
Registering primitives...
334611 primitives registered
18459167 vertices registered
Number of nodes: 211681
Number of primitives: 334611
Number of points: 0
Number of lines: 0
Number of boundaries: 334611
Number of centroids: 0
Number of areas: -
Number of isles: -
-----------------------------------------------------
WARNING: Cleaning polygons, result is not guaranteed!
-----------------------------------------------------
Break polygons:
100%
100%
-----------------------------------------------------
Remove duplicates:
100%
-----------------------------------------------------
Break boundaries:
ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at
allocation.c:82
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done 150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
My guess:
7GB is a lot.
334611 primitives are many, but not that much,
but:
you got 18 millions vertices! (18459167)
That seems a lot.
Depending on how tangled your lines are, a potential mass of features can be generated.
Just a guess,
best,
Achim
Am 26.11.2010 03:14, schrieb Pierre Roudier:
Dear list,
I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621
features) using v.in.ogr, but each attempt fails at the 'break
boundaries'' stage (at 98%!):
GRASS 7.0.svn (NZTM2000):~> v.in.ogr
dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o
Projection of input dataset and current location appear to match
Layer: ni_nzmg
Counting polygons for 211621 features...
Importing map 211621 features...
100%
-----------------------------------------------------
Building topology for vector map<ni_lcdb2_tmp@ni>...
Registering primitives...
334611 primitives registered
18459167 vertices registered
Number of nodes: 211681
Number of primitives: 334611
Number of points: 0
Number of lines: 0
Number of boundaries: 334611
Number of centroids: 0
Number of areas: -
Number of isles: -
-----------------------------------------------------
WARNING: Cleaning polygons, result is not guaranteed!
-----------------------------------------------------
Break polygons:
100%
-----------------------------------------------------
Remove duplicates:
100%
-----------------------------------------------------
Break boundaries:
ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at
allocation.c:82
v.stats _IS_ very slow (for many primitives).
Maybe you could analyse the database, where you can calculate the statistics you like. For spatial statistics you can analyse uploaded coordinates of centroids or sizes of area.
Hope it helps a bit,
Achim
Am 26.11.2010 12:32, schrieb razmjooeis@faunalia.co.uk:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done 150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done 150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
On Fri, Nov 26, 2010 at 10:41 AM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done 150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
On Fri, Nov 26, 2010 at 3:14 AM, Pierre Roudier
<pierre.roudier@gmail.com> wrote:
Dear list,
I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621
features) using v.in.ogr, but each attempt fails at the 'break
boundaries'' stage (at 98%!):
GRASS 7.0.svn (NZTM2000):~ > v.in.ogr
dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o
Projection of input dataset and current location appear to match
Layer: ni_nzmg
Counting polygons for 211621 features...
Importing map 211621 features...
100%
-----------------------------------------------------
Building topology for vector map <ni_lcdb2_tmp@ni>...
Registering primitives...
334611 primitives registered
18459167 vertices registered
Number of nodes: 211681
Number of primitives: 334611
Number of points: 0
Number of lines: 0
Number of boundaries: 334611
Number of centroids: 0
Number of areas: -
Number of isles: -
-----------------------------------------------------
WARNING: Cleaning polygons, result is not guaranteed!
-----------------------------------------------------
Break polygons:
100%
100%
-----------------------------------------------------
Remove duplicates:
100%
-----------------------------------------------------
Break boundaries:
ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at
allocation.c:82
The shp loads without any problems on QGIS.
OS: Opensuse 11.3, with 7Gb RAM.
Is this a 64 bit OS?
I have previously successfully imported larger shapefiles, more than
double in size: >400,000 areas, >40 million vertices, peak memory
consumption was close to 4GB. The import of your shapefile would
approximately require about 2GB of memory, which should not be a
problem with 7GB available, as long as it's all 64 bit and not 32 bit.
On Fri, Nov 26, 2010 at 10:41 AM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done
150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole
set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done
150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
I can't reproduce this in the North Carolina sample dataset, mapset user1 with
g.copy vect=boundary_county@PERMANENT,myboundary_county
v.rast.stats2 -c vector=myboundary_county@user1
raster=elev_state_500m@PERMANENT colprefix=alt
Can you?
Markus M
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done
150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
I can't reproduce this in the North Carolina sample dataset, mapset user1
with
g.copy vect=boundary_county@PERMANENT,myboundary_county
v.rast.stats2 -c vector=myboundary_county@user1
raster=elev_state_500m@PERMANENT colprefix=alt
Can you?
Markus M
On Fri, Nov 26, 2010 at 12:32 PM, <razmjooeis@faunalia.co.uk> wrote:
Dear list
I am trying to run v.rast.stats for a large vector file. It contains
around 700000 buildings representing buildings.
here is my g.region info
g.region -p
projection: 99 (Transverse Mercator)
zone: 0
datum: osgb36
ellipsoid: airy
north: 183010.45829976
south: 153516.19024768
west: 513122.49174945
east: 555953.95182415
nsres: 4.99987592
ewres: 5.00017045
rows: 5899
cols: 8566
cells: 50530834
THe problem I am having is the process is extremely slow. It has done
150
features out of 700000 areas after 2 hours.
I need just the average raster value within each building (feature).
My questions are:
- Is there something I can do to speed up the process?
- Is there a way just to get the average value rather than the whole
set
of statistical parameters (I am not even using -e option)
You could try v.rast.stats2 from grass-addons, it should complete in
less than 1 hour. When using v.rast.stats2 and grass 6.4, you also
need r,univar.zonal from grass-addons (not needed for grass 6.5).
# create array with new column names
col=( `cat "$COLNAMETMP" | tr '\n' ' '` )
This looks like a bash-ism. In case that /bin/sh isn't bash
on Sab's machine but another shell it is likely to fail.
Hmm. Since you (Markus N) are listed as main author and I merely try
to promote v.rast.stats2, and my knowledge of scripting is limited to
bash, do you have a suggestion about how to make this portable to
non-bash?
# create array with new column names
col=( `cat "$COLNAMETMP" | tr '\n' ' '` )
This looks like a bash-ism. In case that /bin/sh isn't bash
on Sab's machine but another shell it is likely to fail.
Hmm. Since you (Markus N) are listed as main author and I merely try
to promote v.rast.stats2, and my knowledge of scripting is limited to
bash, do you have a suggestion about how to make this portable to
non-bash?
# create array with new column names
col=( `cat "$COLNAMETMP" | tr '\n' ' '` )
This looks like a bash-ism. In case that /bin/sh isn't bash
on Sab's machine but another shell it is likely to fail.
Hmm. Since you (Markus N) are listed as main author and I merely try
to promote v.rast.stats2, and my knowledge of scripting is limited to
bash, do you have a suggestion about how to make this portable to
non-bash?
# create array with new column names
col=( `cat "$COLNAMETMP" | tr '\n' ' '` )
This looks like a bash-ism. In case that /bin/sh isn't bash
on Sab's machine but another shell it is likely to fail.
Hmm. Since you (Markus N) are listed as main author and I merely try
to promote v.rast.stats2, and my knowledge of scripting is limited to
bash, do you have a suggestion about how to make this portable to
non-bash?
# create array with new column names
col=( `cat "$COLNAMETMP" | tr '\n' ' '` )
This looks like a bash-ism. In case that /bin/sh isn't bash
on Sab's machine but another shell it is likely to fail.
Hmm. Since you (Markus N) are listed as main author and I merely try
to promote v.rast.stats2, and my knowledge of scripting is limited to
bash, do you have a suggestion about how to make this portable to
non-bash?
I have uploaded an attempt to fix the problem to Addons-SVN.
