A possible bug in ../diglib/cindex.c.
This host has 4G RAM and 16G swap memory. With X running top shows 740M
RAM and 15.3G swap free. I'm trying to run a spatial analysis program which
fails after 2.5-4.7 hours (depending on the contour resolutions). This
morning after 2.5 hours it failed with this message:
ERROR: G_realloc: unable to allocate 52980000 bytes of memory at
lib/vector/diglib/cindex.c:113
which is approximately 52M.
This is cindex.c (it's the same in the latest svn checkouts of 7.0, 7.2,
and 7.3):
/* Add new cat - line record */
ci = &(Plus->cidx[si]);
if (ci->n_cats == ci->a_cats) {
ci->a_cats += 5000;
ci->cat = G_realloc(ci->cat, ci->a_cats * 3 * sizeof(int));
}
lines 109-114.
While waiting for an answer -- or a fix -- I'll kill X and try running
r.contour from a console and see if the extra memory does the job. My
project is stalled until I can create vector contours from the LiDAR raster
maps.
TIA,
Rich
On Thu, Sep 22, 2016 at 5:45 PM, Rich Shepard <rshepard@appl-ecosys.com> wrote:
A possible bug in ../diglib/cindex.c.
This host has 4G RAM and 16G swap memory. With X running top shows 740M
RAM and 15.3G swap free. I'm trying to run a spatial analysis program which
fails after 2.5-4.7 hours (depending on the contour resolutions). This
morning after 2.5 hours it failed with this message:
ERROR: G_realloc: unable to allocate 52980000 bytes of memory at
lib/vector/diglib/cindex.c:113
which is approximately 52M.
Apparently there are less than 52 MB of free memory left. The
resulting vector contours must be quite large. Try to
export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Maybe also use a larger step or less levels.
HTH,
Markus M
This is cindex.c (it's the same in the latest svn checkouts of 7.0, 7.2,
and 7.3):
/* Add new cat - line record */
ci = &(Plus->cidx[si]);
if (ci->n_cats == ci->a_cats) {
ci->a_cats += 5000;
ci->cat = G_realloc(ci->cat, ci->a_cats * 3 * sizeof(int));
}
lines 109-114.
While waiting for an answer -- or a fix -- I'll kill X and try running
r.contour from a console and see if the extra memory does the job. My
project is stalled until I can create vector contours from the LiDAR raster
maps.
TIA,
Rich
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
On Fri, 23 Sep 2016, Markus Metz wrote:
Apparently there are less than 52 MB of free memory left. The resulting
vector contours must be quite large. Try to export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Markus,
I assumed that grass did not see the available memory, but did not realize
it would not use the available swap memory. I'll try exporting that
environment variable and re-run r.contour.
Maybe also use a larger step or less levels.
The second time I almost doubled the step, but it still failed with an
out-of-memory error.
Thanks,
Rich
On Fri, 23 Sep 2016, Markus Metz wrote:
Apparently there are less than 52 MB of free memory left. The resulting
vector contours must be quite large. Try to export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Maybe also use a larger step or less levels.
Markus,
Tried both. Still cannot complete the creation of contours from the
raster.
export GRASS_VECTOR_LOWMEM=1
time r.contour --overwrite in=highest_hit_2010 out=highest_hit_contours step=0.5
WARNING: Vector map <highest_hit_contours> already exists and will be
overwritten
Reading data...
100%
Displacing data...
100%
Writing vector contours (total levels 1313)...
100%
WARNING: 212116 crossings found
Writing attributes...
Building topology for vector map <highest_hit_contours@topography>...
Registering primitives...
Current region rows: 15367, cols: 10015
ERROR: G_realloc: unable to allocate 131280000 bytes of memory at
lib/vector/diglib/cindex.c:113
real 219m0.083s
user 92m5.866s
sys 47m24.872s
I created a mask of the raster DEM using the project bounds (which I set
as the computational region) yet it appears that the entire DEM coverage is
being converted to contours and failing.
Thanks,
Rich
On Fri, 23 Sep 2016, Markus Metz wrote:
Apparently there are less than 52 MB of free memory left. The resulting
vector contours must be quite large. Try to export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Maybe also use a larger step or less levels.
Took a different approach by reducing the size of the source DEMs (which
cover a very large area compared to the area of interest).
Using v.to.rast I converted the project boundary vector to a raster file,
then made it into a mask with r.mask. The mask then limited the area of the
source DEMs with g.copy.
Now re-running r.contour on the bare earth DEM; it should easily fit into
the 4G RAM on this host.
Thanks, Markus,
Rich
Rich Shepard wrote:
I assumed that grass did not see the available memory, but did not realize
it would not use the available swap memory.
Is GRASS built for 32-bit (x86) or 64-bit (x86-64 aka amd64)? Running
"file" on e.g. r.contour will tell you.
A 32-bit process can't use more than 4 GiB of memory regardless of how
much RAM or swap is available. The amount available to malloc() etc
will be somewhat lower due to reservations for the kernel, stack, etc.
--
Glynn Clements <glynn@gclements.plus.com>
On 23/09/16 20:51, Rich Shepard wrote:
On Fri, 23 Sep 2016, Markus Metz wrote:
Apparently there are less than 52 MB of free memory left. The resulting
vector contours must be quite large. Try to export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Maybe also use a larger step or less levels.
Took a different approach by reducing the size of the source DEMs (which
cover a very large area compared to the area of interest).
Using v.to.rast I converted the project boundary vector to a raster file,
then made it into a mask with r.mask. The mask then limited the area of the
source DEMs with g.copy.
FYI: Since GRASS 7 r.mask actually has a vector= parameter which allows setting the mask directly from a vector map.
Now re-running r.contour on the bare earth DEM; it should easily fit into
the 4G RAM on this host.
Thanks, Markus,
Rich
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
On 23/09/16 20:51, Rich Shepard wrote:
On Fri, 23 Sep 2016, Markus Metz wrote:
Apparently there are less than 52 MB of free memory left. The resulting
vector contours must be quite large. Try to export GRASS_VECTOR_LOWMEM=1
before running r.contour.
Maybe also use a larger step or less levels.
Took a different approach by reducing the size of the source DEMs (which
cover a very large area compared to the area of interest).
Using v.to.rast I converted the project boundary vector to a raster file,
then made it into a mask with r.mask. The mask then limited the area of the
source DEMs with g.copy.
FYI: In grass7, r.mask has a 'vector' parameter which allows you create a mask directly from a vector map.
And AFAIK, no need to g.copy. Just run your analysis with the mask set (and the region also correctly set to only the area of interest).
Moritz
On Sat, 1 Oct 2016, Moritz Lennert wrote:
FYI: In grass7, r.mask has a 'vector' parameter which allows you create a
mask directly from a vector map.
Moritz,
I saw that and decided to try that today since I'm not having the results
I want.
And AFAIK, no need to g.copy. Just run your analysis with the mask set
(and the region also correctly set to only the area of interest).
To check my understanding: the project region has been set to Oregon North
NAD83 (HARM) with units of meters. I create the MASK using the analytical
boundary vector file, then set the computational region to that vector. Now
when I run r.contours with in=lidar_dem only those points within the MASK
will be converted to vector contours. Is this correct?
Thanks,
Rich
On Sat, 1 Oct 2016, Glynn Clements wrote:
Is GRASS built for 32-bit (x86) or 64-bit (x86-64 aka amd64)? Running
"file" on e.g. r.contour will tell you.
Glynn,
No need to run file; this host is running a 32-bit CPU; the replacement
machine is 64-bit.
A 32-bit process can't use more than 4 GiB of memory regardless of how
much RAM or swap is available. The amount available to malloc() etc will
be somewhat lower due to reservations for the kernel, stack, etc.
Thanks, I forgot about this since most of what I do does not require large
amounts of memory.
With the build configured with openmp will some grass modules use multiple
processors on both 32- and 64-bit hosts?
Regards,
Rich
Le 1 octobre 2016 14:42:49 GMT+02:00, Rich Shepard <rshepard@appl-ecosys.com> a écrit :
On Sat, 1 Oct 2016, Moritz Lennert wrote:
FYI: In grass7, r.mask has a 'vector' parameter which allows you
create a
mask directly from a vector map.
Moritz,
I saw that and decided to try that today since I'm not having the
results
I want.
And AFAIK, no need to g.copy. Just run your analysis with the mask
set
(and the region also correctly set to only the area of interest).
To check my understanding: the project region has been set to Oregon
North
NAD83 (HARM) with units of meters. I create the MASK using the
analytical
boundary vector file, then set the computational region to that vector.
Now
when I run r.contours with in=lidar_dem only those points within the
MASK
will be converted to vector contours. Is this correct?
The mask limits the area in which data is read, so only the parts of the dem inside the mask will be read, so contours will only be created for that area. But if you want to make sure, create a very small mask area and try 
Moritz
Thanks,
Rich
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
On Sat, 1 Oct 2016, Moritz Lennert wrote:
The mask limits the area in which data is read, so only the parts of the
dem inside the mask will be read, so contours will only be created for
that area. But if you want to make sure, create a very small mask area and
try
Moritz,
Thanks for confirming. Actually, the analytical boundary is very small
compared to the LiDAR data areal coverage.
Much appreciated,
Rich