Hi Folks,
I have an fcell grid of elevations for the state of North Carolina (51000 rows 133000 columns 6783000000 cells) . I tried to run r.terraflow in GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I patched them according to Glynn’s email , http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and tried again ( Would it be better to change the dimension variable to int instead of short int?) .
This time my Streams file builds to about 26 GB and then r.terraflow bombs with :
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 808.00M (847249408) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
r.terraflow: grass2str.h:145: AMI_STREAM*
cell2stream(char*, elevation_type, long int*) [with T =
float, elevation_type = float]: Assertion `nrows * ncols ==
str->stream_len()’ failed.
The memory size is interesting, because I’m giving it 8GB of RAM out of 16 GB in the command. The temp directory has about 900GB of space, so it has plenty of room .
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.
I have an fcell grid of elevations for the state of North Carolina (51000
rows 133000 columns 6783000000 cells) . I tried to run r.terraflow in
GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I
patched them according to Glynn's email , http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and
tried again ( Would it be better to change the dimension variable to int
instead of short int?) .
I think that dimension_type has to be used consistently. There is a
class, "ijBaseType" which contains a pair of dimension_type members
(i.e. it's a coordinate pair), and there are also several classes
which derive from ijBaseType. If r.terraflow creates large arrays of
instances of ijBaseType or its subclasses, changing dimension_type to
32-bit could significantly increase memory consumption.
This time my Streams file builds to about 26 GB and then r.terraflow bombs
with :
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 808.00M (847249408) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
r.terraflow: grass2str.h:145: AMI_STREAM<T>*
cell2stream(char*, elevation_type, long int*) [with T =
float, elevation_type = float]: Assertion `nrows * ncols ==
str->stream_len()' failed.
The memory size is interesting, because I'm giving it 8GB of RAM out of 16
GB in the command. The temp directory has about 900GB of space, so it has
plenty of room .
Hi Folks,
I have an fcell grid of elevations for the state of North Carolina (51000
rows 133000 columns 6783000000 cells) . I tried to run r.terraflow in
GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I
patched them according to Glynn's email , http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and
tried again ( Would it be better to change the dimension variable to int
instead of short int?) .
This time my Streams file builds to about 26 GB and then r.terraflow bombs
with :
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 808.00M (847249408) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
r.terraflow: grass2str.h:145: AMI_STREAM<T>*
cell2stream(char*, elevation_type, long int*) [with T =
float, elevation_type = float]: Assertion `nrows * ncols ==
str->stream_len()' failed.
The memory size is interesting, because I'm giving it 8GB of RAM out of 16
GB in the command.
Note that the memory manager of the iostream lib used by r.terraflow
can only handle up to 2047 MB of RAM because the memory option in MB
is internally converted to Byte and stored as int, thus the max
recognized value is 2^31 - 1 Byte.
Note that the memory manager of the iostream lib used by r.terraflow
can only handle up to 2047 MB of RAM because the memory option in MB
is internally converted to Byte and stored as int, thus the max
recognized value is 2^31 - 1 Byte.
The iostream library uses size_t. However, r.terraflow itself had a
bug (scale then cast instead of cast then scale), which should be
fixed by r47641.
><Doug_Newcomb@fws.gov> wrote: >> >> Hi Folks, >> I have an fcell grid of elevations for the state of North Carolina (51000 >> rows 133000 columns 6783000000 cells) . I tried to run r.terraflow in >> GRASS7 ( 8/8/2011 svn snapshot) and ran into the dimension limits. So I >> patched them according to Glynn's email , >> ``http://www.osgeo.org/pipermail/grass-user/2004-February/024722.html and >> tried again ( Would it be better to change the dimension variable to int >> instead of short int?) . >> >> This time my Streams file builds to about 26 GB and then r.terraflow bombs >> with : >> >> MFD flow direction >> D8CUT=999999986991104.000000 >> Memory size: 808.00M (847249408) bytes >> Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. >> r.terraflow: grass2str.h:145: AMI_STREAM<T>* >> cell2stream(char*, elevation_type, long int*) [with T = >> float, elevation_type = float]: Assertion nrows * ncols ==>> str->stream_len()’ failed.>>>> The memory size is interesting, because I’m giving it 8GB of RAM out of 16>> GB in the command.`
>Note that the memory manager of the iostream lib used by r.terraflow >can only handle up to 2047 MB of RAM because the memory option in MB >is internally converted to Byte and stored as int, thus the max >recognized value is 2^31 - 1 Byte.
>Markus M
This appears to be kind of like pulling on a corner of a spiderweb. :-)
Should the limit to memory be indicated in the gui? Would there be any benefit to other parts of GRASS in bumping up this memory limit? I can see the reason on 32 bit windows XP, but that is starting to go away.
The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.
> The iostream library uses size_t. However, r.terraflow
> itself had a bug (scale then cast instead of cast then scale),
> which should be fixed by r47641.
Should the limit to memory be indicated in the gui?
Would there be any benefit to other parts of GRASS in bumping up this
memory limit? I can see the reason on 32 bit windows XP, but that is
starting to go away.
Arbitrary 32-bit limits should ideally be fixed, but it's a game of
Whac-a-Mole. The problem usually arises as a result of using "int"
instead of long, size_t, off_t, etc.
I've fixed the original issue and one other with r47665: