[GRASS-dev] [GRASS GIS] #2356: r.stream.basisns uses more memory than given in memory option

GDAL · June 30, 2014, 5:43am

#2356: r.stream.basisns uses more memory than given in memory option
--------------------------------------+-------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Keywords: r.stream.basisns, memory | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------
I tried to run r.stream.basins on using large flow direction / stream maps
generated with r.watershed.

With m-flag and memory option set to 35GB (~2/3 of the RAM of my system)
the process got killed (by out-of-mmemory-killer) after input maps were
read.

So, I re-ran the module with a smaller amount of memory allocated (20 and
10 GB) in the memory option and monitored memory usage (UNIX top command).
There I found that r.stream.basisns used 2x the amount of RAM allowed, on
each input map. In other words, with memory=10000, it used

20GB for reading flow direction map, plus

20GB for reading streams map (40GB in total).

Finally, the process gets killed when it starts finding nodes.

I am using Ubuntu 12.04 LTS 64bit and GRASS 7.0 (revision 58620).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2356>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 8:24am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------
Changes (by hamish):

* keywords: r.stream.basisns, memory => r.stream.basins, memory
* component: Raster => Addons

Comment:

were the input maps CELL, FCELL or DCELL?

fwiw here's the code,

{{{
  opt_swapsize->description = _("Maximum memory used in memory swap mode
(MB)");
  opt_swapsize->answer = "300";
...
  number_of_segs = (int)atof(opt_swapsize->answer);
  number_of_segs = number_of_segs < 32 ? (int)(32 / 0.12) : number_of_segs
/ 0.12;
}}}

if the stream_rast= option is used then two ram_create_map()s happen.
Likely the above number_of_segs isn't taking that into account.

did it work ok when you halved the memory= option?

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 8:26am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by hamish):

I wrote:
> if the stream_rast= option is used then two ram_create_map()s happen.

sorry, two seg_create_map() happen without stream_rast=, three if with it.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 8:41am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by sbl):

Both input maps (dirs and streams) were generated by r.watershed and are
CELL.

r.stream.basins worked just fine when I set the memory option to ~1/7 of
the available RAM.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 8:50am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by hamish):

how many rows and columns in the raster array?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 8:53am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by sbl):

Both input maps had:
Rows: 180752
Columns: 141312
The region followed exactly those maps (it was not changed after maps were
generated with r.watershed).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 9:41am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by hamish):

so for an unsegmented cell map, 180752 * 141312 * 4 bytes = ~97GB to hold
one whole map in ram.

r.stream.basins/io.h defines segments as 256x256, so for memory=10000 (mb)
{{{
seg_create_map(&map_dirs, SROWS, SCOLS, number_of_segs, CELL_TYPE);
}}}

we get
bytes = ~20.8GB, and needs that for each of the two or three open maps.

can you try r61149?

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 9:55am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by sbl):

FYI, 20.8GB is exactly the amount of memory consumption for reading the
input maps reported by the top command, when I ran r.stream.basins with
memory=10000.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · July 4, 2014, 10:07am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by hamish):

Replying to [comment:7 sbl]:
> FYI, 20.8GB is exactly the amount of memory consumption for reading
> the input maps reported by the top command, when I ran
> r.stream.basins with memory=10000.

ok, then the math works.

can someone familiar with this module give a 3rd party confirmation that
the memory= option is now properly respected?

thanks,
Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2356#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · November 3, 2014, 8:05am

#2356: r.stream.basisns uses more memory than given in memory option
-------------------------------------+--------------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.1.0
Component: Addons | Version: svn-trunk
Keywords: r.stream.basins, memory | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------

Comment(by neteler):

TODO for all r.stream.* modules (reported by mmetz in grass-dev):

Most importantly it is the NULL handling that needs to be fixed, and
synced between the ram and disk modes. More generally, the ram and
disk modes need to be synced and validated. For ease of maintenance,
the ram mode could be removed.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2356#comment:9>
GRASS GIS <http://grass.osgeo.org>