[GRASS-dev] r.hydrodem addon fails on very big rasters (150, 000 x 120, 000 pixels)

GRASS GIS GRASS GIS developer
1

Dear all,

I am testing, trying, learning GRASS 7`s hydrology moduls at the moment. And I am very, very satisfied with the available tools. r.watershed managed to process a really big elevation model (150,000 x 120,000 pixels), which is amazing.

However, when I use r.hydrodem addon on the same big grid with 150,000 x 120,000 pixels (with the memory option) I get the following warning (and finally error) message:

r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro

Load elevation map

100%

WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal value(s)

WARNING: seg_open(): illegal configuration parameter(s)

Remove one cell extremas

100%

Set edge points

Segmentation fault (core dumped)

Exactly the same command on a 45,000 x 20,000 pixel region works fine (and very fast btw.):

Load elevation map

100%

Remove one cell extremas

100%

Set edge points

100%

A* Search…

100%

Processing 2641798 sinks

100%

Write conditioned DEM

100%

Any ideas?

I can process the DEM in tiles (i.e. subdivided into major water regions), so there is a work around…

Almost forgot to mention: I am using GRASS 7 Revision: 58048 on Ubuntu 12.04.2 LTS server and r.hydrodem installed through g.extension today.

BTW. I tried to install all addons (through g.extension) and there were several moduls which did not compile successfully, should I try to collect a list of Addons with compilation problems (and error messages)?

Cheers

Stefan

2

On Thu, Feb 6, 2014 at 1:14 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:

Dear all,

I am testing, trying, learning GRASS 7`s hydrology moduls at the moment. And
I am very, very satisfied with the available tools. r.watershed managed to
process a really big elevation model (150,000 x 120,000 pixels), which is
amazing.

Wow! Please consider to add a note to
http://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_data_processing

(or in a blog entry or the like)

However, when I use r.hydrodem addon on the same big grid with 150,000 x
120,000 pixels (with the memory option) I get the following warning (and
finally error) message:

r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro

...

WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal value(s)

This is an integer overflow of ncols etc:

[neteler@pgis_north r.hydrodem]$ grep nrows seg.c
seg_open(SSEG *sseg, int nrows, int ncols, int row_in_seg, int col_in_seg,

In r.watershed it is defined differently:

[neteler@pgis_north r.watershed]$ grep nrows seg/sseg_open.c
seg_open(SSEG * sseg, GW_LARGE_INT nrows, GW_LARGE_INT ncols, int
row_in_seg, int col_in_seg,
    if (0 > (errflag = segment_format(fd, nrows, ncols,

I suppose that this is an "easy" fix given the experience of r.watershed.
To not interfere, I'll leave the change to the module author.

Best,
markusN

3

On Thu, Feb 6, 2014 at 5:57 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Thu, Feb 6, 2014 at 1:14 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:

Dear all,

I am testing, trying, learning GRASS 7`s hydrology moduls at the moment. And
I am very, very satisfied with the available tools. r.watershed managed to
process a really big elevation model (150,000 x 120,000 pixels), which is
amazing.

Wow! Please consider to add a note to
http://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_data_processing

(or in a blog entry or the like)

However, when I use r.hydrodem addon on the same big grid with 150,000 x
120,000 pixels (with the memory option) I get the following warning (and
finally error) message:

r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro

...

WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal value(s)

This is an integer overflow of ncols etc:

[neteler@pgis_north r.hydrodem]$ grep nrows seg.c
seg_open(SSEG *sseg, int nrows, int ncols, int row_in_seg, int col_in_seg,

In r.watershed it is defined differently:

[neteler@pgis_north r.watershed]$ grep nrows seg/sseg_open.c
seg_open(SSEG * sseg, GW_LARGE_INT nrows, GW_LARGE_INT ncols, int
row_in_seg, int col_in_seg,
    if (0 > (errflag = segment_format(fd, nrows, ncols,

I suppose that this is an "easy" fix given the experience of r.watershed.
To not interfere, I'll leave the change to the module author.

The fix would be relatively easy, I will do it as soon as time permits...

Note that r.hydrodem and r.watershed use the same mechanism, therefore
r.hydrodem can not improve a DEM for r.watershed. The r.hydrodem
module is meant to perform hydrological conditioning (sink removal) of
a DEM when the DEM is used with other modules or other software.
However, to my knowledge no other software exists that is able to
perform surface flow accumulation on such a large raster. Two examples
for sophisticated hydrological modelling packages are TauDEM and
WhiteBox (formerly TAS). The last time I checked, both were not able
to process DEMs with more than 2 billion grid cells. Even r.terraflow,
designed for massive grids, can not process a 150,000 x 120,000 DEM.

Markus M

4

Hei Markus and Markus

Thanks for your reply and the information that r.hydrodem would not improve the results of r.watershed.

The manual of r.hydrodem btw., said different. There it is written e.g. that "If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed (...)" or "the removal of one-cell extrema could already be sufficient to improve r.watershed results in difficult terrain, particularly nearly flat areas." I guess the manual applies / applied for an older version of r.watershed?

Anyway, I am impressed of the performance of the hydrological models in GRASS and I shall add a note about the Performance of GRASS to the wiki.

Cheers
Stefan

-----Original Message-----
From: Markus Metz [mailto:markus.metz.giswork@gmail.com]
Sent: 10. februar 2014 09:03
To: Markus Neteler
Cc: Blumentrath, Stefan; GRASS developers list
Subject: Re: [GRASS-dev] r.hydrodem addon fails on very big rasters (150, 000 x 120, 000 pixels)

On Thu, Feb 6, 2014 at 5:57 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Thu, Feb 6, 2014 at 1:14 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:

Dear all,

I am testing, trying, learning GRASS 7`s hydrology moduls at the
moment. And I am very, very satisfied with the available tools.
r.watershed managed to process a really big elevation model (150,000
x 120,000 pixels), which is amazing.

Wow! Please consider to add a note to
http://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Large_raster_dat
a_processing

(or in a blog entry or the like)

However, when I use r.hydrodem addon on the same big grid with
150,000 x
120,000 pixels (with the memory option) I get the following warning
(and finally error) message:

r.hydrodem input=dem_10m memory=10000 output=dem_10m _hydro

...

WARNING: segment_format(fd,1,-221837149,1,262144,16): illegal
value(s)

This is an integer overflow of ncols etc:

[neteler@pgis_north r.hydrodem]$ grep nrows seg.c seg_open(SSEG *sseg,
int nrows, int ncols, int row_in_seg, int col_in_seg,

In r.watershed it is defined differently:

[neteler@pgis_north r.watershed]$ grep nrows seg/sseg_open.c
seg_open(SSEG * sseg, GW_LARGE_INT nrows, GW_LARGE_INT ncols, int
row_in_seg, int col_in_seg,
    if (0 > (errflag = segment_format(fd, nrows, ncols,

I suppose that this is an "easy" fix given the experience of r.watershed.
To not interfere, I'll leave the change to the module author.

The fix would be relatively easy, I will do it as soon as time permits...

Note that r.hydrodem and r.watershed use the same mechanism, therefore r.hydrodem can not improve a DEM for r.watershed. The r.hydrodem module is meant to perform hydrological conditioning (sink removal) of a DEM when the DEM is used with other modules or other software.
However, to my knowledge no other software exists that is able to perform surface flow accumulation on such a large raster. Two examples for sophisticated hydrological modelling packages are TauDEM and WhiteBox (formerly TAS). The last time I checked, both were not able to process DEMs with more than 2 billion grid cells. Even r.terraflow, designed for massive grids, can not process a 150,000 x 120,000 DEM.

Markus M