[GRASS-dev] r.watershed2 with MFD

GRASS GIS GRASS GIS developer
1

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it, but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

A first result for flow accumulation is here:
http://markus.metz.giswork.googlepages.com/rw2_mfd_large.jpg
This output has been calculated from elev_lid1972_1m in the North
Carolina sample dataset.

Please compare to the original result and flow accumulation of other
modules as available here:
http://www.grassbook.org/gallery/chapters/s050404f070_flow.jpg
a) old r.watershed
b) r.flow
c) r.terraflow with MFD
d) r.sim.water

References:
original MFD algorithm, also referenced to by r.terraflow
Quinn, P., Beven, K., Chevallier, P., and Planchon, 0. 1991. The
prediction of hillslope flow paths for distributed hydrological
modelling using digital terrain models, Hydrol. Process., 5, 59-79.

The algorithm of Quinn et al. (1991) was modified by Holmgren (1994) and
is implemented here:
Holmgren, P. 1994. Multiple flow direction algorithms for runoff
modelling in grid based elevation models: an empirical evaluation.
Hydrol. Process., 8, 327-334.

I added a convergence factor after Holmgren (1994): a lower convergence
factor results in more diverging flow, output will be similar to
r.terraflow in MFD, a higher convergence factor will result in more
converging flow, getting more similar to, but not identical to SFD.

Additional memory requirements are not noticeable ( <2 KB) and
independent of the region size, additional time required is a few
seconds for elev_lid1972_1m, increasing by a few more seconds with
higher convergence factor.

The color coding suggested by Hamish with absolute logarithm works nice
for most outputs, I have hardcoded it for flow accumulation output.
The color coding of the example image above has been adjusted a bit to
show more detail, but abs log of rainbow looks nice too!

Markus Metz

2

Sorry, a mistake in the additional time requirements for MFD on
elev_lid1972_1m:
r.watershed2 with SFD: 0.7 seconds
r.watershed2 with MFD: 1.0 seconds

Markus Metz wrote:

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it, but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

A first result for flow accumulation is here:
http://markus.metz.giswork.googlepages.com/rw2_mfd_large.jpg
This output has been calculated from elev_lid1972_1m in the North
Carolina sample dataset.

Please compare to the original result and flow accumulation of other
modules as available here:
http://www.grassbook.org/gallery/chapters/s050404f070_flow.jpg
a) old r.watershed
b) r.flow
c) r.terraflow with MFD
d) r.sim.water

References:
original MFD algorithm, also referenced to by r.terraflow
Quinn, P., Beven, K., Chevallier, P., and Planchon, 0. 1991. The
prediction of hillslope flow paths for distributed hydrological
modelling using digital terrain models, Hydrol. Process., 5, 59-79.

The algorithm of Quinn et al. (1991) was modified by Holmgren (1994) and
is implemented here:
Holmgren, P. 1994. Multiple flow direction algorithms for runoff
modelling in grid based elevation models: an empirical evaluation.
Hydrol. Process., 8, 327-334.

I added a convergence factor after Holmgren (1994): a lower convergence
factor results in more diverging flow, output will be similar to
r.terraflow in MFD, a higher convergence factor will result in more
converging flow, getting more similar to, but not identical to SFD.

Additional memory requirements are not noticeable ( <2 KB) and
independent of the region size, additional time required is a few
seconds for elev_lid1972_1m, increasing by a few more seconds with
higher convergence factor.

The color coding suggested by Hamish with absolute logarithm works nice
for most outputs, I have hardcoded it for flow accumulation output.
The color coding of the example image above has been adjusted a bit to
show more detail, but abs log of rainbow looks nice too!

Markus Metz

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

On 05/12/08 10:55, Markus Metz wrote:

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it

Great job !!! Thanks a lot.

but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

After the discussion of the past days, is there any reason not to apply you changes directly in SVN ?

Moritz

4

On Fri, Dec 5, 2008 at 1:55 AM, Markus Metz <markus_metz@gmx.de> wrote:

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it, but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

A first result for flow accumulation is here:
http://markus.metz.giswork.googlepages.com/rw2_mfd_large.jpg
This output has been calculated from elev_lid1972_1m in the North
Carolina sample dataset.

Please compare to the original result and flow accumulation of other
modules as available here:
http://www.grassbook.org/gallery/chapters/s050404f070_flow.jpg
a) old r.watershed
b) r.flow
c) r.terraflow with MFD
d) r.sim.water

References:
original MFD algorithm, also referenced to by r.terraflow
Quinn, P., Beven, K., Chevallier, P., and Planchon, 0. 1991. The
prediction of hillslope flow paths for distributed hydrological
modelling using digital terrain models, Hydrol. Process., 5, 59-79.

The algorithm of Quinn et al. (1991) was modified by Holmgren (1994) and
is implemented here:
Holmgren, P. 1994. Multiple flow direction algorithms for runoff
modelling in grid based elevation models: an empirical evaluation.
Hydrol. Process., 8, 327-334.

I added a convergence factor after Holmgren (1994): a lower convergence
factor results in more diverging flow, output will be similar to
r.terraflow in MFD, a higher convergence factor will result in more
converging flow, getting more similar to, but not identical to SFD.

Additional memory requirements are not noticeable ( <2 KB) and
independent of the region size, additional time required is a few
seconds for elev_lid1972_1m, increasing by a few more seconds with
higher convergence factor.

The color coding suggested by Hamish with absolute logarithm works nice
for most outputs, I have hardcoded it for flow accumulation output.
The color coding of the example image above has been adjusted a bit to
show more detail, but abs log of rainbow looks nice too!

Markus Metz

Very impressive Markus! I certainly wasn't expecting someone to
implement such a complex feature so quickly. Is there any way to
cleanly merge this with the existing r.watershed code in SVN so that
we can test it?

Great job!

Dylan

5

Dylan Beaudette wrote:

On Fri, Dec 5, 2008 at 1:55 AM, Markus Metz <markus_metz@gmx.de> wrote:
  

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it, but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

Is there any way to
cleanly merge this with the existing r.watershed code in SVN so that
we can test it?
  

Also considering Michael's remarks, I suggets that I then not submit to
grass-addons, but to grass 7. Not to grass-6.4.x, because this code is
now more experimental and documentation still needs to be written.
The MFD algorithm seems to be robust and produces the expected results
with nice differences according to the convergence factor, but the other
outputs are different and I don't know yet if this is ok and desired or not.
The basins and half-basins are different, less so for coarser
resolution, rather disastrous with MFD for the 1m LIDAR DEM in the North
Carolina dataset. It seems that such a DEM should be processed with
coarser resolution to obtain basins and half-basins that make sense,
apparently both for SFD and MFD, but these a just first impressions.
Slope length (LS for USLE) and slope steepness (S for USLE) are about
99.9% identical between SFD and MFD with the DEM <elevation> in the
North Carolina dataset. Streams are again different, i.e. MFD streams
need to be thinned then they are very similar to SFD, but MFD streams
make sense.
Once I have added MFD to segmented mode too and updated the
documentation, the module is ready to be submitted and scrutinised by
others.

This is no easy feat to modify r.watershed, I want to make sure the old
behaviour is preserved when adding new functionality and I need some
more time before submitting. I don't like to submit code with bugs...

Markus Metz

6

Markus,

at this point, as you suggest, the best would be to work on this in grass7 - do you have SVN commit access?

Helena

On Dec 5, 2008, at 3:02 PM, Markus Metz wrote:

Dylan Beaudette wrote:

On Fri, Dec 5, 2008 at 1:55 AM, Markus Metz <markus_metz@gmx.de> wrote:

I took the request for MFD support in r.watershed by Helena and Dylan to
heart and implemented it, but still need a few more days to clean up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

Is there any way to
cleanly merge this with the existing r.watershed code in SVN so that
we can test it?

Also considering Michael's remarks, I suggets that I then not submit to
grass-addons, but to grass 7. Not to grass-6.4.x, because this code is
now more experimental and documentation still needs to be written.
The MFD algorithm seems to be robust and produces the expected results
with nice differences according to the convergence factor, but the other
outputs are different and I don't know yet if this is ok and desired or not.
The basins and half-basins are different, less so for coarser
resolution, rather disastrous with MFD for the 1m LIDAR DEM in the North
Carolina dataset. It seems that such a DEM should be processed with
coarser resolution to obtain basins and half-basins that make sense,
apparently both for SFD and MFD, but these a just first impressions.

You may be using a threshold that is too small.

Slope length (LS for USLE) and slope steepness (S for USLE) are about
99.9% identical between SFD and MFD with the DEM <elevation> in the
North Carolina dataset. Streams are again different, i.e. MFD streams
need to be thinned then they are very similar to SFD, but MFD streams
make sense.
Once I have added MFD to segmented mode too and updated the
documentation, the module is ready to be submitted and scrutinised by
others.

This is no easy feat to modify r.watershed, I want to make sure the old
behaviour is preserved when adding new functionality and I need some
more time before submitting. I don't like to submit code with bugs...

Markus Metz

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

7

I have SVN commit access for grass-addons, is this also valid for grass7?

BTW my nightmare is to screw up the svn history that's why I hesitate to
submit directly to grass7 (or grass64).

Helena Mitasova wrote:

Markus,

at this point, as you suggest, the best would be to work on this in
grass7 - do you have SVN commit access?

Helena

On Dec 5, 2008, at 3:02 PM, Markus Metz wrote:

Dylan Beaudette wrote:

On Fri, Dec 5, 2008 at 1:55 AM, Markus Metz <markus_metz@gmx.de> wrote:

I took the request for MFD support in r.watershed by Helena and
Dylan to
heart and implemented it, but still need a few more days to clean
up the
code, then I want to submit it as r.watershed2.mfd to grass-addons.

Is there any way to
cleanly merge this with the existing r.watershed code in SVN so that
we can test it?

Also considering Michael's remarks, I suggets that I then not submit to
grass-addons, but to grass 7. Not to grass-6.4.x, because this code is
now more experimental and documentation still needs to be written.
The MFD algorithm seems to be robust and produces the expected results
with nice differences according to the convergence factor, but the other
outputs are different and I don't know yet if this is ok and desired
or not.
The basins and half-basins are different, less so for coarser
resolution, rather disastrous with MFD for the 1m LIDAR DEM in the North
Carolina dataset. It seems that such a DEM should be processed with
coarser resolution to obtain basins and half-basins that make sense,
apparently both for SFD and MFD, but these a just first impressions.

You may be using a threshold that is too small.

Slope length (LS for USLE) and slope steepness (S for USLE) are about
99.9% identical between SFD and MFD with the DEM <elevation> in the
North Carolina dataset. Streams are again different, i.e. MFD streams
need to be thinned then they are very similar to SFD, but MFD streams
make sense.
Once I have added MFD to segmented mode too and updated the
documentation, the module is ready to be submitted and scrutinised by
others.

This is no easy feat to modify r.watershed, I want to make sure the old
behaviour is preserved when adding new functionality and I need some
more time before submitting. I don't like to submit code with bugs...

Markus Metz

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

8

On Friday 05 December 2008, Markus Metz wrote:

Dylan Beaudette wrote:
> On Fri, Dec 5, 2008 at 1:55 AM, Markus Metz <markus_metz@gmx.de> wrote:
>> I took the request for MFD support in r.watershed by Helena and Dylan to
>> heart and implemented it, but still need a few more days to clean up the
>> code, then I want to submit it as r.watershed2.mfd to grass-addons.
>
> Is there any way to
> cleanly merge this with the existing r.watershed code in SVN so that
> we can test it?

Also considering Michael's remarks, I suggets that I then not submit to
grass-addons, but to grass 7. Not to grass-6.4.x, because this code is
now more experimental and documentation still needs to be written.

Good points. Perhaps my anxiousness to give the new MFD functionality clouded
my better judgment :slight_smile: .

The MFD algorithm seems to be robust and produces the expected results
with nice differences according to the convergence factor, but the other
outputs are different and I don't know yet if this is ok and desired or
not. The basins and half-basins are different, less so for coarser
resolution, rather disastrous with MFD for the 1m LIDAR DEM in the North
Carolina dataset. It seems that such a DEM should be processed with
coarser resolution to obtain basins and half-basins that make sense,
apparently both for SFD and MFD, but these a just first impressions.
Slope length (LS for USLE) and slope steepness (S for USLE) are about
99.9% identical between SFD and MFD with the DEM <elevation> in the
North Carolina dataset. Streams are again different, i.e. MFD streams
need to be thinned then they are very similar to SFD, but MFD streams
make sense.
Once I have added MFD to segmented mode too and updated the
documentation, the module is ready to be submitted and scrutinised by
others.

Looking forward to it!

This is no easy feat to modify r.watershed, I want to make sure the old
behaviour is preserved when adding new functionality and I need some
more time before submitting. I don't like to submit code with bugs...

Markus Metz

Indeed. Please go through the development cycle as suggested by experienced
developers.

Cheers,

Dylan

--
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341

9

Markus Metz wrote:

BTW my nightmare is to screw up the svn history that's why I hesitate to
submit directly to grass7 (or grass64).

The only thing that can really screw up the history is to make a local
copy, then add it to the repository as if it were new code. This not
only loses all prior history, but even the fact that it is a copy.

If you want to create a fork or clone of an existing module, use
"svn copy", not e.g. "cp -r". Also the source should be repository URL
rather than a working copy directory. E.g.:

cd raster
svn copy https://svn.osgeo.org/grass/grass/trunk/raster/r.watershed r.watershed.mfd

This way, you can merge the changes since the fork back into the
original. Or if you decide to replace the original with the forked
version, the fork will have inherited the original's history when it
was created.

grass-addons is part of the same repository as grass, so it's possible
to copy, rename, merge, etc between grass-addons and grass.

--
Glynn Clements <glynn@gclements.plus.com>

10

MFD is now ready for testing. So far, MFD with different settings of the
convergence factor works fine for elev_lid972_1m and elevation in the
North Carolina dataset.
Please read the updated documentation before using it!

A note on elev_lid972_1m: in order to get usable results, I changed
vertical units to millimeter with
r.mapcalc "elev_lid972_1m.1mm = round(elev_lid972_1m * 1000)"
Then try convergence = 1, 5, and 10, that gives a nice illustration of
the effect of the convergence factor.

<elevation>: most areas show SFD-like flow because the DEM doesn't
really offer more directions. But compare to r.terraflow output.

Changes in MFD output compared to SFD apart from flow accumulation
flow direction: not changed
basins, half-basins and streams: changed, looks usable for
elev_lid972_1m, but not for elevation
slope length (LS factor) and slope steepness (S factor): 99% identical,
differences not scatted but clumped, reason yet unknown, seems usable

I can not commit to grass7 (no write access or too stupid), therefore I
want to ask you developers if one of you could get the mfd module
including .svn entries developed against grass7 from here
http://markus.metz.giswork.googlepages.com/r.watershed.mfd.tar.gz
and commit the changes to grass7. The code has been indented with
grass_indent.sh
Single flow direction method is unaltered.
There is one global change in r.watershed, flow accumulation is now
DCELL, no longer CELL.

Thanks,

Markus

11

Markus Metz wrote:

I can not commit to grass7 (no write access or too stupid), therefore I
want to ask you developers if one of you could get the mfd module
including .svn entries developed against grass7 from here
http://markus.metz.giswork.googlepages.com/r.watershed.mfd.tar.gz
and commit the changes to grass7. The code has been indented with
grass_indent.sh
Single flow direction method is unaltered.
There is one global change in r.watershed, flow accumulation is now
DCELL, no longer CELL.

I've committed this as r34789, and fixed a minor bug (missed
opt16/opt17 renumbering) in r34790.

I note a couple of compile warnings:

make[1]: Entering directory `/usr/local/src/grass/svn/raster/r.watershed/seg'

close_maps.c: In function `close_maps':
close_maps.c:103: warning: passing arg 1 of `cseg_close' from incompatible pointer type
close_maps.c:8: warning: unused variable `value'

no_stream.c: In function `no_stream':
no_stream.c:89: warning: passing arg 2 of `cseg_put' from incompatible pointer type

The first one is due to "wat" having been changed from CSEG to DSEG;
presumably it also needs a change from cseg_close to dseg_close?

The second is due to "value" having been changed from CELL to DCELL,
but still being stored in "haf" which is CSEG. The value being stored
is actually an integer; it appears that "value" was simply being used
as a temporary variable, so I presume that it would suffice to do:

- value = basin_num - 1;
- cseg_put(&haf, &value, row, col);
+ CELL tmp = basin_num - 1;
+ cseg_put(&haf, &tmp, row, col);

However, I don't want to make these changes without checking.

--
Glynn Clements <glynn@gclements.plus.com>

12

Markus,

On Fri, Dec 5, 2008 at 10:05 PM, Markus Metz <markus_metz@gmx.de> wrote:

I have SVN commit access for grass-addons, is this also valid for grass7?

No, that's not valid, see:
http://trac.osgeo.org/grass/wiki/HowToContribute#WriteaccesstotheMainGRASS-SVNrepository

Please consider to ask for GRASS main SVN write access :slight_smile:

markusN