[GRASS-dev] r.watershed and swap memory

GRASS GRASS developer
1

Dear Grass Users and Developers, sorry for cross posting but we hope the
argument can be of interest for all and we hope someone can give us a
solution to this problem..

We have this kind of problem:
* a large DEM (150000000 cells)
* an ordinary computer (2 GB ram)
* we must obtain the drainage map using r.watershed (without changing
resolution), because then we need to be able to calculate the upstream
basin for each cell (r.water.outlet).
* we have tried r.watershed straigth (but after few seconds a memory
allocation problem crashed the program)
* we have tried the "-m" option but after 4 days of work it is still at
0%.
* giving up the last option, because it takes too long, we have
monitored the ram usage by means of "free -m" and we have seen that
r.watershed rapidly saturate the ram and then, after a little usage of
swap (20 mb) crashes.. so it seems that r.watershed doesn't use swap
memory... (and is then unuseful, as we did, to increase the swap
memory)
* we have tried to modify
"Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
without success... the error is still the same

so at this point:
* is adding ram to the computer the only solution?
* if yes, how can we estimate the ram to buy!
* can we obtain some better results compiling grass with option
--enable-largefile
* does someone solved a similar problem in some other way?

Thank you for your suggestions!!!
we hope really in your help

Luca & Ivan

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

2

Ivan - you may talk to Yann on this list before you buy more memory - he is trying to do the same as you , but with a bigger DEM and 8GB of memory (Yann I hope it is OK with you that I am revealing this here).
My experience with large DEMs (up to 10,000x10,000) has been that I had to split the area into sections that were about
2000x2000 on 1GB memory computer) to get it done (I was able to do that for Panama because of its shape - many small watersheds rather than a single big one) - it took me several days to do that.
Then I ran r.terraflow and I got it in 3 hours.

Yann says that r.terraflow did not work for him - now I remember what the problem was when I tried to run it recently -
it needs a LOT of hard drive space which is not a problem these days, BUT the default has been changed
to /tmp which for my linux box is only 2GB or so. But when running r.terraflow you can define where
you want the temporary files to be written - so give it something with a lot of space (tens of gigabytes at least)
and it should run. I think that the default should be changed to where it was - I think it is the regular grass tmp where people usually have a lot of space for the data.

If even that does not work you can give a try to brand new TerraSTREAM - see the link below
(and let me know whether it works for you),

Helena

  TerraSTREAM provides a series of components that
perform flow modeling and terrain analysis tasks on very large digital
elevation models and works equally well on TIN and grid DEMs. The algorithms
used in the libraries have provable efficient performance in the worst case,
even on very large terrains that do not fit in the main memory of the computer.
TerraSTREAM 0.2 comes with direct GRASS and ArcGIS support as well as a simple
standalone graphical user interface and powerful command line tools that can be
used alone or integrated into most GIS environments by scripting. For more
information about this release and for contact information, visit
http://madalgo.au.dk/Trac-TerraSTREAM/.
The TerraSTREAM 0.2 users guide is available here:
http://madalgo.au.dk/Trac-TerraSTREAM/wiki/UsersGuide .

On Mar 18, 2008, at 5:32 AM, ivan marchesini wrote:

Dear Grass Users and Developers, sorry for cross posting but we hope the
argument can be of interest for all and we hope someone can give us a
solution to this problem..

We have this kind of problem:
* a large DEM (150000000 cells)
* an ordinary computer (2 GB ram)
* we must obtain the drainage map using r.watershed (without changing
resolution), because then we need to be able to calculate the upstream
basin for each cell (r.water.outlet).
* we have tried r.watershed straigth (but after few seconds a memory
allocation problem crashed the program)
* we have tried the "-m" option but after 4 days of work it is still at
0%.
* giving up the last option, because it takes too long, we have
monitored the ram usage by means of "free -m" and we have seen that
r.watershed rapidly saturate the ram and then, after a little usage of
swap (20 mb) crashes.. so it seems that r.watershed doesn't use swap
memory... (and is then unuseful, as we did, to increase the swap
memory)
* we have tried to modify
"Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
without success... the error is still the same

so at this point:
* is adding ram to the computer the only solution?
* if yes, how can we estimate the ram to buy!
* can we obtain some better results compiling grass with option
--enable-largefile
* does someone solved a similar problem in some other way?

Thank you for your suggestions!!!
we hope really in your help

Luca & Ivan

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

Dear Helena,
Thank you very much for your answer...
my problems are:

* is the flowdirection output produced by r.terraflow suitable for basin
creation (like I can do with the drainage map created by r.watershed and
given as input to r.water.outlet)? Because this is my final target...
It seems to me that r.terraflow outputs aren't useful for basin
delineation... or I'm wrong?

* I'm really interested in testing TerraSTREAM but, probably due to my
fault, I wasn't able to obtain a login and password...

many many thanks

Ivan

Il giorno mar, 18/03/2008 alle 09.33 -0400, Helena Mitasova ha scritto:

Ivan - you may talk to Yann on this list before you buy more memory -
he is trying to do the same as you , but with a bigger DEM and 8GB of
memory (Yann I hope it is OK with you that I am revealing this here).
My experience with large DEMs (up to 10,000x10,000) has been that I
had to split the area into sections that were about
2000x2000 on 1GB memory computer) to get it done (I was able to do
that for Panama because of its shape - many small watersheds rather
than a single big one) - it took me several days to do that.
Then I ran r.terraflow and I got it in 3 hours.

Yann says that r.terraflow did not work for him - now I remember what
the problem was when I tried to run it recently -
it needs a LOT of hard drive space which is not a problem these days,
BUT the default has been changed
to /tmp which for my linux box is only 2GB or so. But when running
r.terraflow you can define where
you want the temporary files to be written - so give it something
with a lot of space (tens of gigabytes at least)
and it should run. I think that the default should be changed to
where it was - I think it is the regular grass tmp where people
usually have a lot of space for the data.

If even that does not work you can give a try to brand new
TerraSTREAM - see the link below
(and let me know whether it works for you),

Helena

  TerraSTREAM provides a series of components that
perform flow modeling and terrain analysis tasks on very large digital
elevation models and works equally well on TIN and grid DEMs. The
algorithms
used in the libraries have provable efficient performance in the
worst case,
even on very large terrains that do not fit in the main memory of the
computer.
TerraSTREAM 0.2 comes with direct GRASS and ArcGIS support as well as
a simple
standalone graphical user interface and powerful command line tools
that can be
used alone or integrated into most GIS environments by scripting. For
more
information about this release and for contact information, visit
http://madalgo.au.dk/Trac-TerraSTREAM/.
The TerraSTREAM 0.2 users guide is available here:
http://madalgo.au.dk/Trac-TerraSTREAM/wiki/UsersGuide .

On Mar 18, 2008, at 5:32 AM, ivan marchesini wrote:

> Dear Grass Users and Developers, sorry for cross posting but we
> hope the
> argument can be of interest for all and we hope someone can give us a
> solution to this problem..
>
> We have this kind of problem:
> * a large DEM (150000000 cells)
> * an ordinary computer (2 GB ram)
> * we must obtain the drainage map using r.watershed (without changing
> resolution), because then we need to be able to calculate the upstream
> basin for each cell (r.water.outlet).
> * we have tried r.watershed straigth (but after few seconds a memory
> allocation problem crashed the program)
> * we have tried the "-m" option but after 4 days of work it is
> still at
> 0%.
> * giving up the last option, because it takes too long, we have
> monitored the ram usage by means of "free -m" and we have seen that
> r.watershed rapidly saturate the ram and then, after a little usage of
> swap (20 mb) crashes.. so it seems that r.watershed doesn't use swap
> memory... (and is then unuseful, as we did, to increase the swap
> memory)
> * we have tried to modify
> "Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
> without success... the error is still the same
>
>
> so at this point:
> * is adding ram to the computer the only solution?
> * if yes, how can we estimate the ram to buy!
> * can we obtain some better results compiling grass with option
> --enable-largefile
> * does someone solved a similar problem in some other way?
>
> Thank you for your suggestions!!!
> we hope really in your help
>
> Luca & Ivan
>
>
>
>
>
>
>
>
>
>
>
> --
> Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
> Preferisco formati liberi.
> Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
> I prefer free formats.
> http://it.wikipedia.org/wiki/Formato_aperto
> http://en.wikipedia.org/wiki/Open_format
>
> Ivan Marchesini
> Department of Civil and Environmental Engineering
> University of Perugia
> Via G. Duranti 93/a
> 06125
> Perugia (Italy)
> Socio fondatore GFOSS "Geospatial Free and Open Source Software"
> http://www.gfoss.it
> e-mail: marchesini@unipg.it
> ivan.marchesini@gmail.com
> tel: +39(0)755853760
> fax (university): +39(0)755853756
> fax (home): +39(0)5782830887
> jabber: geoivan73@jabber.org
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

4

Hi Ivan and Helena,

149,040,000 Cells for the srtm 90m of Mekong river.
computer is on 8Gb RAM.
About 65 % done after 43.x hours, r.watershed basically does it, it seems.
Of course efficiency is a problem here, but it does it.

yes r.terraflow goes to /tmp, that is considered a bug to me, not an
enhancement...
The location from which we have the GRASS dataset is generally the
best guess for large empty Disk space.

Cheers,
Yann

On 19/03/2008, ivan marchesini <marchesini@unipg.it> wrote:

Dear Helena,
Thank you very much for your answer...
my problems are:

* is the flowdirection output produced by r.terraflow suitable for basin
creation (like I can do with the drainage map created by r.watershed and
given as input to r.water.outlet)? Because this is my final target...
It seems to me that r.terraflow outputs aren't useful for basin
delineation... or I'm wrong?

* I'm really interested in testing TerraSTREAM but, probably due to my
fault, I wasn't able to obtain a login and password...

many many thanks

Ivan

Il giorno mar, 18/03/2008 alle 09.33 -0400, Helena Mitasova ha scritto:

> Ivan - you may talk to Yann on this list before you buy more memory -
> he is trying to do the same as you , but with a bigger DEM and 8GB of
> memory (Yann I hope it is OK with you that I am revealing this here).
> My experience with large DEMs (up to 10,000x10,000) has been that I
> had to split the area into sections that were about
> 2000x2000 on 1GB memory computer) to get it done (I was able to do
> that for Panama because of its shape - many small watersheds rather
> than a single big one) - it took me several days to do that.
> Then I ran r.terraflow and I got it in 3 hours.
>
> Yann says that r.terraflow did not work for him - now I remember what
> the problem was when I tried to run it recently -
> it needs a LOT of hard drive space which is not a problem these days,
> BUT the default has been changed
> to /tmp which for my linux box is only 2GB or so. But when running
> r.terraflow you can define where
> you want the temporary files to be written - so give it something
> with a lot of space (tens of gigabytes at least)
> and it should run. I think that the default should be changed to
> where it was - I think it is the regular grass tmp where people
> usually have a lot of space for the data.
>
> If even that does not work you can give a try to brand new
> TerraSTREAM - see the link below
> (and let me know whether it works for you),
>
> Helena
>
> TerraSTREAM provides a series of components that
> perform flow modeling and terrain analysis tasks on very large digital
> elevation models and works equally well on TIN and grid DEMs. The
> algorithms
> used in the libraries have provable efficient performance in the
> worst case,
> even on very large terrains that do not fit in the main memory of the
> computer.
> TerraSTREAM 0.2 comes with direct GRASS and ArcGIS support as well as
> a simple
> standalone graphical user interface and powerful command line tools
> that can be
> used alone or integrated into most GIS environments by scripting. For
> more
> information about this release and for contact information, visit
> http://madalgo.au.dk/Trac-TerraSTREAM/.
> The TerraSTREAM 0.2 users guide is available here:
> http://madalgo.au.dk/Trac-TerraSTREAM/wiki/UsersGuide .
>
>
>
> On Mar 18, 2008, at 5:32 AM, ivan marchesini wrote:
>
> > Dear Grass Users and Developers, sorry for cross posting but we
> > hope the
> > argument can be of interest for all and we hope someone can give us a
> > solution to this problem..
> >
> > We have this kind of problem:
> > * a large DEM (150000000 cells)
> > * an ordinary computer (2 GB ram)
> > * we must obtain the drainage map using r.watershed (without changing
> > resolution), because then we need to be able to calculate the upstream
> > basin for each cell (r.water.outlet).
> > * we have tried r.watershed straigth (but after few seconds a memory
> > allocation problem crashed the program)
> > * we have tried the "-m" option but after 4 days of work it is
> > still at
> > 0%.
> > * giving up the last option, because it takes too long, we have
> > monitored the ram usage by means of "free -m" and we have seen that
> > r.watershed rapidly saturate the ram and then, after a little usage of
> > swap (20 mb) crashes.. so it seems that r.watershed doesn't use swap
> > memory... (and is then unuseful, as we did, to increase the swap
> > memory)
> > * we have tried to modify
> > "Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
> > without success... the error is still the same
> >
> >
> > so at this point:
> > * is adding ram to the computer the only solution?
> > * if yes, how can we estimate the ram to buy!
> > * can we obtain some better results compiling grass with option
> > --enable-largefile
> > * does someone solved a similar problem in some other way?
> >
> > Thank you for your suggestions!!!
> > we hope really in your help
> >
> > Luca & Ivan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
> > Preferisco formati liberi.
> > Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
> > I prefer free formats.
> > http://it.wikipedia.org/wiki/Formato_aperto
> > http://en.wikipedia.org/wiki/Open_format
> >
> > Ivan Marchesini
> > Department of Civil and Environmental Engineering
> > University of Perugia
> > Via G. Duranti 93/a
> > 06125
> > Perugia (Italy)
> > Socio fondatore GFOSS "Geospatial Free and Open Source Software"
> > http://www.gfoss.it
> > e-mail: marchesini@unipg.it
> > ivan.marchesini@gmail.com
> > tel: +39(0)755853760
> > fax (university): +39(0)755853756
> > fax (home): +39(0)5782830887
> > jabber: geoivan73@jabber.org
> >
> > _______________________________________________
> > grass-dev mailing list
> > grass-dev@lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-dev
>
>
--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

5

Ivan wrote:

We have this kind of problem:
* a large DEM (150000000 cells)
* an ordinary computer (2 GB ram)
* we must obtain the drainage map using r.watershed (without
changing resolution), because then we need to be able to calculate
the upstream basin for each cell (r.water.outlet).
* we have tried r.watershed straigth (but after few seconds a
memory allocation problem crashed the program)

I am pretty sure that in the past I have run r.watershed for a
14000x12000 region on a machine with 2GB RAM (+2 swap) (I'd have to dig
out a backup cd to check exactly; I think that was without the -m flag).
It had to run that over a weekend but it did finish. Perhaps you are just
on a threshold of being too large? Can you crop the region slightly or
run with a slightly coarser resolution to see where the memory limit is?

Hamish

      ____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

6

Yann Chemin wrote:

149,040,000 Cells for the srtm 90m of Mekong river.
computer is on 8Gb RAM.
About 65 % done after 43.x hours, r.watershed basically does it, it
seems.
Of course efficiency is a problem here, but it does it.

how much RAM is the process using? (check with `top`, VIRT RES SHR)

yes r.terraflow goes to /tmp, that is considered a bug to me, not an
enhancement...
The location from which we have the GRASS dataset is generally the
best guess for large empty Disk space.

note these r.terraflow options:
        memory Main memory size (in MB)
                 default: 300
    STREAM_DIR Location of intermediate STREAMs
                 default: /var/tmp

by making memory= larger the required tmp file space should be smaller.
you can set STREAM_DIR= somewhere else if you like.

Hamish

      ____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

7

Hi Hamish,

my mistake, forgot to include that htop one:

VIRT 6323Mb
RES 5232Mb

noted the terraflow options, i am just waiting for this r.watershed
experiment to conclude...

Yann

On 19/03/2008, Hamish <hamish_b@yahoo.com> wrote:

Yann Chemin wrote:
> 149,040,000 Cells for the srtm 90m of Mekong river.
> computer is on 8Gb RAM.
> About 65 % done after 43.x hours, r.watershed basically does it, it
> seems.
> Of course efficiency is a problem here, but it does it.

how much RAM is the process using? (check with `top`, VIRT RES SHR)

> yes r.terraflow goes to /tmp, that is considered a bug to me, not an
> enhancement...
> The location from which we have the GRASS dataset is generally the
> best guess for large empty Disk space.

note these r.terraflow options:
        memory Main memory size (in MB)
                 default: 300
    STREAM_DIR Location of intermediate STREAMs
                 default: /var/tmp

by making memory= larger the required tmp file space should be smaller.
you can set STREAM_DIR= somewhere else if you like.

Hamish

      ____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

8

Hamish wrote:

> yes r.terraflow goes to /tmp, that is considered a bug to me, not an
> enhancement...
> The location from which we have the GRASS dataset is generally the
> best guess for large empty Disk space.

note these r.terraflow options:

    STREAM_DIR Location of intermediate STREAMs
                 default: /var/tmp

This should be in lower case. AFAICT, we never use upper case in
option names.

--
Glynn Clements <glynn@gclements.plus.com>

9

> note these r.terraflow options:

> STREAM_DIR Location of intermediate STREAMs
> default: /var/tmp

Glynn:

This should be in lower case. AFAICT, we never use upper case in
option names.

another task for GRASS 7....

H

      ____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

10

Hi Hamish...
yes.. my threshold is really small (30 cells, because I need a very
detailed stream network in output), and this can probably have a strong
effect on the time of analysis...
However I'm really surprised of your results with 170000000 of cells...
and 2 Gb of ram...!!
really impressive...
I have tried over two different computers (with 2 Gb of ram), but the
result is the same... so I think that the problem is not on the quality
of the ram or other similar thinks but on the dimension of the
dataset...

I will try with different resolution as soon as possible... but at the
end I need this resolution because I have "cleaned" the dem with
v.breach to obtain a hydrologically correct dem and now I must use this
dem to derive the drainage...

Many thanks for your interest....

Ivan

Il giorno mar, 18/03/2008 alle 18.05 -0700, Hamish ha scritto:

e region slightly or
run with a slightly coarser resolution to see where the memory limit
is?

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

11

Ivan - instead of using the small threshold that is killing it, if you need just the stream network use arbitrarily large threshold and derive your stream network from the accum file like this

r.mapcalc "mystreams=if(abs(myaccum)>30,1,null())"
or you can use abs(myaccum) instead of 1 if you need contributing area for each stream cell.

(isn't this in the man page? I have been doing it that way for years)

Helena

On Mar 19, 2008, at 10:43 AM, ivan marchesini wrote:

Hi Hamish...
yes.. my threshold is really small (30 cells, because I need a very
detailed stream network in output), and this can probably have a strong
effect on the time of analysis...
However I'm really surprised of your results with 170000000 of cells...
and 2 Gb of ram...!!
really impressive...
I have tried over two different computers (with 2 Gb of ram), but the
result is the same... so I think that the problem is not on the quality
of the ram or other similar thinks but on the dimension of the
dataset...

I will try with different resolution as soon as possible... but at the
end I need this resolution because I have "cleaned" the dem with
v.breach to obtain a hydrologically correct dem and now I must use this
dem to derive the drainage...

Many thanks for your interest....

Ivan

Il giorno mar, 18/03/2008 alle 18.05 -0700, Hamish ha scritto:

e region slightly or
run with a slightly coarser resolution to see where the memory limit
is?

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

12

On Wed, Mar 19, 2008 at 3:54 PM, Helena Mitasova <hmitaso@unity.ncsu.edu> wrote:

Ivan - instead of using the small threshold that is killing it, if
you need just the stream network use arbitrarily large threshold and
derive your stream network from the accum file like this

r.mapcalc "mystreams=if(abs(myaccum)>30,1,null())"
or you can use abs(myaccum) instead of 1 if you need contributing
area for each stream cell.

(isn't this in the man page? I have been doing it that way for years)

Helena,

apparently it isn't. Would you mind to add it to
raster/r.terraflow/description.html
?
thanks
Markus

13

On Mar 19, 2008, at 10:58 AM, Markus Neteler wrote:

On Wed, Mar 19, 2008 at 3:54 PM, Helena Mitasova <hmitaso@unity.ncsu.edu> wrote:

Ivan - instead of using the small threshold that is killing it, if
you need just the stream network use arbitrarily large threshold and
derive your stream network from the accum file like this

r.mapcalc "mystreams=if(abs(myaccum)>30,1,null())"
or you can use abs(myaccum) instead of 1 if you need contributing
area for each stream cell.

(isn't this in the man page? I have been doing it that way for years)

Helena,

apparently it isn't. Would you mind to add it to
raster/r.terraflow/description.html
?

that is for r.watershed Helena

thanks
Markus

14

just looking for a code that's able to process a 50'000x60'000 grid.
streamflow sounds verz promising, but as it's only availible as a binary
it's not really usable within linux (as you would need exactly the same
specs..)

r.watershed is rather slow and terraflow isn't an option because of the
inscalability of the basin size.

has anybody other options or experiences?

hamish_b wrote:

> note these r.terraflow options:

> STREAM_DIR Location of intermediate STREAMs
> default: /var/tmp

Glynn:

This should be in lower case. AFAICT, we never use upper case in
option names.

another task for GRASS 7....

H

____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

--
View this message in context: http://www.nabble.com/r.watershed-and-swap-memory-tp16116939p16154475.html
Sent from the Grass - Dev mailing list archive at Nabble.com.

15

hmmm
thanks to Helena for the suggestion of use accumulation map and then
r.mapcalc (I knew this method but I didn't remember it :frowning: )
the problem is that, also using this method, the ram wasn't enougth..

at the moment we have solved using another PC (13 GB of RAM)... and the
process is still running... after 3 days...
but if I will find alternatives I will let you know...

many thanks to all...

Il giorno mer, 19/03/2008 alle 15.29 -0700, honegger ha scritto:

just looking for a code that's able to process a 50'000x60'000 grid.
streamflow sounds verz promising, but as it's only availible as a binary
it's not really usable within linux (as you would need exactly the same
specs..)

r.watershed is rather slow and terraflow isn't an option because of the
inscalability of the basin size.

has anybody other options or experiences?

hamish_b wrote:
>
>> > note these r.terraflow options:
>>
>> > STREAM_DIR Location of intermediate STREAMs
>> > default: /var/tmp
>
> Glynn:
>> This should be in lower case. AFAICT, we never use upper case in
>> option names.
>
> another task for GRASS 7....
>
>
> H
>
>
>
>
> ____________________________________________________________________________________
> Never miss a thing. Make Yahoo your home page.
> http://www.yahoo.com/r/hs
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>
>

--
Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
Preferisco formati liberi.
Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
I prefer free formats.
http://it.wikipedia.org/wiki/Formato_aperto
http://en.wikipedia.org/wiki/Open_format

Ivan Marchesini
Department of Civil and Environmental Engineering
University of Perugia
Via G. Duranti 93/a
06125
Perugia (Italy)
Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
e-mail: marchesini@unipg.it
        ivan.marchesini@gmail.com
tel: +39(0)755853760
fax (university): +39(0)755853756
fax (home): +39(0)5782830887
jabber: geoivan73@jabber.org

16

Hello everybody.
While Ivan is working on r.watershed I am trying to reach the result using TerraSTREAM.

With the help of Thomas, I have successfully created a drainage map from the DEM and the idea is to use it
with r.water.outlet. The problem is that it needs a drainage map with values that run from -8 to 8, instead TerraSTREAM prompt out a bitmap (using SFD) or a raster with values from 0 to 360 degree (with MFD). I can reclass it with r.reclass
but I don’t know how handle negative values.

Any suggestion on this?

Thanks

Luca

17

Luca,

The Terrastream direction coding for single flow direction is as
follows

1 = Water flows east
2 = SE
4 = S
8 = SW
16 = W
32 = NW
64 = N
128 = NE

Or pictorially

32 64 128
16 x 1
  8 4 2

In the MFD case, values do NOT represent aspects in the range 0 to 360,
but instead indicate sums of the directions above. Thus if water flows
N, S and SW, the value is 64+8+4 = 76.

In both the SFD/MFD outputs, a value of 0 means this point is local
minimum and there is no flow direction.

r.water.outlet reports:
drainage=name
        Input map: drainage direction. Indicates the "aspect" for each
        cell. Multiplying positive values by 45 will give the direction
        in degrees that the surface runoff will travel from that cell.
        The value -1 indicates that the cell is a depression area. Other
        negative values indicate that surface runoff is leaving the
        boundaries of the current geographic region. The absolute value
        of these negative cells indicates the direction of flow. This
        map is generated from r.watershed
        
I don't know if the direction it refers to is measured from North or
East or if it moves clockwise or counterclockwise.

r.slope.aspect measures ccw from east, TerraStream and TerraFlow are
measuring clockwise from East, and standard compass direction is
measured clockwise from North.

If someone is more familiar with the r.water.outlet drainage directions,
it should be relatively easy to map the values 1,2,4,8,16,32,64 and 128
to 1 through 7. log_2(tflow_dir)+1 will map terrastream flow directions
onto the range 1-7 (handling 0 is special case).

Is there any consistent way of encoding flow direction? TerraStream and
TerraFlow I believe used an encoding from Jenson and Domingue, though in
looking up their reference they started with 1 in the NE and moved
clockwise. Perhaps ESRI products used 1=E. It seems that aspect should
be measured clockwise in degrees from north geographically, but it makes
more sense to measure ccw from east trigonometrically.

TerraStream could easily rotate or reverse the values 1-255 if there is
a standard out there to follow.

-Andy

On Thu, 2008-03-27 at 00:49 +0100, Luca Casagrande wrote:

Hello everybody.
While Ivan is working on r.watershed I am trying to reach the result
using TerraSTREAM.

With the help of Thomas, I have successfully created a drainage map
from the DEM and the idea is to use it
with r.water.outlet. The problem is that it needs a drainage map with
values that run from -8 to 8, instead TerraSTREAM prompt out a bitmap
(using SFD) or a raster with values from 0 to 360 degree (with MFD). I
can reclass it with r.reclass
but I don't know how handle negative values.

Any suggestion on this?

Thanks

Luca

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

18

Hi,

I don't know if they are useful, but I put some Matlab scripts here:

http://trac.osgeo.org/grass/browser/grass-addons/raster/r.terraflow.tools

See d.rast.arrow,
        type Type of existing raster aspect map
               options: grass,compass,agnps,answers
               default: grass

GRASS's usual aspect maps measure theta CCW from East; "Compass" is
traditional navigation CW from North. I've not sure about ANSWERS/AGNPS,
ISTR that they do the 0-8 way every 45 deg; but see the code for a real
answer...

no idea what the SFD or MDF standards "should be", but those are some
already in use in GRASS, and if possible we should aim for consistency
across the GIS.

Hamish

      ____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping

19

I made some test using r.water.outlet with Spearfish location and
output coming from GRASS and from TerraSTREAM:

http://www.movedigital.com/go/doktoreas/114926/grass_terrastream.jpg

The drainage map created with TerraSTREAM has been reclassified using those
rules:

0=-1
128=1
1=2
2=3
4=4
8=5
16=6
32=7
64=0

As you can notice, the result are quite different and the one from TS generate
a really small area.

Any idea on that?

Thanks for your help
Luca

On Thursday 27 March 2008 01:31:07 Andrew Danner wrote:

Luca,

The Terrastream direction coding for single flow direction is as
follows

1 = Water flows east
2 = SE
4 = S
8 = SW
16 = W
32 = NW
64 = N
128 = NE

Or pictorially

32 64 128
16 x 1
  8 4 2

In the MFD case, values do NOT represent aspects in the range 0 to 360,
but instead indicate sums of the directions above. Thus if water flows
N, S and SW, the value is 64+8+4 = 76.

In both the SFD/MFD outputs, a value of 0 means this point is local
minimum and there is no flow direction.

r.water.outlet reports:
drainage=name
        Input map: drainage direction. Indicates the "aspect" for each
        cell. Multiplying positive values by 45 will give the direction
        in degrees that the surface runoff will travel from that cell.
        The value -1 indicates that the cell is a depression area. Other
        negative values indicate that surface runoff is leaving the
        boundaries of the current geographic region. The absolute value
        of these negative cells indicates the direction of flow. This
        map is generated from r.watershed

I don't know if the direction it refers to is measured from North or
East or if it moves clockwise or counterclockwise.

r.slope.aspect measures ccw from east, TerraStream and TerraFlow are
measuring clockwise from East, and standard compass direction is
measured clockwise from North.

If someone is more familiar with the r.water.outlet drainage directions,
it should be relatively easy to map the values 1,2,4,8,16,32,64 and 128
to 1 through 7. log_2(tflow_dir)+1 will map terrastream flow directions
onto the range 1-7 (handling 0 is special case).

Is there any consistent way of encoding flow direction? TerraStream and
TerraFlow I believe used an encoding from Jenson and Domingue, though in
looking up their reference they started with 1 in the NE and moved
clockwise. Perhaps ESRI products used 1=E. It seems that aspect should
be measured clockwise in degrees from north geographically, but it makes
more sense to measure ccw from east trigonometrically.

TerraStream could easily rotate or reverse the values 1-255 if there is
a standard out there to follow.

-Andy

20

Luca,

Are you using the same easting/northing in r.water.outlet for both the
results generated by r.watershed and terrastream? As the images show,
the flow lines are bit different in the two images and you might have to
shift the basin outlet coordinates a bit. Also, are you sure the mapping
of drainage values is correct?

r.water.outlet/basin.h lists two arrays for drainage

#define DRAINVAR = {{ 7,6,5 },{ 8,-17,4 },{ 1,2,3 }}
#define UPDRAINVAR = {{ 3,2,1 },{ 4,-17,8 },{ 5,6,7 }}

which seems to imply the mapping is either

7 6 5
8 x 4
1 2 3

or

3 2 1
4 x 8
5 6 7

neither of which seem to correspond to your mapping

of

7 0 1
6 x 2
5 4 3

I don't think 0 is a valid flow direction for r.water.outlet. running
r.stats on a flow direction map generated by r.watershed shows no 0
values. Some quick checking of the flow directions from r.watershed,
seems to suggest the second mapping above is correct, so try

1 = 8
2 = 7
4 = 6
8 = 5
16 = 4
32 = 3
64 = 2
128 = 1

-Andy

On Thu, 2008-03-27 at 13:40 +0100, Luca Casagrande wrote:

I made some test using r.water.outlet with Spearfish location and
output coming from GRASS and from TerraSTREAM:

http://www.movedigital.com/go/doktoreas/114926/grass_terrastream.jpg

The drainage map created with TerraSTREAM has been reclassified using those
rules:

0=-1
128=1
1=2
2=3
4=4
8=5
16=6
32=7
64=0

As you can notice, the result are quite different and the one from TS generate
a really small area.

Any idea on that?

Thanks for your help
Luca

On Thursday 27 March 2008 01:31:07 Andrew Danner wrote:
> Luca,
>
> The Terrastream direction coding for single flow direction is as
> follows
>
> 1 = Water flows east
> 2 = SE
> 4 = S
> 8 = SW
> 16 = W
> 32 = NW
> 64 = N
> 128 = NE
>
> Or pictorially
>
> 32 64 128
> 16 x 1
> 8 4 2
>
> In the MFD case, values do NOT represent aspects in the range 0 to 360,
> but instead indicate sums of the directions above. Thus if water flows
> N, S and SW, the value is 64+8+4 = 76.
>
> In both the SFD/MFD outputs, a value of 0 means this point is local
> minimum and there is no flow direction.
>
> r.water.outlet reports:
> drainage=name
> Input map: drainage direction. Indicates the "aspect" for each
> cell. Multiplying positive values by 45 will give the direction
> in degrees that the surface runoff will travel from that cell.
> The value -1 indicates that the cell is a depression area. Other
> negative values indicate that surface runoff is leaving the
> boundaries of the current geographic region. The absolute value
> of these negative cells indicates the direction of flow. This
> map is generated from r.watershed
>
> I don't know if the direction it refers to is measured from North or
> East or if it moves clockwise or counterclockwise.
>
> r.slope.aspect measures ccw from east, TerraStream and TerraFlow are
> measuring clockwise from East, and standard compass direction is
> measured clockwise from North.
>
> If someone is more familiar with the r.water.outlet drainage directions,
> it should be relatively easy to map the values 1,2,4,8,16,32,64 and 128
> to 1 through 7. log_2(tflow_dir)+1 will map terrastream flow directions
> onto the range 1-7 (handling 0 is special case).
>
> Is there any consistent way of encoding flow direction? TerraStream and
> TerraFlow I believe used an encoding from Jenson and Domingue, though in
> looking up their reference they started with 1 in the NE and moved
> clockwise. Perhaps ESRI products used 1=E. It seems that aspect should
> be measured clockwise in degrees from north geographically, but it makes
> more sense to measure ccw from east trigonometrically.
>
> TerraStream could easily rotate or reverse the values 1-255 if there is
> a standard out there to follow.
>
>
> -Andy