[GRASSLIST:7445] r.terraflow failured

hello

At first time I tried to run r.terraflow in a small test area. I failured. It gave this error:

r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir swatershed=el5-watershed accumulation=el5-accu tci=el5-tci

assigning directions on plateaus
File size limit exceeded

Could you tell me why ?

regards

Ahmet Temiz

______________________________________
XamimeLT - installed on mailserver for domain @deprem.gov.tr
Queries to: postmaster@deprem.gov.tr
______________________________________
The views and opinions expressed in this e-mail message are the sender's own
and do not necessarily represent the views and the opinions of Earthquake Research Dept.
of General Directorate of Disaster Affairs.

Bu e-postadaki fikir ve gorusler gonderenin sahsina ait olup, yasal olarak T.C.
B.I.B. Afet Isleri Gn.Mud. Deprem Arastirma Dairesi'ni baglayici nitelikte degildir.

r.terraflow makes large temp files during the operation. Check the
STREAM_DIR (defaults to /var/tmp) you assigned I would bet that one of
those temp files exceeded 2 Gb. That seems to be the filesize limit on
my distribution of linux.

Davids

On 7/4/05, orkun <temiz@deprem.gov.tr> wrote:

hello

At first time I tried to run r.terraflow in a small test area. I
failured. It gave this error:

r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir
swatershed=el5-watershed accumulation=el5-accu tci=el5-tci

assigning directions on plateaus
File size limit exceeded

Could you tell me why ?

regards

Ahmet Temiz

______________________________________
XamimeLT - installed on mailserver for domain @deprem.gov.tr
Queries to: postmaster@deprem.gov.tr
______________________________________
The views and opinions expressed in this e-mail message are the sender's own
and do not necessarily represent the views and the opinions of Earthquake Research Dept.
of General Directorate of Disaster Affairs.

Bu e-postadaki fikir ve gorusler gonderenin sahsina ait olup, yasal olarak T.C.
B.I.B. Afet Isleri Gn.Mud. Deprem Arastirma Dairesi'ni baglayici nitelikte degildir.

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

David Finlayson wrote:

r.terraflow makes large temp files during the operation. Check the
STREAM_DIR (defaults to /var/tmp) you assigned I would bet that one of
those temp files exceeded 2 Gb. That seems to be the filesize limit on
my distribution of linux.

Davids

On 7/4/05, orkun <temiz@deprem.gov.tr> wrote:

hello

At first time I tried to run r.terraflow in a small test area. I
failured. It gave this error:

r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir
swatershed=el5-watershed accumulation=el5-accu tci=el5-tci

assigning directions on plateaus
File size limit exceeded

Could you tell me why ?

regards

Ahmet Temiz

______________________________________
XamimeLT - installed on mailserver for domain @deprem.gov.tr
Queries to: postmaster@deprem.gov.tr
______________________________________
The views and opinions expressed in this e-mail message are the sender's own
and do not necessarily represent the views and the opinions of Earthquake Research Dept.
of General Directorate of Disaster Affairs.

Bu e-postadaki fikir ve gorusler gonderenin sahsina ait olup, yasal olarak T.C.
B.I.B. Afet Isleri Gn.Mud. Deprem Arastirma Dairesi'ni baglayici nitelikte degildir.

thank you
You are right.
that file made my disk full.
Although I used a small region for testing purposes. It produced
giantic files.
I do not know what will happen if I use whole region ?

what do you suggest ?

regards

______________________________________
XamimeLT - installed on mailserver for domain @deprem.gov.tr
Queries to: postmaster@deprem.gov.tr
______________________________________
The views and opinions expressed in this e-mail message are the sender's own
and do not necessarily represent the views and the opinions of Earthquake Research Dept.
of General Directorate of Disaster Affairs.

Bu e-postadaki fikir ve gorusler gonderenin sahsina ait olup, yasal olarak T.C.
B.I.B. Afet Isleri Gn.Mud. Deprem Arastirma Dairesi'ni baglayici nitelikte degildir.

orkun wrote:

>>At first time I tried to run r.terraflow in a small test area. I
>>failured. It gave this error:
>>
>>r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir
>>swatershed=el5-watershed accumulation=el5-accu tci=el5-tci
>>
>>assigning directions on plateaus
>>File size limit exceeded
>>
>>
>>Could you tell me why ?

David Finlayson wrote:

>r.terraflow makes large temp files during the operation. Check the
>STREAM_DIR (defaults to /var/tmp) you assigned I would bet that one
>of those temp files exceeded 2 Gb. That seems to be the filesize
>limit on my distribution of linux.
>
>Davids

orkun wrote:

thank you
You are right.
that file made my disk full.
Although I used a small region for testing purposes. It produced
giantic files.
I do not know what will happen if I use whole region ?

what do you suggest ?

Try it and see what happens! (It has always worked for me)
Worst case you run out of disk space again. Maybe the resolution was
set way too high?

How big was the region that failed? (output of 'g.region -p'?)

Hamish

I've been having trouble with this too. It seems that on my system,
r.terraflow cannot make files larger than 2Gb. I CAN create large
files on my system. For example:

dd if=/dev/zero of=test.tmp bs=1024k count=4000

creates a 4 GB file without problems. So it must be either something
intrinsic to the code in r.terraflow or a library it is sharing.
That's were my skills run out. I'm not a real programmer.

Interestingly enough, I met Dr. Toma when she was writing the original
version of this code for her PhD (I supplied her a big DEM to test the
algorithm). If we can't make progress on our own, I can send her an
email and see if she has any thoughts on the source of the problem.

David

On 7/4/05, Hamish <hamish_nospam@yahoo.com> wrote:

orkun wrote:
> >>At first time I tried to run r.terraflow in a small test area. I
> >>failured. It gave this error:
> >>
> >>r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir
> >>swatershed=el5-watershed accumulation=el5-accu tci=el5-tci
> >>
> >>assigning directions on plateaus
> >>File size limit exceeded
> >>
> >>
> >>Could you tell me why ?

David Finlayson wrote:
> >r.terraflow makes large temp files during the operation. Check the
> >STREAM_DIR (defaults to /var/tmp) you assigned I would bet that one
> >of those temp files exceeded 2 Gb. That seems to be the filesize
> >limit on my distribution of linux.
> >
> >Davids

orkun wrote:
> thank you
> You are right.
> that file made my disk full.
> Although I used a small region for testing purposes. It produced
> giantic files.
> I do not know what will happen if I use whole region ?
>
> what do you suggest ?

Try it and see what happens! (It has always worked for me)
Worst case you run out of disk space again. Maybe the resolution was
set way too high?

How big was the region that failed? (output of 'g.region -p'?)

Hamish

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

A couple of things:

1) creating and accessing files larger than 2GB requires large file
support under Linux. Typically you can do this by including the compile
flags -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE when compiling
r.terraflow. If you do not see these flags when compiling, it will not
be able to read/write files bigger than 2GB (a 32-bit file offset)

2) The application must use off_t data types for file offsets and not
int/long/whatever... The compiler flags above make off_t a 64-bit type
and allows files upto something like 8 billion GB. There may be some
offsets in r.terraflow that are still int/long and need to convert to
off_t

3) I have some familiarity with the r.terraflow code and I'll be
looking at it for some clean-up starting next week. I can keep you
posted.

-Andy

On Tue, 2005-07-05 at 01:36 -0700, David Finlayson wrote:

I've been having trouble with this too. It seems that on my system,
r.terraflow cannot make files larger than 2Gb. I CAN create large
files on my system. For example:

dd if=/dev/zero of=test.tmp bs=1024k count=4000

creates a 4 GB file without problems. So it must be either something
intrinsic to the code in r.terraflow or a library it is sharing.
That's were my skills run out. I'm not a real programmer.

Interestingly enough, I met Dr. Toma when she was writing the original
version of this code for her PhD (I supplied her a big DEM to test the
algorithm). If we can't make progress on our own, I can send her an
email and see if she has any thoughts on the source of the problem.

David

On 7/4/05, Hamish <hamish_nospam@yahoo.com> wrote:
> orkun wrote:
> > >>At first time I tried to run r.terraflow in a small test area. I
> > >>failured. It gave this error:
> > >>
> > >>r.terraflow elev=el_5 filled=el5-filled dir=el5-mfdir
> > >>swatershed=el5-watershed accumulation=el5-accu tci=el5-tci
> > >>
> > >>assigning directions on plateaus
> > >>File size limit exceeded
> > >>
> > >>
> > >>Could you tell me why ?
>
> David Finlayson wrote:
> > >r.terraflow makes large temp files during the operation. Check the
> > >STREAM_DIR (defaults to /var/tmp) you assigned I would bet that one
> > >of those temp files exceeded 2 Gb. That seems to be the filesize
> > >limit on my distribution of linux.
> > >
> > >Davids
>
> orkun wrote:
> > thank you
> > You are right.
> > that file made my disk full.
> > Although I used a small region for testing purposes. It produced
> > giantic files.
> > I do not know what will happen if I use whole region ?
> >
> > what do you suggest ?
>
>
> Try it and see what happens! (It has always worked for me)
> Worst case you run out of disk space again. Maybe the resolution was
> set way too high?
>
> How big was the region that failed? (output of 'g.region -p'?)
>
>
>
> Hamish
>

Pretty close actually...I am helping a research team study non-point
source pollution in Hood Canal, Washington (close to
Seattle/Vancouver). It seems that leaky septic systems around the
fjord are exacerbating already low oxygen conditions leading to major
fish-kills during upwelling events.

My PhD research (nearly done!) is on the morphology of low-energy
beaches in fjords and its relationship with sea grasses on the
low-tide terrace. I've been meaning to contact you about using GRASS
with SWAN. Check this out:

http://students.washington.edu/dfinlays/waves/

Thanks,

David

BTW, I'm a physical oceanographer working in the New Zealand fjords. You
aren't by any chance using r.terraflow + land use/geologic data to have
a look at pollution in esturaries are you? We have a new project here
starting up in the next month or two & if that is what you're doing, we
sure would appreciate any pointers.

cheers,
Hamish

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

Thanks Laura.

David

On 7/5/05, Laura Toma <ltoma@bowdoin.edu> wrote:

I changed in r.terraflow/IOStream/lib/src/Makefile

line9: CXX = g++ -Wall

to

CXX = g++ -Wall -D_FILE_OFFSET_BITS=64

Then I cleaned everything, and recompiled, and it works now for me (on
a Linux machine).

cd IOStream/lib/src/
make clean
make
cd ..
make
cd ../../
gmake
etc

> orkun wrote:
>> thank you
>> You are right.
>> that file made my disk full.
>> Although I used a small region for testing purposes. It produced
>> giantic files.
>> I do not know what will happen if I use whole region ?
>>

The files produced have a maxsize of 80N bytes, where N = nrows x
ncols of your grid. They will be smaller, if your grid contains (a lot
of) nodata. Check out the html page of r.terraflow for details.

As you run terraflow, the first lines will tell you what is the grid
size, and the max size of the temporary files.
How small was the region you tried it on? To produce a file >2GB, you
must have used a grid larger than 3600x3600 cells.

Hope this helps,
Laura

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

Hi,

is this large file support detected during configure and then enabled,
if OS supports it? Lots of endusers do not understand that -D blablah
thing.

wbr,
Maris.

On 7/5/05, Andrew Danner <adanner@cs.duke.edu> wrote:

A couple of things:

1) creating and accessing files larger than 2GB requires large file
support under Linux. Typically you can do this by including the compile
flags -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE when compiling
r.terraflow. If you do not see these flags when compiling, it will not
be able to read/write files bigger than 2GB (a 32-bit file offset)

My PhD research (nearly done!) is on the morphology of low-energy
beaches in fjords and its relationship with sea grasses on the
low-tide terrace. I've been meaning to contact you about using GRASS
with SWAN. Check this out:

http://students.washington.edu/dfinlays/waves/

Nice.

Some tips on using GRASS with the SWAN nearshore wave model follow.

SWAN (GPL'd): http://fluidmechanics.tudelft.nl/swan/default.htm
"Simulating WAves Nearshore"

Unknown to me if SWAN works with GCC 4's new fortran 90/95 compiler.
If so, we have an all Free-software chain.

You can use GRASS to make an elevation map then export it directly into
something that can be loaded by the model with r.out.ascii. You can
parse the top lines of that ASCII file to get the region extents for the
model, but be warned that SWAN's use of grid-points vs. cell-center
coordinate conventions aren't terribly consistent, so be careful adding
and subtracting one cell to get everything to match up where you intend
it to be (& triple check).

This can be a problem when you reload the data into GRASS if the data is
shifted by half a cell. If you don't reset the bounds correctly or
double the resolution, the data at each grid point will shift itself to
match the current region (ie by 1/2 a cell both in x and y) and you'll
see all the raster data no longer matching up with e.g. vector coastline
data. Changing the region settings or fixing your adjustments will make
it ok again.

SWAN crashes with a segfault (for me anyway) if you try and make your
computational grid bigger than about 500x500 (give or take period and
radial resolution settings). SWAN's memory handling is IMO pretty
wasteful, but it's a lot better than it used to be. Buy 2gb RAM.

SWAN's "BLOCK" output command can send data to a Matlab array. Do any
post-processing there (obligatory "or use Octave" plug) (well, r.mapcalc
could work too), re-save and import into GRASS with r.in.mat. (Importing
SWAN data into grass is why I wrote r.in.mat by the way) Once you have
your scripts set up to automate the creation of the swan command file
from r.out.ascii output & script the post processing to give you a .mat
file that GRASS can load, it's a really smooth process you can apply to
many different shorelines just by using d.zoom!

To get the shape of the beach correct (which is kind of critical if
you're looking at modelling waves crashing against them) make sure you
are careful when you merge any bathymetry data with coastline, island,
and topography elevation data. They are sure to use different vertical
datums (lowest low water vs. mean high water springs, etc).

good luck,

Hamish

I don't have matlab, but I have achieved much the same thing using GMT
and Python (and a lot of hand editing of the command files). I have a
lot of this to do over the next few months and I wanted to smooth out
the work flow with GRASS since all of the land cover data is
classified imagery. You've given me some good ideas.

David

On 7/10/05, Hamish <hamish_nospam@yahoo.com> wrote:

> My PhD research (nearly done!) is on the morphology of low-energy
> beaches in fjords and its relationship with sea grasses on the
> low-tide terrace. I've been meaning to contact you about using GRASS
> with SWAN. Check this out:
>
> http://students.washington.edu/dfinlays/waves/

Nice.

Some tips on using GRASS with the SWAN nearshore wave model follow.

SWAN (GPL'd): http://fluidmechanics.tudelft.nl/swan/default.htm
"Simulating WAves Nearshore"

Unknown to me if SWAN works with GCC 4's new fortran 90/95 compiler.
If so, we have an all Free-software chain.

You can use GRASS to make an elevation map then export it directly into
something that can be loaded by the model with r.out.ascii. You can
parse the top lines of that ASCII file to get the region extents for the
model, but be warned that SWAN's use of grid-points vs. cell-center
coordinate conventions aren't terribly consistent, so be careful adding
and subtracting one cell to get everything to match up where you intend
it to be (& triple check).

This can be a problem when you reload the data into GRASS if the data is
shifted by half a cell. If you don't reset the bounds correctly or
double the resolution, the data at each grid point will shift itself to
match the current region (ie by 1/2 a cell both in x and y) and you'll
see all the raster data no longer matching up with e.g. vector coastline
data. Changing the region settings or fixing your adjustments will make
it ok again.

SWAN crashes with a segfault (for me anyway) if you try and make your
computational grid bigger than about 500x500 (give or take period and
radial resolution settings). SWAN's memory handling is IMO pretty
wasteful, but it's a lot better than it used to be. Buy 2gb RAM.

SWAN's "BLOCK" output command can send data to a Matlab array. Do any
post-processing there (obligatory "or use Octave" plug) (well, r.mapcalc
could work too), re-save and import into GRASS with r.in.mat. (Importing
SWAN data into grass is why I wrote r.in.mat by the way) Once you have
your scripts set up to automate the creation of the swan command file
from r.out.ascii output & script the post processing to give you a .mat
file that GRASS can load, it's a really smooth process you can apply to
many different shorelines just by using d.zoom!

To get the shape of the beach correct (which is kind of critical if
you're looking at modelling waves crashing against them) make sure you
are careful when you merge any bathymetry data with coastline, island,
and topography elevation data. They are sure to use different vertical
datums (lowest low water vs. mean high water springs, etc).

good luck,

Hamish

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

I recently updated my binary install of grass:

grass6.1.cvs-i686-pc-linux-gnu-13_08_2005

and noticed it still did not have the large file support compiled in
for r.terraflow. Is this something that will remain off by default? In
other words, do I need to get off my duff and learn how to check out
GRASS code from CVS and compile it myself?

David

On 7/8/05, Markus Neteler <neteler@itc.it> wrote:

On Fri, Jul 08, 2005 at 06:06:44PM +1200, Hamish wrote:
> On Tue, 05 Jul 2005 11:16:47 -0400
> Laura Toma <ltoma@bowdoin.edu> wrote:
>
> >
> > I changed in r.terraflow/IOStream/lib/src/Makefile
> >
> > line9: CXX = g++ -Wall
> >
> > to
> >
> > CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
> >
> > Then I cleaned everything, and recompiled, and it works now for me (on
> > a Linux machine).
>
>
> GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
> $(CPPFLAGS).
>
> Does either of those actually get the needed LFS flags if GRASS is
> configured with --enable-largefile ?
>

I don't think so. Look at
grass61/lib/gis/Makefile

there is:

#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
        EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif

This must the added to the r.terraflow Makefiles.

Markus

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

On Fri, Aug 26, 2005 at 11:00:30PM -0700, David Finlayson wrote:

I recently updated my binary install of grass:

grass6.1.cvs-i686-pc-linux-gnu-13_08_2005

and noticed it still did not have the large file support compiled in
for r.terraflow. Is this something that will remain off by default?
In other words, do I need to get off my duff and learn how to check out
GRASS code from CVS and compile it myself?

I just see that large file support is disabled in the
cronjob generating the binaries.
If there are no objections, I can activate it for next
Saturday.

However, I don't know which changes are needed in the CVS to
enable it for r.terraflow as well (see old mail below, seems to
be C related rather than C++)

Markus

David

On 7/8/05, Markus Neteler <neteler@itc.it> wrote:
> On Fri, Jul 08, 2005 at 06:06:44PM +1200, Hamish wrote:
> > On Tue, 05 Jul 2005 11:16:47 -0400
> > Laura Toma <ltoma@bowdoin.edu> wrote:
> >
> > >
> > > I changed in r.terraflow/IOStream/lib/src/Makefile
> > >
> > > line9: CXX = g++ -Wall
> > >
> > > to
> > >
> > > CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
> > >
> > > Then I cleaned everything, and recompiled, and it works now for me (on
> > > a Linux machine).
> >
> >
> > GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
> > $(CPPFLAGS).
> >
> > Does either of those actually get the needed LFS flags if GRASS is
> > configured with --enable-largefile ?
> >
>
> I don't think so. Look at
> grass61/lib/gis/Makefile
>
> there is:
>
> #compile if LFS Large File Support present:
> ifneq ($(USE_LARGEFILES),)
> EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
> endif
>
> This must the added to the r.terraflow Makefiles.
>
> Markus
>
>

--
David Finlayson
Marine Geology & Geophysics
School of Oceanography
Box 357940
University of Washington
Seattle, WA 98195-7940
USA

Office: Marine Sciences Building, Room 112
Phone: (206) 616-9407
Web: http://students.washington.edu/dfinlays

--
Markus Neteler <neteler itc it> http://mpa.itc.it
ITC-irst - Centro per la Ricerca Scientifica e Tecnologica
MPBA - Predictive Models for Biol. & Environ. Data Analysis
Via Sommarive, 18 - 38050 Povo (Trento), Italy