On Tue, 05 Jul 2005 11:16:47 -0400
Laura Toma <ltoma@bowdoin.edu> wrote:
I changed in r.terraflow/IOStream/lib/src/Makefile
line9: CXX = g++ -Wall
to
CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
Then I cleaned everything, and recompiled, and it works now for me (on
a Linux machine).
GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
$(CPPFLAGS).
Does either of those actually get the needed LFS flags if GRASS is
configured with --enable-largefile ?
Hamish
On Fri, Jul 08, 2005 at 06:06:44PM +1200, Hamish wrote:
On Tue, 05 Jul 2005 11:16:47 -0400
Laura Toma <ltoma@bowdoin.edu> wrote:>
> I changed in r.terraflow/IOStream/lib/src/Makefile
>
> line9: CXX = g++ -Wall
>
> to
>
> CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
>
> Then I cleaned everything, and recompiled, and it works now for me (on
> a Linux machine).GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
$(CPPFLAGS).Does either of those actually get the needed LFS flags if GRASS is
configured with --enable-largefile ?
I don't think so. Look at
grass61/lib/gis/Makefile
there is:
#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif
This must the added to the r.terraflow Makefiles.
Markus
Don't know, maybe (depends on the configure script?) I guess the safest way is to try it and see what the g++ command becomes when compiling the library.
I only tried it for grass 5.x, and it did not work without the largefile flag in both Makefiles.
-Laura
On Jul 8, 2005, at 2:06 AM, Hamish wrote:
On Tue, 05 Jul 2005 11:16:47 -0400
Laura Toma <ltoma@bowdoin.edu> wrote:I changed in r.terraflow/IOStream/lib/src/Makefile
line9: CXX = g++ -Wall
to
CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
Then I cleaned everything, and recompiled, and it works now for me (on
a Linux machine).GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
$(CPPFLAGS).Does either of those actually get the needed LFS flags if GRASS is
configured with --enable-largefile ?Hamish
> On Tue, 05 Jul 2005 11:16:47 -0400
> Laura Toma <ltoma@bowdoin.edu> wrote:
> >
> > I changed in r.terraflow/IOStream/lib/src/Makefile
> >
> > line9: CXX = g++ -Wall
> >
> > to
> >
> > CXX = g++ -Wall -D_FILE_OFFSET_BITS=64
> >
> > Then I cleaned everything, and recompiled, and it works now for me (on
> > a Linux machine).
>
>
> GRASS 6's r.terraflow/IOStream/lib/src/Makefile mentions $(CXX) and
> $(CPPFLAGS).
>
> Does either of those actually get the needed LFS flags if GRASS is
> configured with --enable-largefile ?
>I don't think so. Look at
grass61/lib/gis/Makefilethere is:
#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endifThis must the added to the r.terraflow Makefiles.
Should that be changed to:
#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
EXTRA_CPPFLAGS = -D_FILE_OFFSET_BITS=64
endif
and then add EXTRA_CPPFLAGS to raster/r.terraflow/IOStream/lib/src/Makefile?
I notice raster/r.terraflow/Makefile also uses $(CXXFLAGS).
a) what's up with having both $(CPPFLAGS) and $(CXXFLAGS) ??
b) add EXTRA_C??FLAGS to all r.terraflow Makefiles or just
IOStream/lib/src ??
Hamish
From the thread history, I surmised that the >2GB exception was fixed. However, when I use the latest snapshop binaries I still get the error. I have verified that my kernel supports large files by creating several >2GB files. Anyone have any advice? I have pasted the output below.
(using grass-6.1.cvs-i686-pc-linux-gnu-21_01_2006.tar.gz on Redhat 9.0)
GRASS 6.1.cvs (lcra):~/grassinstall > r.terraflow elev=test filled=test_out direction=test_dir swatershed=test_shed accumulation=test_acc tci=test_tci
r.terraflow December 2003
region size is 28458 x 28438
cell test header compatible with region header
elevation stored as FLOAT (4B)
STREAM temporary files in /var/tmp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
MFD flow direction
D8CUT=999999986991104.000000
memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
reading data from test to stream /var/tmp/STREAM_Ke0baI: 63% File size limit exceeded (core dumped)
GRASS 6.1.cvs (lcra):~/grassinstall > ls -lh /var/tmp/
total 2.1G
-rw------- 1 endeitz endeitz 2.0G Jan 27 13:43 STREAM_Ke0baI
GRASS 6.1.cvs (lcra):~/grassinstall > dd if=/dev/zero of=testsize bs=1024k count=3000
3000+0 records in
3000+0 records out
GRASS 6.1.cvs (lcra):~/grassinstall > ls -lh /var/tmp
total 5.0G
-rw------- 1 endeitz endeitz 2.0G Jan 27 13:43 STREAM_Ke0baI
-rw-r–r-- 1 endeitz endeitz 2.9G Jan 27 13:47 testsize
----------------------------------end output
Thanks,
Ed.
View this message in context: Re: r.terraflow failured
Sent from the Grass5 - Dev forum at Nabble.com.
Just a followup to my previous message. Instead of using the snapshot binary
I compiled GRASS myself. This apparently fixed the 2GB issue. However,
r.terraflow is still crashing on what appears to be a seek error of some
kind on a file of approximately 7GB (again, is it a file size limit?). I
was wondering if anyone could give me a clue. The error is near the end of
the listing below. I also show ulimit results below, if that is relevant.
-----------------------------
GRASS 6.1.cvs (lcra):~/grassdata > ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3071
virtual memory (kbytes, -v) unlimited
---------------------------------------------------------------
r.terraflow December 2003
region size is 13458 x 13438
cell elev15000 header compatible with region header
elevation stored as FLOAT (4B)
STREAM temporary files in /var/tmp (THESE INTERMEDIATE STREAMS WILL NOT BE
DELETED IN CASE OF ABNORMAL TERM
file stats.out exists - renaming.
reading data from elev15000 to stream /var/tmp/STREAM_djueQG:
total elements=180848604, nodata elements=0
largest temporary files:
FILL: 9.43G (10127521824) [180848604 elements, 56B each]
FLOW: 13.47G (14467888320) [180848604 elements, 80B each]
Will need at least 26.95G (28935776640) space available in /var/tmp
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17722B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17753B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17784B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17815B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17846B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17718B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17780B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17811B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17718B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17780B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17811B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14518B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14549B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14580B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14611B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14642B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14673B.
r.terraflow: 3scan.h:163: void scan3line(FUN&, AMI_STREAM<T>*,
AMI_STREAM<T>*, AMI_STREAM<T>*, BASETYPE, short int) [with T =
waterGridType, BASETYPE = waterWindowBaseType, FUN = waterWindower]:
Assertion `ae == AMI_ERROR_END_OF_STREAM' failed.
--------------------------------------------------
GRASS 6.1.cvs (lcra):~/grassdata > ls -lh /var/tmp
total 11G
-rw------- 1 nedeeds nedeeds 690M Feb 18 16:29 STREAM_a5xEvs
-rw------- 1 nedeeds nedeeds 7.0G Feb 18 17:14 STREAM_LjcOQz
-rw------- 1 nedeeds nedeeds 2.7G Feb 18 16:59 STREAM_TtzchB
GRASS 6.1.cvs (lcra):~/grassdata >
--
View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3033415
Sent from the Grass5 - Dev forum at Nabble.com.
My initial hunch is that this may be an fseek error. The function
scan3line where the assertion fails is opening three file descriptors or
"substreams" on a single file and calling fseek to move to the beginning
of a particular row. However, fseek takes a 32-bit long offset which
overflows at 2GB. There is an fseeko in most stdio.h implementations
that takes a 64 bit off_t offset and works for larger files.
Unfortunately I do not have time in the near future to investigate this
further. My initial suggestion is to change fseek to fseeko in line 397
of
raster/r.terraflow/IOStream/include/ami_stream.h
rebuild the terraflow module and see if that works. Certainly, calling
fseek instead of fseeko is a problem, but I'm not sure if that is the
only problem. Terraflow is using 64-bit types that handle offsets larger
than 2GB in a lot of other places, but it may be a bit buggy in spots.
If you try the fix, let me know what happens and perhaps I can help out
some more if things are still not working.
-Andy
On Mon, 2006-02-20 at 08:24 -0800, endeitz wrote:
Just a followup to my previous message. Instead of using the snapshot binary
I compiled GRASS myself. This apparently fixed the 2GB issue. However,
r.terraflow is still crashing on what appears to be a seek error of some
kind on a file of approximately 7GB (again, is it a file size limit?). I
was wondering if anyone could give me a clue. The error is near the end of
the listing below. I also show ulimit results below, if that is relevant.-----------------------------
GRASS 6.1.cvs (lcra):~/grassdata > ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3071
virtual memory (kbytes, -v) unlimited---------------------------------------------------------------
r.terraflow December 2003
region size is 13458 x 13438
cell elev15000 header compatible with region header
elevation stored as FLOAT (4B)
STREAM temporary files in /var/tmp (THESE INTERMEDIATE STREAMS WILL NOT BE
DELETED IN CASE OF ABNORMAL TERM
file stats.out exists - renaming.
reading data from elev15000 to stream /var/tmp/STREAM_djueQG:total elements=180848604, nodata elements=0
largest temporary files:
FILL: 9.43G (10127521824) [180848604 elements, 56B each]
FLOW: 13.47G (14467888320) [180848604 elements, 80B each]
Will need at least 26.95G (28935776640) space available in /var/tmp
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17722B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17753B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17784B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17815B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17846B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17718B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17780B.
MM warning: limit=314572800B. allocating 157138456B. limit exceeded by
17811B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17718B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17749B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17780B.
MM warning: limit=314572800B. allocating 157015336B. limit exceeded by
17811B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14518B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14549B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14580B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14611B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14642B.
MM warning: limit=314572800B. allocating 157220512B. limit exceeded by
14673B.
r.terraflow: 3scan.h:163: void scan3line(FUN&, AMI_STREAM<T>*,
AMI_STREAM<T>*, AMI_STREAM<T>*, BASETYPE, short int) [with T =
waterGridType, BASETYPE = waterWindowBaseType, FUN = waterWindower]:
Assertion `ae == AMI_ERROR_END_OF_STREAM' failed.--------------------------------------------------
GRASS 6.1.cvs (lcra):~/grassdata > ls -lh /var/tmp
total 11G
-rw------- 1 nedeeds nedeeds 690M Feb 18 16:29 STREAM_a5xEvs
-rw------- 1 nedeeds nedeeds 7.0G Feb 18 17:14 STREAM_LjcOQz
-rw------- 1 nedeeds nedeeds 2.7G Feb 18 16:59 STREAM_TtzchB
GRASS 6.1.cvs (lcra):~/grassdata >--
View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3033415
Sent from the Grass5 - Dev forum at Nabble.com._______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5
Thanks very much for your help. However, after replacing the fseek with
fseeko, the result is the same. Can grass binaries be run under a
debugger(perhaps gdb)? That way I could easily break on the value of the
offset to see if that is where it is crashing.
Cheers,
Ed.
Andrew Danner wrote:
My initial hunch is that this may be an fseek error. The function
scan3line where the assertion fails is opening three file descriptors or
"substreams" on a single file and calling fseek to move to the beginning
of a particular row. However, fseek takes a 32-bit long offset which
overflows at 2GB. There is an fseeko in most stdio.h implementations
that takes a 64 bit off_t offset and works for larger files.Unfortunately I do not have time in the near future to investigate this
further. My initial suggestion is to change fseek to fseeko in line 397
ofraster/r.terraflow/IOStream/include/ami_stream.h
rebuild the terraflow module and see if that works. Certainly, calling
fseek instead of fseeko is a problem, but I'm not sure if that is the
only problem. Terraflow is using 64-bit types that handle offsets larger
than 2GB in a lot of other places, but it may be a bit buggy in spots.If you try the fix, let me know what happens and perhaps I can help out
some more if things are still not working.-Andy
On Mon, 2006-02-20 at 08:24 -0800, endeitz wrote:
Just a followup to my previous message. Instead of using the snapshot
binary-snip-
--
View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3053896
Sent from the Grass5 - Dev forum at Nabble.com.
Ed,
Yes, grass binaries can be run under gdb and perhaps other debuggers. I
typically envoke gdb from within the GRASS command prompt so that the
proper environment variables are already set up.
-Andy
On Tue, 2006-02-21 at 09:17 -0800, endeitz wrote:
Thanks very much for your help. However, after replacing the fseek with
fseeko, the result is the same. Can grass binaries be run under a
debugger(perhaps gdb)? That way I could easily break on the value of the
offset to see if that is where it is crashing.Cheers,
Ed.
Andrew Danner wrote:
>
> My initial hunch is that this may be an fseek error. The function
> scan3line where the assertion fails is opening three file descriptors or
> "substreams" on a single file and calling fseek to move to the beginning
> of a particular row. However, fseek takes a 32-bit long offset which
> overflows at 2GB. There is an fseeko in most stdio.h implementations
> that takes a 64 bit off_t offset and works for larger files.
>
> Unfortunately I do not have time in the near future to investigate this
> further. My initial suggestion is to change fseek to fseeko in line 397
> of
>
> raster/r.terraflow/IOStream/include/ami_stream.h
>
> rebuild the terraflow module and see if that works. Certainly, calling
> fseek instead of fseeko is a problem, but I'm not sure if that is the
> only problem. Terraflow is using 64-bit types that handle offsets larger
> than 2GB in a lot of other places, but it may be a bit buggy in spots.
>
> If you try the fix, let me know what happens and perhaps I can help out
> some more if things are still not working.
>
>
> -Andy
>
> On Mon, 2006-02-20 at 08:24 -0800, endeitz wrote:
>> Just a followup to my previous message. Instead of using the snapshot
>> binary
>
> -snip-
>
>
--
View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3053896
Sent from the Grass5 - Dev forum at Nabble.com._______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5
Sorry to keep pestering. . .
Where would I put the flag to compile with debugging symbols on? Would it
suffice to just put it in the Makefile in the r.terraflow directory, or does
it have to appear higher up the tree?
Cheers,
Ed.
Andrew Danner wrote:
Ed,
Yes, grass binaries can be run under gdb and perhaps other debuggers. I
typically envoke gdb from within the GRASS command prompt so that the
proper environment variables are already set up.-Andy
On Tue, 2006-02-21 at 09:17 -0800, endeitz wrote:
Thanks very much for your help. However, after replacing the fseek with
fseeko, the result is the same. Can grass binaries be run under a
debugger(perhaps gdb)? That way I could easily break on the value of the
offset to see if that is where it is crashing.Cheers,
Ed.
Andrew Danner wrote:
>
> My initial hunch is that this may be an fseek error. The function
> scan3line where the assertion fails is opening three file descriptors
or
> "substreams" on a single file and calling fseek to move to the
beginning
> of a particular row. However, fseek takes a 32-bit long offset which
> overflows at 2GB. There is an fseeko in most stdio.h implementations
> that takes a 64 bit off_t offset and works for larger files.
>
> Unfortunately I do not have time in the near future to investigate this
> further. My initial suggestion is to change fseek to fseeko in line 397
> of
>
> raster/r.terraflow/IOStream/include/ami_stream.h
>
> rebuild the terraflow module and see if that works. Certainly, calling
> fseek instead of fseeko is a problem, but I'm not sure if that is the
> only problem. Terraflow is using 64-bit types that handle offsets
larger
> than 2GB in a lot of other places, but it may be a bit buggy in spots.
>
> If you try the fix, let me know what happens and perhaps I can help out
> some more if things are still not working.
>
>
> -Andy
>
> On Mon, 2006-02-20 at 08:24 -0800, endeitz wrote:
>> Just a followup to my previous message. Instead of using the snapshot
>> binary
>
> -snip-
--
View this message in context: http://www.nabble.com/Re%3A-GRASSLIST%3A7448-Re%3A-r.terraflow-failured-t130887.html#a3056650
Sent from the Grass5 - Dev forum at Nabble.com.
endeitz wrote:
Where would I put the flag to compile with debugging symbols on? Would it
suffice to just put it in the Makefile in the r.terraflow directory, or does
it have to appear higher up the tree?
Debugging symbols should be enabled by default. If you're doing
extensive debugging, you probably want to disable optimisation, so
either run configure as:
CXXFLAGS=-g ./configure ...
or build with:
make CXXFLAGS1=-g
--
Glynn Clements <glynn@gclements.plus.com>
Andrew Danner wrote:
My initial hunch is that this may be an fseek error. The function
scan3line where the assertion fails is opening three file descriptors or
"substreams" on a single file and calling fseek to move to the beginning
of a particular row. However, fseek takes a 32-bit long offset which
overflows at 2GB. There is an fseeko in most stdio.h implementations
that takes a 64 bit off_t offset and works for larger files.
Just to clarify: fseeko takes an off_t, which may be either 32 or 64
bits depending upon the value of the _FILE_OFFSET_BITS macro.
Note that, in order to use 64 bit offsets, you need to ensure that
values don't get truncated to 32 bits by C's typing rules and/or
storage in int/long variables.
--
Glynn Clements <glynn@gclements.plus.com>