[GRASS-dev] r.sim.water buffer overflow

Hi,

I'm posting here because I had no reaction on the users email list.

I'm using GRASS64 RC6 on Linux 10.04 (64-bit) and I'm trying to run
r.sim.water on a catchment about 5000 km2 in size.

I started with a very fine resolution in my region (00:00:00.18) and
got the following:
-----
GRASS 6.4.0RC6 (world_wgs84):~ > r.sim.water
elevin=lieb_dem_25m_clipped dxin=lieb_dem_25m_clipped_dx
dyin=lieb_dem_25m_clipped_dy rain_val=2.0 manin_val=0.1
infil_val=13.0 depth=rsimwater.depth.18s disch=rsimwater.disch.18s
default nwalk=600505650, rwalk=600505650.000000
Killed
GRASS 6.4.0RC6 (world_wgs84):~ >
(python:1572): Pango-CRITICAL **: pango_layout_get_cursor_pos:
assertion `index >= 0 && index <= layout->length' failed
^C
----

Since this resolution gives about 300 million cells in the region, I
progressively coarsened the resolution, but I get buffer overflows
even with just 2736 cells in the region (resolution = 00:01:00)

Below is an example of the error I got at a resolution of 1 minute.
The output at all resolutions (except 00:00:00.18) is similar:
-----
GRASS 6.4.0RC6 (world_wgs84):~ > r.sim.water
elevin=lieb_dem_25m_clipped dxin=lieb_dem_25m_clipped_dx
dyin=lieb_dem_25m_clipped_dy rain_val=50.0 manin_val=0.1
infil_val=13.0 depth=rsimwater.depth_1min disch=rsimwater.disch_1min
--overwrite
default nwalk=5472, rwalk=5472.000000

Min elevation = 1500.00 m
Max elevation = 2351.17 m
Mean Source Rate (rainf. excess or sediment) = 0.000006 m/s or kg/m2s
Mean flow velocity = 1.135392 m/s
Mean Mannings = 0.170467
Number of iterations = 41128 cells
Time step = 0.00 s
*** buffer overflow detected ***: r.sim.water terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x7ffab34e7207]
/lib/libc.so.6(+0xfe0c0)[0x7ffab34e60c0]
/lib/libc.so.6(+0xfd529)[0x7ffab34e5529]
/lib/libc.so.6(_IO_default_xsputn+0xcc)[0x7ffab345dd1c]
/lib/libc.so.6(_IO_vfprintf+0x3d34)[0x7ffab34310d4]
/lib/libc.so.6(__vsprintf_chk+0x99)[0x7ffab34e55c9]
/lib/libc.so.6(__sprintf_chk+0x7f)[0x7ffab34e550f]
/usr/lib/grass64/lib/libgrass_sim.so(output_data+0x1b1a)[0x7ffab5f32d35]
r.sim.water(main+0x1041)[0x4037e5]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7ffab3406c4d]
r.sim.water[0x4026e9]
======= Memory map: ========
00400000-00405000 r-xp 00000000 08:05 545723
/usr/lib/grass64/bin/r.sim.water
00604000-00605000 r--p 00004000 08:05 545723
/usr/lib/grass64/bin/r.sim.water
00605000-00606000 rw-p 00005000 08:05 545723
/usr/lib/grass64/bin/r.sim.water
00606000-12bc4000 rw-p 00000000 00:00 0
14a19000-14ab6000 rw-p 00000000 00:00 0 [heap]
7ffaaa3c1000-7ffaaa3cd000 r-xp 00000000 08:05 262845
/lib/libnss_files-2.11.1.so
7ffaaa3cd000-7ffaaa5cc000 ---p 0000c000 08:05 262845
/lib/libnss_files-2.11.1.so
7ffaaa5cc000-7ffaaa5cd000 r--p 0000b000 08:05 262845
/lib/libnss_files-2.11.1.so
7ffaaa5cd000-7ffaaa5ce000 rw-p 0000c000 08:05 262845
/lib/libnss_files-2.11.1.so
7ffaaa5ce000-7ffaaa5d8000 r-xp 00000000 08:05 263050
/lib/libnss_nis-2.11.1.so
7ffaaa5d8000-7ffaaa7d7000 ---p 0000a000 08:05 263050
/lib/libnss_nis-2.11.1.so
7ffaaa7d7000-7ffaaa7d8000 r--p 00009000 08:05 263050
/lib/libnss_nis-2.11.1.so
7ffaaa7d8000-7ffaaa7d9000 rw-p 0000a000 08:05 263050
/lib/libnss_nis-2.11.1.so
7ffaaa7d9000-7ffaaa7e1000 r-xp 00000000 08:05 262840
/lib/libnss_compat-2.11.1.so
7ffaaa7e1000-7ffaaa9e0000 ---p 00008000 08:05 262840
/lib/libnss_compat-2.11.1.so
7ffaaa9e0000-7ffaaa9e1000 r--p 00007000 08:05 262840
/lib/libnss_compat-2.11.1.so
7ffaaa9e1000-7ffaaa9e2000 rw-p 00008000 08:05 262840
/lib/libnss_compat-2.11.1.so
7ffaaa9e2000-7ffaaa9e5000 r-xp 00000000 08:05 261719
/lib/libgpg-error.so.0.4.0
7ffaaa9e5000-7ffaaabe4000 ---p 00003000 08:05 261719
/lib/libgpg-error.so.0.4.0
7ffaaabe4000-7ffaaabe5000 r--p 00002000 08:05 261719
/lib/libgpg-error.so.0.4.0
7ffaaabe5000-7ffaaabe6000 rw-p 00003000 08:05 261719
/lib/libgpg-error.so.0.4.0
7ffaaabe6000-7ffaaabf6000 r-xp 00000000 08:05 137210
/usr/lib/libtasn1.so.3.1.7
7ffaaabf6000-7ffaaadf5000 ---p 00010000 08:05 137210
/usr/lib/libtasn1.so.3.1.7
7ffaaadf5000-7ffaaadf6000 r--p 0000f000 08:05 137210
/usr/lib/libtasn1.so.3.1.7
7ffaaadf6000-7ffaaadf7000 rw-p 00010000 08:05 137210
/usr/lib/libtasn1.so.3.1.7
7ffaaadf7000-7ffaaae10000 r-xp 00000000 08:05 137156
/usr/lib/libsasl2.so.2.0.23
7ffaaae10000-7ffaab00f000 ---p 00019000 08:05 137156
/usr/lib/libsasl2.so.2.0.23
7ffaab00f000-7ffaab010000 r--p 00018000 08:05 137156
/usr/lib/libsasl2.so.2.0.23
7ffaab010000-7ffaab011000 rw-p 00019000 08:05 137156
/usr/lib/libsasl2.so.2.0.23
7ffaab011000-7ffaab027000 r-xp 00000000 08:05 263380
/lib/libresolv-2.11.1.so
7ffaab027000-7ffaab226000 ---p 00016000 08:05 263380
/lib/libresolv-2.11.1.so
7ffaab226000-7ffaab227000 r--p 00015000 08:05 263380
/lib/libresolv-2.11.1.so
7ffaab227000-7ffaab228000 rw-p 00016000 08:05 263380
/lib/libresolv-2.11.1.so
7ffaab228000-7ffaab22a000 rw-p 00000000 00:00 0
7ffaab22a000-7ffaab22c000 r-xp 00000000 08:05 261726
/lib/libkeyutils-1.2.so
7ffaab22c000-7ffaab42b000 ---p 00002000 08:05 261726
/lib/libkeyutils-1.2.so
7ffaab42b000-7ffaab42c000 r--p 00001000 08:05 261726
/lib/libkeyutils-1.2.so
7ffaab42c000-7ffaab42d000 rw-p 00002000 08:05 261726
/lib/libkeyutils-1.2.so
7ffaab42d000-7ffaab434000 r-xp 00000000 08:05 135716
/usr/lib/libkrb5support.so.0.1
7ffaab434000-7ffaab633000 ---p 00007000 08:05 135716
/usr/lib/libkrb5support.so.0.1
7ffaab633000-7ffaab634000 r--p 00006000 08:05 135716
/usr/lib/libkrb5support.so.0.1
7ffaab634000-7ffaab635000 rw-p 00007000 08:05 135716
/usr/lib/libkrb5support.so.0.1
7ffaab635000-7ffaab659000 r-xp 00000000 08:05 132573
/usr/lib/libk5crypto.so.3.1
7ffaab659000-7ffaab859000 ---p 00024000 08:05 132573
/usr/lib/libk5crypto.so.3.1
7ffaab859000-7ffaab85a000 r--p 00024000 08:05 132573
/usr/lib/libk5crypto.so.3.1
7ffaab85a000-7ffaab85b000 rw-p 00025000 08:05 132573
/usr/lib/libk5crypto.so.3.1
7ffaab85b000-7ffaab872000 r-xp 00000000 08:05 262839
/lib/libnsl-2.11.1.so
7ffaab872000-7ffaaba71000 ---p 00017000 08:05 262839
/lib/libnsl-2.11.1.so
7ffaaba71000-7ffaaba72000 r--p 00016000 08:05 262839
/lib/libnsl-2.11.1.so
7ffaaba72000-7ffaaba73000 rw-p 00017000 08:05 262839
/lib/libnsl-2.11.1.so
7ffaaba73000-7ffaaba75000 rw-p 00000000 00:00 0
7ffaaba75000-7ffaabaea000 r-xp 00000000 08:05 261715
/lib/libgcrypt.so.11.5.2
7ffaabaea000-7ffaabce9000 ---p 00075000 08:05 261715
/lib/libgcrypt.so.11.5.2
7ffaabce9000-7ffaabcea000 r--p 00074000 08:05 261715
/lib/libgcrypt.so.11.5.2
7ffaabcea000-7ffaabced000 rw-p 00075000 08:05 261715
/lib/libgcrypt.so.11.5.2
7ffaabced000-7ffaabd89000 r-xp 00000000 08:05 136704
/usr/lib/libgnutls.so.26.14.12
7ffaabd89000-7ffaabf88000 ---p 0009c000 08:05 136704
/usr/lib/libgnutls.so.26.14.12
7ffaabf88000-7ffaabf8e000 r--p 0009b000 08:05 136704
/usr/lib/libgnutls.so.26.14.12
7ffaabf8e000-7ffaabf8f000 rw-p 000a1000 08:05 136704
/usr/lib/libgnutls.so.26.14.12
7ffaabf8f000-7ffaabf9c000 r-xp 00000000 08:05 133966
/usr/lib/liblber-2.4.so.2.5.4Aborted
-----

I can run the Spearfish example as shown in the r.sim.water manual
page and that region contains 2654802 cells.

My DEM is not rectangular and I'm setting the region to a vector file
that is an irregular boundary. So, the DEM has NULL values outside the
vector boundary but inside the region. Could this be a problem?

Or can someone perhaps spot another problem?

Thanks
Hanlie

On Wed, Aug 18, 2010 at 8:31 AM, Hanlie Pretorius
<hanlie.pretorius@gmail.com> wrote:

Hi,

I'm posting here because I had no reaction on the users email list.

(Hint: if you post an example for a sampe data set like Spearfish or
North Carolina, then it is way easier to reproduce for us...)

I'm using GRASS64 RC6 on Linux 10.04 (64-bit) and I'm trying to run
r.sim.water on a catchment about 5000 km2 in size.

I started with a very fine resolution in my region (00:00:00.18) and
got the following:
-----
GRASS 6.4.0RC6 (world_wgs84):~ > r.sim.water
elevin=lieb_dem_25m_clipped dxin=lieb_dem_25m_clipped_dx
dyin=lieb_dem_25m_clipped_dy rain_val=2.0 manin_val=0.1
infil_val=13.0 depth=rsimwater.depth.18s disch=rsimwater.disch.18s
default nwalk=600505650, rwalk=600505650.000000
Killed

Here for NC:

g.region -d res=1000 -p
r.slope.aspect elevation=elevation dx=elevation_dx dy=elevation_dy
r.sim.water elevin=elevation dxin=elevation_dx dyin=elevation_dy \
        rain_val=2.0 manin_val=0.1 infil_val=13.0 \
        depth=rsimwater.depth.18s disch=rsimwater.disch.18s
...
WARNING: Infiltration exceeds the rainfall rate everywhere! No overland
         flow.
...
Number of iterations = 0 cells
Time step = 129663.85 s
GRASS 6.4.0svn (nc_spm_08):~ >

So no crash. But my region had way less pixels.

----

Since this resolution gives about 300 million cells in the region, I
progressively coarsened the resolution, but I get buffer overflows
even with just 2736 cells in the region (resolution = 00:01:00)

The question is; how big are the internal tmp files?
Did you compile GRASS with large file support (--enable-largefile)?

Below is an example of the error I got at a resolution of 1 minute.
The output at all resolutions (except 00:00:00.18) is similar:

Second possibility is a bug when having LatLong coordinates. I
used a metric system.

...

Time step = 0.00 s
*** buffer overflow detected ***: r.sim.water terminated
======= Backtrace: =========

We would need a full backtrace (maybe not posted to the list)
which requires GRASS compiled with debugging symbols.

I can run the Spearfish example as shown in the r.sim.water manual
page and that region contains 2654802 cells.

Could you prepare a LatLong example with public data?

My DEM is not rectangular and I'm setting the region to a vector file
that is an irregular boundary. So, the DEM has NULL values outside the
vector boundary but inside the region. Could this be a problem?

I don't think so.

Markus

I am guessing it is the latlong but I would have to try it out,

Helena Mitasova
Associate Professor
Department of Marine, Earth and Atmospheric Sciences
North Carolina State University
Raleigh, NC 27695
hmitaso@unity.ncsu.edu

On Aug 18, 2010, at 11:03 AM, Markus Neteler wrote:

On Wed, Aug 18, 2010 at 8:31 AM, Hanlie Pretorius
<hanlie.pretorius@gmail.com> wrote:

Hi,

I'm posting here because I had no reaction on the users email list.

(Hint: if you post an example for a sampe data set like Spearfish or
North Carolina, then it is way easier to reproduce for us...)

I'm using GRASS64 RC6 on Linux 10.04 (64-bit) and I'm trying to run
r.sim.water on a catchment about 5000 km2 in size.

I started with a very fine resolution in my region (00:00:00.18) and
got the following:
-----
GRASS 6.4.0RC6 (world_wgs84):~ > r.sim.water
elevin=lieb_dem_25m_clipped dxin=lieb_dem_25m_clipped_dx
dyin=lieb_dem_25m_clipped_dy rain_val=2.0 manin_val=0.1
infil_val=13.0 depth=rsimwater.depth.18s disch=rsimwater.disch.18s
default nwalk=600505650, rwalk=600505650.000000
Killed

Here for NC:

g.region -d res=1000 -p
r.slope.aspect elevation=elevation dx=elevation_dx dy=elevation_dy
r.sim.water elevin=elevation dxin=elevation_dx dyin=elevation_dy \
       rain_val=2.0 manin_val=0.1 infil_val=13.0 \
       depth=rsimwater.depth.18s disch=rsimwater.disch.18s
...
WARNING: Infiltration exceeds the rainfall rate everywhere! No overland
        flow.
...
Number of iterations = 0 cells
Time step = 129663.85 s
GRASS 6.4.0svn (nc_spm_08):~ >

So no crash. But my region had way less pixels.

----

Since this resolution gives about 300 million cells in the region, I
progressively coarsened the resolution, but I get buffer overflows
even with just 2736 cells in the region (resolution = 00:01:00)

The question is; how big are the internal tmp files?
Did you compile GRASS with large file support (--enable-largefile)?

Below is an example of the error I got at a resolution of 1 minute.
The output at all resolutions (except 00:00:00.18) is similar:

Second possibility is a bug when having LatLong coordinates. I
used a metric system.

...

Time step = 0.00 s
*** buffer overflow detected ***: r.sim.water terminated
======= Backtrace: =========

We would need a full backtrace (maybe not posted to the list)
which requires GRASS compiled with debugging symbols.

I can run the Spearfish example as shown in the r.sim.water manual
page and that region contains 2654802 cells.

Could you prepare a LatLong example with public data?

My DEM is not rectangular and I'm setting the region to a vector file
that is an irregular boundary. So, the DEM has NULL values outside the
vector boundary but inside the region. Could this be a problem?

I don't think so.

Markus
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

2010/8/18, Markus Neteler <neteler@osgeo.org>:

On Wed, Aug 18, 2010 at 8:31 AM, Hanlie Pretorius
<hanlie.pretorius@gmail.com> wrote:

Hi,

I'm posting here because I had no reaction on the users email list.

(Hint: if you post an example for a sampe data set like Spearfish or
North Carolina, then it is way easier to reproduce for us...)

I'm using GRASS64 RC6 on Linux 10.04 (64-bit) and I'm trying to run
r.sim.water on a catchment about 5000 km2 in size.

I started with a very fine resolution in my region (00:00:00.18) and
got the following:
-----
GRASS 6.4.0RC6 (world_wgs84):~ > r.sim.water
elevin=lieb_dem_25m_clipped dxin=lieb_dem_25m_clipped_dx
dyin=lieb_dem_25m_clipped_dy rain_val=2.0 manin_val=0.1
infil_val=13.0 depth=rsimwater.depth.18s disch=rsimwater.disch.18s
default nwalk=600505650, rwalk=600505650.000000
Killed

Here for NC:

g.region -d res=1000 -p
r.slope.aspect elevation=elevation dx=elevation_dx dy=elevation_dy
r.sim.water elevin=elevation dxin=elevation_dx dyin=elevation_dy \
        rain_val=2.0 manin_val=0.1 infil_val=13.0 \
        depth=rsimwater.depth.18s disch=rsimwater.disch.18s
...
WARNING: Infiltration exceeds the rainfall rate everywhere! No overland
         flow.
...
Number of iterations = 0 cells
Time step = 129663.85 s
GRASS 6.4.0svn (nc_spm_08):~ >

So no crash. But my region had way less pixels.

----

Since this resolution gives about 300 million cells in the region, I
progressively coarsened the resolution, but I get buffer overflows
even with just 2736 cells in the region (resolution = 00:01:00)

The question is; how big are the internal tmp files?
Did you compile GRASS with large file support (--enable-largefile)?

I did not compile GRASS myself - have not tried that before - so I
just used the version as the Ubuntu package manager installed it.

I will try to find out how to compile GRASS and then use to options to
enable large file support and the debugging symbols.

Below is an example of the error I got at a resolution of 1 minute.
The output at all resolutions (except 00:00:00.18) is similar:

Second possibility is a bug when having LatLong coordinates. I
used a metric system.

...

Time step = 0.00 s
*** buffer overflow detected ***: r.sim.water terminated
======= Backtrace: =========

We would need a full backtrace (maybe not posted to the list)
which requires GRASS compiled with debugging symbols.

I can run the Spearfish example as shown in the r.sim.water manual
page and that region contains 2654802 cells.

Could you prepare a LatLong example with public data?

I will get the Spearfish dataset into a LatLong location and see what
happens when I run r.sim.water on that. I could place the Spearfish
latlong location on a temporary web site if needed.

My DEM is not rectangular and I'm setting the region to a vector file
that is an irregular boundary. So, the DEM has NULL values outside the
vector boundary but inside the region. Could this be a problem?

I don't think so.

Markus

Thanks for the help.

2010/8/19, Hanlie Pretorius <hanlie.pretorius@gmail.com>:

I will get the Spearfish dataset into a LatLong location and see what
happens when I run r.sim.water on that. I could place the Spearfish
latlong location on a temporary web site if needed.

I couldn't convert the spearfish60 location into a WGS84 location:
-----
GRASS 6.4.0RC6 (spearfish_wgs84):~ > r.proj input=elevation.10m
location=spearfish60 mapset=PERMANENT output=elevation.10m
method=cubic --overwrite
Input Projection Parameters: +proj=utm +zone=13 +a=6378206.4
+rf=294.9786982 +no_defs +nadgrids=/usr/lib/grass64/etc/nad/conus
Input Unit Factor: 1
Output Projection Parameters: +proj=longlat +no_defs +a=6378137
+rf=298.257223563 +towgs84=0.000,0.000,0.000
Output Unit Factor: 1
WARNING: pj_transform() failed: latitude or longitude exceeded limits
WARNING: pj_transform() failed: latitude or longitude exceeded limits
WARNING: pj_transform() failed: latitude or longitude exceeded limits
.
. (lots of these...)
.
WARNING: pj_transform() failed: latitude or longitude exceeded limits
WARNING: pj_transform() failed: latitude or longitude exceeded limits
WARNING: pj_transform() failed: latitude or longitude exceeded limits

Input:
Cols: 1899 (1899)
Rows: 1398 (1398)
North: 4928000.000000 (4928000.000000)
South: 4914020.000000 (4914020.000000)
West: 590010.000000 (590010.000000)
East: 609000.000000 (609000.000000)
EW-res: 10.000000
NS-res: 10.000000

Output:
Cols: 1 (360)
Rows: 1 (180)
North: 45.000000 (90.000000)
South: 44.000000 (-90.000000)
West: -104.000000 (-180.000000)
East: -103.000000 (180.000000)
EW-res: 1.000000
NS-res: 1.000000

Allocating memory and reading input map...
100%
Projecting...
100%
r.proj complete.
-----

r.info for the result:
-----
| Type of Map: raster Number of Categories: 255
| Data Type: FCELL
| Rows: 1
| Columns: 1
| Total Cells: 1
| Projection: Latitude-Longitude
| N: 45N S: 44N Res: 1
| E: 103W W: 104W Res: 1
| Range of data: min = -nan max = -nan
-----

g.region was set to:
-----
g.region -p n=90 s=-90 e=180 w=-180
projection: 3 (Latitude-Longitude)
zone: 0
datum: wgs84
ellipsoid: wgs84
north: 90N
south: 90S
west: 180W
east: 180E
nsres: 1
ewres: 1
rows: 180
cols: 360
cells: 64800
-----

However, I do have my dataset in a Transverse Mercator projected
coordinate system, so I ran r.sim.water using that data (66MB, posted
on http://www.nedbib.za.net/rsimwater/ if you want to download the
location) and I didn't get the buffer overflow, but I now get strange
results.

At a resolution of 100m (822120 cells), I get an output map that looks
like something ('lieb_depth_100m_res' raster in my downloadable
location). However, at a resolution of 25m - the original resolution
of the DEM - (13153920 cells) I get 0 depth over the whole region
('lieb_depth_25m_res' raster in my downloadable location)

To set the region, I use a command such as:
g.region -p vect=lieb_border res=25

Also, r.sim.water runs implausibly fast and I don't get all the output
that I used to get, like the time it took for the command to complete.
For example:
-----
GRASS 6.4.0RC6 (sa_tm_29deg_E_extract):~ > r.sim.water
elevin=lieb_dem_25m dxin=lieb_dem_25m_dx dyin=lieb_dem_25m_dy
depth=lieb_depth_25m_res --overwrite
default nwalk=26307840, rwalk=26307840.000000

Min elevation = 1498.27 m
Max elevation = 2445.12 m
Mean Source Rate (rainf. excess or sediment) = 0.000006 m/s or kg/m2s
Mean flow velocity = 1.166379 m/s
Mean Mannings = 0.170640
Number of iterations = 27 cells
Time step = 5.36 s
GRASS 6.4.0RC6 (sa_tm_29deg_E_extract):~ >
-----

So, it seems the LatLong coordinate system was part of the problem,
but I'm not out of the woods yet. Should I still try to compile GRASS
myself with the debugging symbols?

Regards
Hanlie