[GRASS-user] execGRASS("r.diversity") Done. No rasters created (Large rasters)

Sandra_MacFadyen · June 14, 2014, 7:21am

Dear GRASS Users,

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through R (R version 3.0.2 win32) on Windows 7 64bit.

Testing from GRASS and North_Carolina dataset's lsat5_1987_10 raster (cells=29535), [r.diversity --o input=lsat5_1987_10@landsat prefix=lsat5_1987_div_ alpha=0.5 size=5] reports as Done with the different diversity rasters added to the location fine (see # North_Carolina# code and results below).

However, when running the same code on a larger image (cells=6746328) from my own location, although it reports Done, no rasters are created. If I subset the image (cells=1632830) and run it again its works (see # sub2Kruger # code and results below). So I'm guessing it is a memory issue? My question is, how can I run r.diversity on an entire Landsat scence (cells= 54886341) as illustrated in http://www.faunalia.eu/pdf/ECOINF_2013.pdf

Thank you
Sandra

# North_Carolina#
g.gisenv set=DEBUG=1
g.gisenv -n
g.region -p
g.region rast=lsat5_1987_10@landsat
g.region -p
r.diversity --o input=lsat5_1987_10@landsat prefix=lsat5_1987_div_ alpha=0.5 size=5
g.gisenv set=DEBUG=0

#------------------------------------
(Sat Jun 14 08:01:50 2014)
g.gisenv set=DEBUG=1
(Sat Jun 14 08:01:51 2014) Command finished (0 sec)
(Sat Jun 14 08:05:42 2014)
g.gisenv -n
MAPSET=PERMANENT
GISDBASE=C:\Users\Sands\Documents\grassdata
LOCATION_NAME=North_Carolina
GUI=wxpython
DEBUG=1
(Sat Jun 14 08:05:43 2014) Command finished (0 sec)
(Sat Jun 14 08:02:00 2014)
g.region -p
D1/1: <PROJ_EPSG> file not found for location
<North_Carolina>
projection: 99 (Lambert Conformal Conic)
zone: 0
datum: nad83
ellipsoid: a=6378137 es=0.006694380022900787
north: 221230
south: 219580
west: 637740
east: 639530
nsres: 10
ewres: 10
rows: 165
cols: 179
cells: 29535
(Sat Jun 14 08:02:00 2014) Command finished (0 sec)
(Sat Jun 14 08:02:09 2014)
g.region rast=lsat5_1987_10@landsat
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
(Sat Jun 14 08:02:10 2014) Command finished (0 sec)
(Sat Jun 14 08:02:18 2014)
g.region -p
D1/1: <PROJ_EPSG> file not found for location
<North_Carolina>
projection: 99 (Lambert Conformal Conic)
zone: 0
datum: nad83
ellipsoid: a=6378137 es=0.006694380022900787
north: 228513
south: 214975.5
west: 629992.5
east: 645012
nsres: 28.5
ewres: 28.5
rows: 475
cols: 527
cells: 250325
(Sat Jun 14 08:02:18 2014) Command finished (0 sec)
(Sat Jun 14 08:03:10 2014)
r.diversity --o input=lsat5_1987_10@landsat prefix=lsat5_1987_div_ alpha=0.5 size=5
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: grass.script.core.start_command(): r.li.simpson --o
--q input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster(): name=MASK mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat5_1987_div__simpson_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat5_1987_div__simpson_size_5.0 mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__simpson_size_5.0 mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__simpson_size_5.0 mapset=PERMANENT
D1/1: grass.script.core.start_command(): r.li.shannon --o
--q input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__shannon_size_5.0
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster(): name=MASK mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat5_1987_div__shannon_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat5_1987_div__shannon_size_5.0 mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__shannon_size_5.0 mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__shannon_size_5.0 mapset=PERMANENT
D1/1: grass.script.core.start_command(): r.li.pielou --o --q
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__pielou_size_5.0
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster(): name=MASK mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat5_1987_div__pielou_size_5.0 compressed
D1/1: G_find_raster2(): name=lsat5_1987_div__pielou_size_5.0
mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_div__pielou_size_5.0
mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_div__pielou_size_5.0
mapset=PERMANENT
D1/1: grass.script.core.start_command(): r.li.renyi --o --q
output=lsat5_1987_div__renyi_size_5.0_alpha_0.5
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
alpha=0.5
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster2(): name=lsat5_1987_10 mapset=landsat
D1/1: G_find_raster(): name=MASK mapset=PERMANENT
D1/1: G_find_raster2(): name=lsat5_1987_10@landsat mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat5_1987_div__renyi_size_5.0_alpha_0.5
compressed
D1/1: G_find_raster2():
name=lsat5_1987_div__renyi_size_5.0_alpha_0.5
mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__renyi_size_5.0_alpha_0.5
mapset=PERMANENT
D1/1: G_find_raster2():
name=lsat5_1987_div__renyi_size_5.0_alpha_0.5
mapset=PERMANENT
D1/1: grass.script.core.start_command(): g.message
message=Done.
Done.
(Sat Jun 14 08:03:23 2014) Command finished (12 sec)
(Sat Jun 14 08:03:59 2014)
g.gisenv set=DEBUG=0
(Sat Jun 14 08:03:59 2014) Command finished (0 sec)

#XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# sub2Kruger #
g.gisenv set=DEBUG=1
g.gisenv -n
g.region -p
g.region rast=lsat7_1993s1@tSub2
g.region -p
r.diversity --o input= lsat7_1993s1@tSub2 prefix=lsat7_1993s1_div_ alpha=0.5 size=5
r.in.gdal input=D:\lndsr.LE71999353-band1_utm_clip2.tif output=lsat7_1993s2 -e
g.region -p
r.diversity --o input=lsat7_1993s2@tSub2 prefix=lsat7_1993s2_div_ alpha=0.5 size=5
g.gisenv set=DEBUG=0

#--------------------------------------------------
g.gisenv set=DEBUG=1
(Sat Jun 14 08:18:48 2014) Command finished (0 sec)
(Sat Jun 14 08:18:58 2014)
g.gisenv -n
MAPSET=tSub2
GISDBASE=C:\Users\Sands\Documents\grassdata\
LOCATION_NAME=sub2Kruger
GUI=wxpython
DEBUG=1
(Sat Jun 14 08:18:58 2014) Command finished (0 sec)
(Sat Jun 14 08:19:25 2014)
g.proj -p
-PROJ_INFO-------------------------------------------------
name : Universal Transverse Mercator
proj : utm
datum : wgs84
ellps : wgs84
zone : 36
south : defined
no_defs : defined
towgs84 : 0.000,0.000,0.000
-PROJ_UNITS------------------------------------------------
unit : meter
units : meters
meters : 1
(Sat Jun 14 08:19:25 2014) Command finished (0 sec)
(Sat Jun 14 08:20:18 2014)
g.region -p
D1/1: <PROJ_EPSG> file not found for location <sub2Kruger>
projection: 1 (UTM)
zone: -36
datum: wgs84
ellipsoid: wgs84
north: 7270285.00009493
south: 7207645.00009493
west: 342134.99995605
east: 439064.99995605
nsres: 30
ewres: 30
rows: 2088
cols: 3231
cells: 6746328
(Sat Jun 14 08:20:18 2014) Command finished (0 sec)
(Sat Jun 14 08:21:36 2014)
r.diversity input=lsat7_1993s1@tSub2 prefix=lsat7_1993s1_div_ alpha=0.5 size=5
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: grass.script.core.start_command(): r.li.simpson --q
input=lsat7_1993s1@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s1_div__simpson_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s1_div__simpson_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s1_div__simpson_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__simpson_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__simpson_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.shannon --q
input=lsat7_1993s1@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s1_div__shannon_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s1_div__shannon_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s1_div__shannon_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__shannon_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__shannon_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.pielou --q
input=lsat7_1993s1@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s1_div__pielou_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s1_div__pielou_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s1_div__pielou_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__pielou_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__pielou_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.renyi --q
output=lsat7_1993s1_div__renyi_size_5.0_alpha_0.5
input=lsat7_1993s1@tSub2 conf=conf_diversity_5.0 alpha=0.5
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s1@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s1_div__renyi_size_5.0_alpha_0.5
compressed
D1/1: G_find_raster2():
name=lsat7_1993s1_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s1_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: grass.script.core.start_command(): g.message
message=Done.
Done.
(Sat Jun 14 08:22:43 2014) Command finished (1 min 6 sec)
(Sat Jun 14 08:38:34 2014)
r.in.gdal input=D:\lndsr.LE71999353-band1_utm_clip2.tif output=lsat7_1993s2 -e
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: Creating support files for lsat7_1993s2
D1/1: close lsat7_1993s2 compressed
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
Raster map <lsat7_1993s2> created.
Region for the current mapset updated
r.in.gdal complete.
(Sat Jun 14 08:38:35 2014) Command finished (1 sec)
(Sat Jun 14 08:41:03 2014)
g.region -p
D1/1: <PROJ_EPSG> file not found for location <sub2Kruger>
projection: 1 (UTM)
zone: -36
datum: wgs84
ellipsoid: wgs84
north: 7250125.00009493
south: 7213705.00009493
west: 354434.99995605
east: 394784.99995605
nsres: 30
ewres: 30
rows: 1214
cols: 1345
cells: 1632830
(Sat Jun 14 08:41:04 2014) Command finished (0 sec)
(Sat Jun 14 08:21:36 2014)
r.diversity input=lsat7_1993s2@tSub2 prefix=lsat7_1993s2_div_ alpha=0.5 size=5
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: grass.script.core.start_command(): r.li.simpson --q
input=lsat7_1993s2@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s2_div__simpson_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s2_div__simpson_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s2_div__simpson_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__simpson_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__simpson_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.shannon --q
input=lsat7_1993s2@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s2_div__shannon_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s2_div__shannon_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s2_div__shannon_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__shannon_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__shannon_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.pielou --q
input=lsat7_1993s2@tSub2 conf=conf_diversity_5.0
output=lsat7_1993s2_div__pielou_size_5.0
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s2_div__pielou_size_5.0 compressed
D1/1: G_find_raster2():
name=lsat7_1993s2_div__pielou_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__pielou_size_5.0 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__pielou_size_5.0 mapset=tSub2
D1/1: grass.script.core.start_command(): r.li.renyi --q
output=lsat7_1993s2_div__renyi_size_5.0_alpha_0.5
input=lsat7_1993s2@tSub2 conf=conf_diversity_5.0 alpha=0.5
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2 mapset=tSub2
D1/1: G_find_raster(): name=MASK mapset=tSub2
D1/1: G_find_raster2(): name=lsat7_1993s2@tSub2 mapset=
D1/1: r.li.daemon pathSetup: [C:\Users\Sands\AppData\Roa
ming\GRASS7\r.li\conf_diversity_5.0]
D1/1: close lsat7_1993s2_div__renyi_size_5.0_alpha_0.5
compressed
D1/1: G_find_raster2():
name=lsat7_1993s2_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: G_find_raster2():
name=lsat7_1993s2_div__renyi_size_5.0_alpha_0.5 mapset=tSub2
D1/1: grass.script.core.start_command(): g.message
message=Done.
Done.
(Sat Jun 14 08:22:43 2014) Command finished (1 min 6 sec)

neteler · June 20, 2014, 5:49am

Dear Sandra,

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

Dear GRASS Users,

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through R (R version 3.0.2 win32) on Windows 7 64bit.

Testing from GRASS and North_Carolina dataset's lsat5_1987_10 raster (cells=29535), [r.diversity --o input=lsat5_1987_10@landsat prefix=lsat5_1987_div_ alpha=0.5 size=5] reports as Done with the different diversity rasters added to the location fine (see # North_Carolina# code and results below).

Yes, same here on Linux. So far so nice.

BTW: I have changed r.diversity in SVN and removed the quiet flags to
print the output map names rather than only "Done".

However, when running the same code on a larger image (cells=6746328) from my own location, although it reports Done, no rasters are created. If I subset the image (cells=1632830) and run it again its works (see # sub2Kruger # code and results below). So I'm guessing it is a memory issue?

ok I simply simulate higher resolution:

g.region -p rast=lsat5_1987_10 res=5 -a
...
cells: 8137540

and then execute the r.diversity call again. This then calls

r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0

and it consumes a lot of memory... will check on a bigger machine,
perhaps a memory leak.

Markus

neteler · July 29, 2014, 1:19pm

Hi Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through R (R version 3.0.2 win32) on Windows 7 64bit.

...

However, when running the same code on a larger image (cells=6746328) from my own location, although it reports Done, no rasters are created. If I subset the image (cells=1632830) and run it again its works (see # sub2Kruger # code and results below). So I'm guessing it is a memory issue?

I tried again, directly executing the r.li.simpson command and one of
those called by r.dversity:

# NC dataset
g.region -p rast=lsat5_1987_10@landsat -a

r.li.simpson --o input=lsat5_1987_10@landsat \
conf=conf_diversity_5.0 \
output=lsat5_1987_div__simpson_size_5.0

It consumes (too much) memory.

Debugging it a bit. valgrind suggests that something leaking here:

# raster/r.li/r.li.daemon/avl.c, line 204
long avl_to_array(avl_node * root, long i, AVL_table * a)
{

    if (root != NULL) {
        i = avl_to_array(root->left_child, i, a);
        if (a == NULL)
            G_fatal_error("avl, avl_to_array: null value");
        else {
            a[i] = G_malloc(sizeof(AVL_tableRow));
            a[i]->k = root->key;
            a[i]->tot = root->counter;
            i++;
            i = avl_to_array(root->right_child, i, a);
        }
    }
    return i;
}

Memory is allocated in every call of avl_to_array() but perhaps not
properly freed later?
Any developer having an idea?

Markus

(attachments)

conf_diversity_5.0 (85 Bytes)

neteler · September 6, 2014, 3:15am

Dear Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through R
(R version 3.0.2 win32) on Windows 7 64bit.

...

However, when running the same code on a larger image (cells=6746328) from my
own location, although it reports Done, no rasters are created. If I subset the image
(cells=1632830) and run it again its works (see # sub2Kruger # code and results below).
So I'm guessing it is a memory issue?

...

... it consumes a lot of memory... will check on a bigger machine,
perhaps a memory leak.

The assumption turned out to be right and I think we got it today!

Vaclav Petras checked it and discovered an "unfortunate" memory
allocation which he fixed in r.li.* in
revision http://trac.osgeo.org/grass/changeset/61812 ("r.li: fix
memory handling (memory leak in avl_to_array function))".

Now r.li has become very fast, on my laptop:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10 res=10 -a
...
rows: 1355
cols: 1503
cells: 2036565

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 29.32 seconds

or with a simulated higher resolution:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=5 -aprojection: 99 (Lambert Conformal Conic)
...
rows: 2708
cols: 3005
cells: 8137540

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 227.37 seconds (used to be > 2 hours)

So, to grab this improvement for Windows, grab the version from here:
http://wingrass.fsv.cvut.cz/grass71/

(or via OSGeo4W installer). Be sure that the revision is at least
r61812 which is indicated in the file name.

Please let us know if all works to avoid that the change has any
negative impact.
Tests here did not show any changes in the output except for the speed
improvement and solved memory leak.

Subsequently also r.diversity should behave now.

I'll backport it to GRASS 7.0 release branch after some testing.

Markus