[GRASS-user] Solved: execGRASS("r.diversity") Done. No rasters created (Large rasters)

Dear Markus,

Excellent! Thanks to everyone for their help.
I tested r.diversity in the new revision (61840) on a raster with 12933032 cells and it took under 3 minutes to complete r.li.simpson, r.li.shannon, r.li.pielou and r.li.renyi outputs.
Amazing :slight_smile:

execGRASS("r.diversity",flags="overwrite", parameters=list(input="flow", prefix="flow_div", alpha=0.5, size=3))
r.li.simpson complete. Raster map <flow_div_simpson_size_3.0> created.
r.li.shannon complete. Raster map <flow_div_shannon_size_3.0> created.
r.li.pielou complete. Raster map <flow_div_pielou_size_3.0> created.
r.li.renyi complete. Raster map <flow_div_renyi_size_3.0_alpha_0.5>
created.
Done.

Thanks again and keep well.

Cheers
Sandra

-----Original Message-----
From: neteler.osgeo@gmail.com [mailto:neteler.osgeo@gmail.com] On Behalf Of Markus Neteler
Sent: 06 September 2014 05:15 AM
To: Sandra MacFadyen
Cc: GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] execGRASS("r.diversity") Done. No rasters created (Large rasters)

Dear Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through
R (R version 3.0.2 win32) on Windows 7 64bit.

...

However, when running the same code on a larger image (cells=6746328)
from my own location, although it reports Done, no rasters are
created. If I subset the image
(cells=1632830) and run it again its works (see # sub2Kruger # code and results below).
So I'm guessing it is a memory issue?

...

... it consumes a lot of memory... will check on a bigger machine,
perhaps a memory leak.

The assumption turned out to be right and I think we got it today!

Vaclav Petras checked it and discovered an "unfortunate" memory allocation which he fixed in r.li.* in revision http://trac.osgeo.org/grass/changeset/61812 ("r.li: fix memory handling (memory leak in avl_to_array function))".

Now r.li has become very fast, on my laptop:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10 res=10 -a ...
rows: 1355
cols: 1503
cells: 2036565

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 29.32 seconds

or with a simulated higher resolution:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=5 -aprojection: 99 (Lambert Conformal Conic) ...
rows: 2708
cols: 3005
cells: 8137540

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 227.37 seconds (used to be > 2 hours)

So, to grab this improvement for Windows, grab the version from here:
http://wingrass.fsv.cvut.cz/grass71/

(or via OSGeo4W installer). Be sure that the revision is at least
r61812 which is indicated in the file name.

Please let us know if all works to avoid that the change has any negative impact.
Tests here did not show any changes in the output except for the speed improvement and solved memory leak.

Subsequently also r.diversity should behave now.

I'll backport it to GRASS 7.0 release branch after some testing.

Markus

On Wed, Sep 17, 2014 at 7:40 PM, Sandra MacFadyen <sandramf@live.co.za>
wrote:

Dear Markus,

Excellent! Thanks to everyone for their help.
I tested r.diversity in the new revision (61840) on a raster with 12933032
cells and it took under 3 minutes to complete r.li.simpson, r.li.shannon,
r.li.pielou and r.li.renyi outputs.
Amazing :slight_smile:

> execGRASS("r.diversity",flags="overwrite", parameters=list(input="flow",
prefix="flow_div", alpha=0.5, size=3))
> r.li.simpson complete. Raster map <flow_div_simpson_size_3.0> created.
> r.li.shannon complete. Raster map <flow_div_shannon_size_3.0> created.
> r.li.pielou complete. Raster map <flow_div_pielou_size_3.0> created.
> r.li.renyi complete. Raster map <flow_div_renyi_size_3.0_alpha_0.5>
> created.
> Done.

Thanks again and keep well.

Good to hear that. Can you say if the results are correct? I computed

difference of one map before and after and it was OK. But it would be great
to have more tests. Let's start with: does the result make sense?

Vaclav

Cheers
Sandra

-----Original Message-----
From: neteler.osgeo@gmail.com [mailto:neteler.osgeo@gmail.com] On Behalf
Of Markus Neteler
Sent: 06 September 2014 05:15 AM
To: Sandra MacFadyen
Cc: GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] execGRASS("r.diversity") Done. No rasters
created (Large rasters)

Dear Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:
> On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za>
wrote:
>> I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through
>> R (R version 3.0.2 win32) on Windows 7 64bit.
...
>> However, when running the same code on a larger image (cells=6746328)
>> from my own location, although it reports Done, no rasters are
>> created. If I subset the image
>> (cells=1632830) and run it again its works (see # sub2Kruger # code and
results below).
>> So I'm guessing it is a memory issue?
...
> ... it consumes a lot of memory... will check on a bigger machine,
> perhaps a memory leak.

The assumption turned out to be right and I think we got it today!

Vaclav Petras checked it and discovered an "unfortunate" memory allocation
which he fixed in r.li.* in revision
http://trac.osgeo.org/grass/changeset/61812 ("r.li: fix memory handling
(memory leak in avl_to_array function))".

Now r.li has become very fast, on my laptop:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=10 -a ...
rows: 1355
cols: 1503
cells: 2036565

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 29.32 seconds

or with a simulated higher resolution:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=5 -aprojection: 99 (Lambert Conformal Conic) ...
rows: 2708
cols: 3005
cells: 8137540

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o
input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
--> 227.37 seconds (used to be > 2 hours)

So, to grab this improvement for Windows, grab the version from here:
http://wingrass.fsv.cvut.cz/grass71/

(or via OSGeo4W installer). Be sure that the revision is at least
r61812 which is indicated in the file name.

Please let us know if all works to avoid that the change has any negative
impact.
Tests here did not show any changes in the output except for the speed
improvement and solved memory leak.

Subsequently also r.diversity should behave now.

I'll backport it to GRASS 7.0 release branch after some testing.

Markus

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Dear Vaclav,

I did a quick manual test to see if the results were correct for a few 5x5 windows and yes, everything appears 100%:

Example data from one 5x5 window

test1 = c(572,617,661,638,616,

662,617,593,638,616,

662,639,572,661,616,

662,662,639,662,616,

594,572,594,639,593)

Calculate info needed for diversity indices

freq1 = as.data.frame(table(test1)) # Calculate frequency table

S = length(unique(test1)) # Total number of species (S)

N = length(test1) # Total number of individuals (N)

ln.S = log(S) #Natural log of species (ln S)

ln.N = log(N) # Natural log of individuals (ln N)

Pi = freq1$Freq/N # Proportion of individuals that belong to each species

Pi2 = (Pi ^ 2)

ln.pi = log(Pi)

pi.ln.pi = Pi * ln.pi

pi.ln.pi2 = (ln.pi^2)*Pi

Calculate different diversity indices

(M = (S-1)/ln.N) # Margalef’s index (M)

(i.Di = 1/sum(Pi2)) # Simpson’s index [Inverse] (i.Di)

(i.Dc = 1 - (sum(Pi2))) # Simpson’s index [Complement] (i.Dc) * This one calculated with r.diversity

(H = sum(pi.ln.pi)*-1) # Shannon-Wiener index (H’)

(J = H/ln.S) # Pielou’s index (J)

Thanks again

Cheers

Sandra

From: Vaclav Petras [mailto:wenzeslaus@gmail.com]
Sent: 18 September 2014 02:46 AM
To: Sandra MacFadyen
Cc: Markus Neteler; GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] Solved: execGRASS(“r.diversity”) Done. No rasters created (Large rasters)

On Wed, Sep 17, 2014 at 7:40 PM, Sandra MacFadyen <sandramf@live.co.za> wrote:

Dear Markus,

Excellent! Thanks to everyone for their help.
I tested r.diversity in the new revision (61840) on a raster with 12933032 cells and it took under 3 minutes to complete r.li.simpson, r.li.shannon, r.li.pielou and r.li.renyi outputs.
Amazing :slight_smile:

execGRASS(“r.diversity”,flags=“overwrite”, parameters=list(input=“flow”, prefix=“flow_div”, alpha=0.5, size=3))
r.li.simpson complete. Raster map <flow_div_simpson_size_3.0> created.
r.li.shannon complete. Raster map <flow_div_shannon_size_3.0> created.
r.li.pielou complete. Raster map <flow_div_pielou_size_3.0> created.
r.li.renyi complete. Raster map <flow_div_renyi_size_3.0_alpha_0.5>
created.
Done.

Thanks again and keep well.

Good to hear that. Can you say if the results are correct? I computed difference of one map before and after and it was OK. But it would be great to have more tests. Let’s start with: does the result make sense?

Vaclav

Cheers
Sandra

-----Original Message-----
From: neteler.osgeo@gmail.com [mailto:neteler.osgeo@gmail.com] On Behalf Of Markus Neteler
Sent: 06 September 2014 05:15 AM
To: Sandra MacFadyen
Cc: GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] execGRASS(“r.diversity”) Done. No rasters created (Large rasters)

Dear Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through
R (R version 3.0.2 win32) on Windows 7 64bit.
…
However, when running the same code on a larger image (cells=6746328)
from my own location, although it reports Done, no rasters are
created. If I subset the image
(cells=1632830) and run it again its works (see # sub2Kruger # code and results below).
So I’m guessing it is a memory issue?
…
… it consumes a lot of memory… will check on a bigger machine,
perhaps a memory leak.

The assumption turned out to be right and I think we got it today!

Vaclav Petras checked it and discovered an “unfortunate” memory allocation which he fixed in r.li.* in revision http://trac.osgeo.org/grass/changeset/61812 (“r.li: fix memory handling (memory leak in avl_to_array function))”.

Now r.li has become very fast, on my laptop:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10 res=10 -a …
rows: 1355
cols: 1503
cells: 2036565

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
→ 29.32 seconds

or with a simulated higher resolution:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=5 -aprojection: 99 (Lambert Conformal Conic) …
rows: 2708
cols: 3005
cells: 8137540

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
→ 227.37 seconds (used to be > 2 hours)

So, to grab this improvement for Windows, grab the version from here:
http://wingrass.fsv.cvut.cz/grass71/

(or via OSGeo4W installer). Be sure that the revision is at least
r61812 which is indicated in the file name.

Please let us know if all works to avoid that the change has any negative impact.
Tests here did not show any changes in the output except for the speed improvement and solved memory leak.

Subsequently also r.diversity should behave now.

I’ll backport it to GRASS 7.0 release branch after some testing.

Markus


grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

This is great Sandra. Now I am curious to see final ecological relationships you are searching for!
D

···

2014-09-30 3:24 GMT+02:00 Sandra MacFadyen <sandramf@live.co.za>:

Dear Vaclav,

I did a quick manual test to see if the results were correct for a few 5x5 windows and yes, everything appears 100%:

Example data from one 5x5 window

test1 = c(572,617,661,638,616,

662,617,593,638,616,

662,639,572,661,616,

662,662,639,662,616,

594,572,594,639,593)

Calculate info needed for diversity indices

freq1 = as.data.frame(table(test1)) # Calculate frequency table

S = length(unique(test1)) # Total number of species (S)

N = length(test1) # Total number of individuals (N)

ln.S = log(S) #Natural log of species (ln S)

ln.N = log(N) # Natural log of individuals (ln N)

Pi = freq1$Freq/N # Proportion of individuals that belong to each species

Pi2 = (Pi ^ 2)

ln.pi = log(Pi)

pi.ln.pi = Pi * ln.pi

pi.ln.pi2 = (ln.pi^2)*Pi

Calculate different diversity indices

(M = (S-1)/ln.N) # Margalef’s index (M)

(i.Di = 1/sum(Pi2)) # Simpson’s index [Inverse] (i.Di)

(i.Dc = 1 - (sum(Pi2))) # Simpson’s index [Complement] (i.Dc) * This one calculated with r.diversity

(H = sum(pi.ln.pi)*-1) # Shannon-Wiener index (H’)

(J = H/ln.S) # Pielou’s index (J)

Thanks again

Cheers

Sandra

From: Vaclav Petras [mailto:wenzeslaus@gmail.com]
Sent: 18 September 2014 02:46 AM
To: Sandra MacFadyen
Cc: Markus Neteler; GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] Solved: execGRASS(“r.diversity”) Done. No rasters created (Large rasters)

On Wed, Sep 17, 2014 at 7:40 PM, Sandra MacFadyen <sandramf@live.co.za> wrote:

Dear Markus,

Excellent! Thanks to everyone for their help.
I tested r.diversity in the new revision (61840) on a raster with 12933032 cells and it took under 3 minutes to complete r.li.simpson, r.li.shannon, r.li.pielou and r.li.renyi outputs.
Amazing :slight_smile:

execGRASS(“r.diversity”,flags=“overwrite”, parameters=list(input=“flow”, prefix=“flow_div”, alpha=0.5, size=3))
r.li.simpson complete. Raster map <flow_div_simpson_size_3.0> created.
r.li.shannon complete. Raster map <flow_div_shannon_size_3.0> created.
r.li.pielou complete. Raster map <flow_div_pielou_size_3.0> created.
r.li.renyi complete. Raster map <flow_div_renyi_size_3.0_alpha_0.5>
created.
Done.

Thanks again and keep well.

Good to hear that. Can you say if the results are correct? I computed difference of one map before and after and it was OK. But it would be great to have more tests. Let’s start with: does the result make sense?

Vaclav

Cheers
Sandra

-----Original Message-----
From: neteler.osgeo@gmail.com [mailto:neteler.osgeo@gmail.com] On Behalf Of Markus Neteler
Sent: 06 September 2014 05:15 AM
To: Sandra MacFadyen
Cc: GRASS user list; Duccio Rocchini; Luca Delucchi
Subject: Re: [GRASS-user] execGRASS(“r.diversity”) Done. No rasters created (Large rasters)

Dear Sandra,

On Fri, Jun 20, 2014 at 7:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Sat, Jun 14, 2014 at 9:21 AM, Sandra MacFadyen <sandramf@live.co.za> wrote:

I am using r.diversity (GRASS GIS 7.0.0svn build 60785 win32) through
R (R version 3.0.2 win32) on Windows 7 64bit.
…
However, when running the same code on a larger image (cells=6746328)
from my own location, although it reports Done, no rasters are
created. If I subset the image
(cells=1632830) and run it again its works (see # sub2Kruger # code and results below).
So I’m guessing it is a memory issue?
…
… it consumes a lot of memory… will check on a bigger machine,
perhaps a memory leak.

The assumption turned out to be right and I think we got it today!

Vaclav Petras checked it and discovered an “unfortunate” memory allocation which he fixed in r.li.* in revision http://trac.osgeo.org/grass/changeset/61812 (“r.li: fix memory handling (memory leak in avl_to_array function))”.

Now r.li has become very fast, on my laptop:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10 res=10 -a …
rows: 1355
cols: 1503
cells: 2036565

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
→ 29.32 seconds

or with a simulated higher resolution:

GRASS 7.1.svn (nc_spm_08_grass7):~ > g.region -p rast=lsat5_1987_10
res=5 -aprojection: 99 (Lambert Conformal Conic) …
rows: 2708
cols: 3005
cells: 8137540

GRASS 7.1.svn (nc_spm_08_grass7):~ > time -p r.li.simpson --o input=lsat5_1987_10@landsat conf=conf_diversity_5.0
output=lsat5_1987_div__simpson_size_5.0
r.li.simpson complete. Raster map <lsat5_1987_div__simpson_size_5.0>
created.
→ 227.37 seconds (used to be > 2 hours)

So, to grab this improvement for Windows, grab the version from here:
http://wingrass.fsv.cvut.cz/grass71/

(or via OSGeo4W installer). Be sure that the revision is at least
r61812 which is indicated in the file name.

Please let us know if all works to avoid that the change has any negative impact.
Tests here did not show any changes in the output except for the speed improvement and solved memory leak.

Subsequently also r.diversity should behave now.

I’ll backport it to GRASS 7.0 release branch after some testing.

Markus


grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

–

Duccio Rocchini, PhD

http://gis.cri.fmach.it/rocchini/

Fondazione Edmund Mach
Research and Innovation Centre
Department of Biodiversity and Molecular Ecology
GIS and Remote Sensing Unit
Via Mach 1, 38010 San Michele all’Adige (TN) - Italy
Phone +39 0461 615 570
ducciorocchini@gmail.com
duccio.rocchini@fmach.it
skype: duccio.rocchini
Google Scholar Profile: scholar.google.it/citations?hl=it&user=OJtw7agAAAAJ