[GRASS-user] Calculating standard error of many maps?

GRASS GRASS user
1

Hi

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

Is there an easy way of doing this in GRASS?

Thanks,

Rainer

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa

2

Just one clarification: I would like to calculate these descriptive
stats for each cell, to obtain "variability maps".

Rainer

On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:

Hi

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

Is there an easy way of doing this in GRASS?

Thanks,

Rainer

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa

3

hello Rainer,

just look at r.series.

If the naming of the files is similar you can try something like:

r.series input="`g.mlist pattern='insitu_data.*' sep=,`"
output=insitu_data.stddev method=stddev

regards, Martin

On Montag, 9. Februar 2009 16:59:33 Rainer M Krug wrote:

Just one clarification: I would like to calculate these descriptive
stats for each cell, to obtain "variability maps".

Rainer

On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:
> Hi
>
> I have 25000 maps, generated by simulation predictions, and I would
> like to calculate some descriptive stats, like mean, standard
> deviation, median, quartiles.
>
> Is there an easy way of doing this in GRASS?
>
> Thanks,
>
> Rainer
>
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Faculty of Science
> Natural Sciences Building
> Private Bag X1
> University of Stellenbosch
> Matieland 7602
> South Africa

--
**********************************************************************

University of Wuerzburg
Institute of Geography
Department of Remote Sensing
Remote Sensing and Biodiversity Unit
Am Hubland
97074 Wuerzburg, Germany
@
German Aerospace Center (DLR)
German Remote Sensing Data Center (DFD)

Phone: +49-(0)931-888-4797
Fax: +49-(0)931-888-4961
Email: martin.wegmann@uni-wuerzburg.de
url: http://www.remote-sensing.uni-wuerzburg.de

**********************************************************************

4

Hi Martin

On Mon, Feb 9, 2009 at 6:12 PM, Martin Wegmann
<martin.wegmann@uni-wuerzburg.de> wrote:

hello Rainer,

just look at r.series.

That is exactly what I was looking for.

Thanks a lot

Rainer

If the naming of the files is similar you can try something like:

r.series input="`g.mlist pattern='insitu_data.*' sep=,`"
output=insitu_data.stddev method=stddev

regards, Martin

On Montag, 9. Februar 2009 16:59:33 Rainer M Krug wrote:

Just one clarification: I would like to calculate these descriptive
stats for each cell, to obtain "variability maps".

Rainer

On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:
> Hi
>
> I have 25000 maps, generated by simulation predictions, and I would
> like to calculate some descriptive stats, like mean, standard
> deviation, median, quartiles.
>
> Is there an easy way of doing this in GRASS?
>
> Thanks,
>
> Rainer
>
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Faculty of Science
> Natural Sciences Building
> Private Bag X1
> University of Stellenbosch
> Matieland 7602
> South Africa

--
**********************************************************************

University of Wuerzburg
Institute of Geography
Department of Remote Sensing
Remote Sensing and Biodiversity Unit
Am Hubland
97074 Wuerzburg, Germany
@
German Aerospace Center (DLR)
German Remote Sensing Data Center (DFD)

Phone: +49-(0)931-888-4797
Fax: +49-(0)931-888-4961
Email: martin.wegmann@uni-wuerzburg.de
url: http://www.remote-sensing.uni-wuerzburg.de

**********************************************************************

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa

5

On 09/02/09 16:59, Rainer M Krug wrote:

Just one clarification: I would like to calculate these descriptive
stats for each cell, to obtain "variability maps".

Rainer

On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:

Hi

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

Is there an easy way of doing this in GRASS?

r.series ?

You will probably have to use xargs to collate that many map names...

Moritz

6

On Feb 9, 2009, at 8:15 AM, Moritz Lennert wrote:

On 09/02/09 16:59, Rainer M Krug wrote:

Just one clarification: I would like to calculate these descriptive

stats for each cell, to obtain “variability maps”.

Rainer

On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:

Hi

I have 25000 maps, generated by simulation predictions, and I would

like to calculate some descriptive stats, like mean, standard

deviation, median, quartiles.

Is there an easy way of doing this in GRASS?

r.series ?

You will probably have to use xargs to collate that many map names…

Or you can use g.mlist if the names have a common prefix, suffix, etc. This assumes you want a single map that displays the result of all the input maps.

E.g.

r.series input=“g.mlist rast pattern='prefix_*' sep=,” output=prefix_stddev method=stddev

John

7

On 09/02/09 17:19, John C. Tull wrote:

On Feb 9, 2009, at 8:15 AM, Moritz Lennert wrote:

On 09/02/09 16:59, Rainer M Krug wrote:

Just one clarification: I would like to calculate these descriptive
stats for each cell, to obtain "variability maps".
Rainer
On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com <mailto:r.m.krug@gmail.com>> wrote:

Hi

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

Is there an easy way of doing this in GRASS?

r.series ?

You will probably have to use xargs to collate that many map names...

Or you can use g.mlist if the names have a common prefix, suffix, etc. This assumes you want a single map that displays the result of all the input maps.

E.g.

r.series input="`g.mlist rast pattern='prefix_*' sep=,`" output=prefix_stddev method=stddev

Yes, but if I'm not mistaken, this won't work with 25000 maps as it would create a too long command line. But please correct me if I'm wrong.

Moritz

8

Moritz Lennert wrote:

>>> Just one clarification: I would like to calculate these descriptive
>>> stats for each cell, to obtain "variability maps".
>>> Rainer
>>> On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com
>>> <mailto:r.m.krug@gmail.com>> wrote:
>>>> Hi
>>>>
>>>> I have 25000 maps, generated by simulation predictions, and I would
>>>> like to calculate some descriptive stats, like mean, standard
>>>> deviation, median, quartiles.
>>>>
>>>> Is there an easy way of doing this in GRASS?
>>
>>
>> r.series ?
>>
>> You will probably have to use xargs to collate that many map names...
>
> Or you can use g.mlist if the names have a common prefix, suffix, etc.
> This assumes you want a single map that displays the result of all the
> input maps.
>
> E.g.
>
> r.series input="`g.mlist rast pattern='prefix_*' sep=,`"
> output=prefix_stddev method=stddev

Yes, but if I'm not mistaken, this won't work with 25000 maps as it
would create a too long command line.

25000 maps will probably also exceed the resource limit for the number of
open descriptors. It's common for user processes to be limited to 1024
descriptors; you need to modify /etc/security/limits.conf to increase the
limit.

--
Glynn Clements <glynn@gclements.plus.com>

9

On Tue, Feb 10, 2009 at 12:06 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Moritz Lennert wrote:

>>> Just one clarification: I would like to calculate these descriptive
>>> stats for each cell, to obtain "variability maps".
>>> Rainer
>>> On Mon, Feb 9, 2009 at 5:57 PM, Rainer M Krug <r.m.krug@gmail.com
>>> <mailto:r.m.krug@gmail.com>> wrote:
>>>> Hi
>>>>
>>>> I have 25000 maps, generated by simulation predictions, and I would
>>>> like to calculate some descriptive stats, like mean, standard
>>>> deviation, median, quartiles.
>>>>
>>>> Is there an easy way of doing this in GRASS?
>>
>>
>> r.series ?
>>
>> You will probably have to use xargs to collate that many map names...
>
> Or you can use g.mlist if the names have a common prefix, suffix, etc.
> This assumes you want a single map that displays the result of all the
> input maps.
>
> E.g.
>
> r.series input="`g.mlist rast pattern='prefix_*' sep=,`"
> output=prefix_stddev method=stddev

Yes, but if I'm not mistaken, this won't work with 25000 maps as it
would create a too long command line.

25000 maps will probably also exceed the resource limit for the number of
open descriptors. It's common for user processes to be limited to 1024
descriptors; you need to modify /etc/security/limits.conf to increase the
limit.

I asked the same question on the R mailing list, and got the following
promising response:
###
This is how can you can do it with the raster package

# install.packages("raster", repos="http://R-Forge.R-project.org")
require(raster)

# Try it for a few files first..
n <- 10

# create a list (or vector) of file names, e.g. :
fn <- list()
for (i in 1:n) { fn[i] <- paste('myfile', i, '.tif', sep='') }

# make a RasterStack
s <- stack(fn)

r1 <- mCalc(s, fun=mean)
r2 <- mCalc(s, fun=sd)

r can be plotted, coerced to sp objects, etc.
plot(r1)

# or saved to file
r1 <- setFilename(r1, 'cellmeans.tif')
r1 <- writeRaster(r1, format='GTiff')

####

I am trying it at the moment with 1002 maps and, apart from the fact
that it takes some time (which I am sure it would in GRASS as well),
it seems to be working.

I will be looking at R.series again later, when I run the simulations
again. But I assume that there will be serious problems with 25000
maps - I might have to do it with a sample of maps.

Rainer

--
Glynn Clements <glynn@gclements.plus.com>

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa

10

Rainer Krug wrote:

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

I am trying it at the moment with 1002 maps and, apart from the fact
that it takes some time (which I am sure it would in GRASS as well),
it seems to be working.

I will be looking at R.series again later, when I run the simulations
again. But I assume that there will be serious problems with 25000
maps - I might have to do it with a sample of maps.

Hi,

for the standard stats available from r.univar
n
minimum
maximum
range
mean
mean of absolute values
standard deviation
variance
variation coefficient
sum

(ie not median, quartiles) you can use r.series in a loop, eg 1000 maps
at a time, and then sum up the resulting 25 summary maps before doing a
little math to get the overall answers. (or lots of 100 maps x250, ...)

I think you would need to add a new sum_of_squares method to r.series
to calculate variance/stdev. see r.univar/stats.c print_stats()

Hamish

11

On Wed, Feb 11, 2009 at 4:06 AM, Hamish <hamish_b@yahoo.com> wrote:

Rainer Krug wrote:

I have 25000 maps, generated by simulation predictions, and I would
like to calculate some descriptive stats, like mean, standard
deviation, median, quartiles.

I am trying it at the moment with 1002 maps and, apart from the fact
that it takes some time (which I am sure it would in GRASS as well),
it seems to be working.

I will be looking at R.series again later, when I run the simulations
again. But I assume that there will be serious problems with 25000
maps - I might have to do it with a sample of maps.

Hi,

for the standard stats available from r.univar
n
minimum
maximum
range
mean
mean of absolute values
standard deviation
variance
variation coefficient
sum

(ie not median, quartiles) you can use r.series in a loop, eg 1000 maps
at a time, and then sum up the resulting 25 summary maps before doing a
little math to get the overall answers. (or lots of 100 maps x250, ...)

I think you would need to add a new sum_of_squares method to r.series
to calculate variance/stdev. see r.univar/stats.c print_stats()

Thanks a lot Hamish - I will look into that and see what can do. In
the meantime, it works in R, as described above.

Cheers

Rainer

Hamish

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Faculty of Science
Natural Sciences Building
Private Bag X1
University of Stellenbosch
Matieland 7602
South Africa