[GRASS-dev] big speedups for i.landsat.rgb

Hi,

I just committed some changes to i.landsat.rgb(.sh) in 6.5svn
which may be of interest.. For my test map what used to take
40 sec to run now completes in 6.8 sec.

* combine two repeated r.univar calls into a single run now
   that r.univar supports multiple percentile=s.
  -- this easily cuts the module run-time in half at little cost.

* (optionally) run r.univar on the three RGB bands in parallel.
   This uses 3x the RAM but completes in 1/3rd of the wall time.

Experimental, but seems to work fine. I've done a similar, but
more advanced, poor man's parallelization with r3.in.xyz in
grass6 addons, which seems to work well. I've been slowly coming
to the realization that the higher level the parallelization
happens at, the more efficient it will be. (at the cost of
narrowed influence)

If the approach is well received, i.oif(.sh) is also low-hanging
fruit, and we can also port the method over to python scripts in
grass7, as described here:
  http://lists.osgeo.org/pipermail/grass-user/2012-February/063627.html

I explored using r.univar's multiple-input map capability
to simplify the serial-mode version of the code, but it uses 3x
the RAM as a single map and actually runs slightly slower than
calling r.univar 3 times + using awk to combine the results.
I guess the large malloc() are to blame for the slight slowdown.
(one reason to keep serial mode around is for a low-RAM mode)

However-- this indicates that r.univar's multiple input map
capability is an easy place to parallelize the code, by processing
each map in its own thread. Not sure if pthreads or OpenMP would
be better for that; OpenMP includes some automatic management
code to only create as many new threads as you have cores (e.g.
if you try to process 20 maps at once it will only spawn 4
threads at a time on a quad-core CPU). Maybe pthreads is smart
like that too or just goes by what you hardcode or set with the
enviro variables?

Any other multi-band imagery modules which jump out as easy
pickings ...? (r.in.gdal when importing interleaved multibands?)

Hamish

Hamish wrote:

Experimental, but seems to work fine.

(only tested on Linux)

Hamish

Hamish wrote:

i.oif(.sh) is also low-hanging fruit, and we can also port the
method over to python scripts

Hi,

The r.univar part of i.oif is now parallelized in 6.5svn (.sh)
and trunk (.py) versions.

.sh: https://trac.osgeo.org/grass/changeset/52126
.py: https://trac.osgeo.org/grass/changeset/52128

I'm guessing we can borrow some ideas from xargs/gnu parallel to
make a new reusable python wrapper function to de-clutter the
python method's implementation.

i.landsat.rgb.py in trunk needs a more invasive change so I
haven't touched that yet.

Michael: would this work with for your new imagery module?
(is that slow to run or very processor bound?)

Hamish

On Jun 18, 2012, at 8:35 PM, Hamish wrote:

Hamish wrote:

i.oif(.sh) is also low-hanging fruit, and we can also port the
method over to python scripts

Hi,

The r.univar part of i.oif is now parallelized in 6.5svn (.sh)
and trunk (.py) versions.

.sh: Changeset 52126 – GRASS GIS
.py: Changeset 52128 – GRASS GIS

I'm guessing we can borrow some ideas from xargs/gnu parallel to
make a new reusable python wrapper function to de-clutter the
python method's implementation.

i.landsat.rgb.py in trunk needs a more invasive change so I
haven't touched that yet.

Michael: would this work with for your new imagery module?
(is that slow to run or very processor bound?)

Exactly what I was thinking. It requires processing 3 bands. Histogram equalization might also benefit. It runs pretty quickly on a few hundred K cells, but once you get up to the size of full landsats, it slows down.

Michael

Hamish

_____________________
C. Michael Barton
Visiting Scientist, Integrated Science Program
National Center for Atmospheric Research &
University Consortium for Atmospheric Research
303-497-2889 (voice)

Director, Center for Social Dynamics & Complexity
Professor of Anthropology, School of Human Evolution & Social Change
Arizona State University
www: http://www.public.asu.edu/~cmbarton, http://csdc.asu.edu