[GRASS-dev] Re: GRASS 7 status overview

GRASS GRASS developer
,
1

Markus,

This WIKI page is very very helpful (http://trac.osgeo.org/grass/wiki/Grass7/NewFeatures). There are a lot of enhancements to talk about.

I did a quick look at all the new modules and offer the following comments:

Should r.statistics simply be replaced with r.statistics2? No explanation of difference, for example, between kurtosis and kurtosis2.
It seems that r.statistics3 should be renamed to something more descriptive as it differs functionally from r.statistics

r.external.out and r.colors.out do not generate the GUI when run from the command line. r.colors.out generates a segmentation fault if run without arguments.

Running v.krige generates the following message: Loading packages, please wait...
Rpy2 not found. Please install it and re-run.

I thought this was not a *requirement* and that v.krige would run but that some methods simply would not be available. If it is a requirement, shouldn't we make Rpy2 a dependency?

There is a typo in the description for i.emissivity. It should be "sparse" not "spares"

The new image processing look very cool. It looks like some of them should be packaged together under a single menu header (like r.li). Which should be packaged together?

Some of the new modules relate to solar radiance. Should any of them go under the 'solar radiance and shadows' section in raster? (i.sunhours? i.albedo?). If so, should any be renamed from i.* to r.* for consistency? (or should r.sun and r.sunmask be renamed to i.sun and i.sunmask?)

i.vi: should the user be able to choose which kind of vegetation index is created? Or does the module just create them all?

Trying to run ximgview generates the following error: ERROR: Incompatible library version for module. You need to rebuild GRASS
      or untangle multiple installations.

Michael

On Feb 12, 2010, at 2:26 PM, Markus Neteler wrote:

On Fri, Feb 12, 2010 at 7:35 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Michael Barton wrote:

Anyway, is there a place that gives an overview of the underlying
changes or plans for GRASS 7 that I should mention?

I don't think so.

@Glynn: because it should not be done or there isn't?

An overview page I have compiled here:
http://trac.osgeo.org/grass/wiki/Grass7/NewFeatures

It is yet incomplete unfortunately.

Are there some
things that either of you have been doing that you could summarize the
highlights for me to mention? Is the plan still to reorganize the raster
file hierarchy to something more along the lines of the vector
hierarchy? Anything else?

Apart from the display changes, most of it falls under the heading of
"clean-up"; even if some of it has been quite radical, e.g. splitting
off the raster functions from lib/gis to lib/raster, ditching curses
support, etc.

(see above trac page)

For any substantial raster changes, the main issue is how to support
legacy formats. IMHO, the ideal solution would be to implement the
core GRASS raster I/O "natively" in GDAL (meaning: not depending upon
GRASS libraries), so that they can be accessed using r.external[.out].
GRASS itself would then only need to understand the new format.

This would be a great idea, also to get rid of 50% of the GRASS-GDAL
plugin issues.

- Plans: implement more openMP support for multi-core CPUs.
- GRASS 7 ideas collection - GRASS-Wiki

Some stuff I have presented here:
25 Years of GRASS GIS | PPT

Feel free to recylce material (there is a ODP download link somewhere
on that page).

Markus

2

Thanks much for the explanations. A few responses below.

On Feb 13, 2010, at 7:25 PM, Glynn Clements wrote:

Michael Barton wrote:

Should r.statistics simply be replaced with r.statistics2?

You need both r.statistics2 (accumulator-based statistics) and
r.statistics3 (quantiles) to get most of the functionality of
r.statistics.

Neither module can calculate the mode, although I'm not sure that's
meaningful for FP data (in the absence of quantisation in the original
data, you would expect each value to be unique).

No explanation of difference, for example, between kurtosis and kurtosis2.

The methods whose descriptions begin with "(2-pass)" use a two-pass
algorithm. The first pass calculates the mean, the second pass
calculates the aggregate from the deviations.

The single-pass algorithms can have very poor accuracy if the standard
deviation is small relative to the mean. You end up calculating:

  (mean^N + a) - (mean^N + b)

where mean^N swamps a and b, leading to excessive relative error in
the result.

It seems that r.statistics3 should be renamed to something more
descriptive as it differs functionally from r.statistics

It can be used as a substitute for r.statistics method=median, but can
calculate arbitrary quantiles. It's derived from r.quantile.

r.statistics relies upon r.stats, which doesn't work with FP data and
which performs poorly with large numbers of categories.

So it sounds like r.statistics2 and r.statistics3 are significant improvements over r.statistics.

r.external.out and r.colors.out do not generate the GUI when run from
the command line.

Both generate a GUI when run with --ui. r.external.out doesn't have
any required arguments (mainly due to -f), while r.colors.out has a
bug:

It is unclear how r.external.out works. Does it make a GDAL-compatible file appear as a GRASS raster in a mapset? If so, shouldn't this be r.external.in? (i.e., it makes an external raster usable by GRASS)

r.colors.out generates a segmentation fault if run
without arguments.

Right; map= should be a required option; fixed in r40984.

Thanks.

Trying to run ximgview generates the following error: ERROR:
Incompatible library version for module. You need to rebuild GRASS
     or untangle multiple installations.

Either it failed to compile and you're trying to run an old version
with a newer library, or the -L flags for wxWidgets caused it to pick
up an old version of the libraries.

I compiled after an svn up, followed by make distclean. So I don't know how I could be trying to run an old version. I don't know about the -L flags in wxWidgets. I'm using a Mac where wxWidgets is wrapped in wxPython.

Also, ximgview should be superseded by wximgview (except, wximgview
currently doesn't build on Windows, for the same reason as xganim and
the vdigit and nviz modules, i.e. trying to link against wxWidgets
crashes "ld").

OK wximgview works. I guess I misunderstood what this did. So it's to offer a way to view graphics created by d.* commands?

Michael

3

Michael Barton wrote:

>> r.external.out and r.colors.out do not generate the GUI when run from
>> the command line.
>
> Both generate a GUI when run with --ui. r.external.out doesn't have
> any required arguments (mainly due to -f), while r.colors.out has a
> bug:

It is unclear how r.external.out works. Does it make a GDAL-compatible
file appear as a GRASS raster in a mapset? If so, shouldn't this be
r.external.in? (i.e., it makes an external raster usable by GRASS)

r.external.out causes new raster maps to be created as files via GDAL.

Whereas r.external turns a specific file into a GRASS raster map,
r.external.out enables a mode whereby all new rasters are created as
files via GDAL.

It creates a key-value file named "GDAL" in the mapset directory
containing various parameters (output directory, file format, creation
parameters). If this file is present, Rast_open_*_new() and
Rast_put_*_row() go via GDAL.

> Also, ximgview should be superseded by wximgview (except, wximgview
> currently doesn't build on Windows, for the same reason as xganim and
> the vdigit and nviz modules, i.e. trying to link against wxWidgets
> crashes "ld").

OK wximgview works. I guess I misunderstood what this did. So it's to
offer a way to view graphics created by d.* commands?

Yes. Specifically, it continually refreshes the display from the
contents of the image file (which must be written using
GRASS_PNG_MAPPED=TRUE, so that the file is modified in place, never
truncated), providing a rough substitute for XDRIVER (i.e. run d.*
commands from the command line, observe the result in a window).

--
Glynn Clements <glynn@gclements.plus.com>

4

On Feb 14, 2010, at 11:20 AM, Glynn Clements wrote:

Michael Barton wrote:

r.external.out and r.colors.out do not generate the GUI when run from
the command line.

Both generate a GUI when run with --ui. r.external.out doesn't have
any required arguments (mainly due to -f), while r.colors.out has a
bug:

It is unclear how r.external.out works. Does it make a GDAL-compatible
file appear as a GRASS raster in a mapset? If so, shouldn't this be
r.external.in? (i.e., it makes an external raster usable by GRASS)

r.external.out causes new raster maps to be created as files via GDAL.

Whereas r.external turns a specific file into a GRASS raster map,
r.external.out enables a mode whereby all new rasters are created as
files via GDAL.

It creates a key-value file named "GDAL" in the mapset directory
containing various parameters (output directory, file format, creation
parameters). If this file is present, Rast_open_*_new() and
Rast_put_*_row() go via GDAL.

So the files can be opened and used subsequently as GRASS files even though they are created in a non-GRASS format (e.g., geotiff)?

Michael

5

Michael Barton wrote:

>> It is unclear how r.external.out works. Does it make a GDAL-compatible
>> file appear as a GRASS raster in a mapset? If so, shouldn't this be
>> r.external.in? (i.e., it makes an external raster usable by GRASS)
>
> r.external.out causes new raster maps to be created as files via GDAL.
>
> Whereas r.external turns a specific file into a GRASS raster map,
> r.external.out enables a mode whereby all new rasters are created as
> files via GDAL.
>
> It creates a key-value file named "GDAL" in the mapset directory
> containing various parameters (output directory, file format, creation
> parameters). If this file is present, Rast_open_*_new() and
> Rast_put_*_row() go via GDAL.

So the files can be opened and used subsequently as GRASS files even
though they are created in a non-GRASS format (e.g., geotiff)?

Yes. Files created this way are automatically linked as GRASS rasters
as if by r.external.

--
Glynn Clements <glynn@gclements.plus.com>

6

Michael wrote:

So the files can be opened and used subsequently as GRASS
files even though they are created in a non-GRASS format
(e.g., geotiff)?

thanks to the genius of the main r.external module.

Hamish

7

Hello,
i have created a simple graphical model of the r.external work-flow,
attached as PNG file. (for presentation purpose)
If this is correct, i can add this to the help-pages or Wiki?

Soeren

2010/2/15 Glynn Clements <glynn@gclements.plus.com>:

Michael Barton wrote:

>> It is unclear how r.external.out works. Does it make a GDAL-compatible
>> file appear as a GRASS raster in a mapset? If so, shouldn't this be
>> r.external.in? (i.e., it makes an external raster usable by GRASS)
>
> r.external.out causes new raster maps to be created as files via GDAL.
>
> Whereas r.external turns a specific file into a GRASS raster map,
> r.external.out enables a mode whereby all new rasters are created as
> files via GDAL.
>
> It creates a key-value file named "GDAL" in the mapset directory
> containing various parameters (output directory, file format, creation
> parameters). If this file is present, Rast_open_*_new() and
> Rast_put_*_row() go via GDAL.

So the files can be opened and used subsequently as GRASS files even
though they are created in a non-GRASS format (e.g., geotiff)?

Yes. Files created this way are automatically linked as GRASS rasters
as if by r.external.

--
Glynn Clements <glynn@gclements.plus.com>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

(attachments)

Direct_raster_file_access_grass.png

8

Michael:

> OK wximgview works. [...] So it's to
> offer a way to view graphics created by d.* commands?

Glynn:

Yes. Specifically, it continually refreshes the display from the
contents of the image file (which must be written using
GRASS_PNG_MAPPED=TRUE, so that the file is modified in place, never
truncated), providing a rough substitute for XDRIVER (i.e. run d.*
commands from the command line, observe the result in a window).

would it be possible to have it watch the file's timestamps and only
refresh if it sees that the file has been changed?

Besides the raw d.* on the command line use, a common use of a super-light
weight GUI for me is for running grass via tunneled ssh+X. At work this
is just to the number cruncher down the hall from my desktop P4, but if
I'm running a big job I might leave a GNU Screen session running overnight
and it is nice to be able to ssh from home, quickly check the status and
display results, and then launch the next run. With d.mon this works
beautifully. It's actually quite usable over our really crappy ADSL.
(PNG driver + `qiv -e file.ppm` then click or mouse-wheel to refresh
works really well. the wxgui is unusable over that connection, I
suspect the tcltk GUI could be usable)

With continual refresh I worry that ximgview and wximgview will not be
much use ssh'ed from the near-dialup connection speeds from home.

with percent=10 gkrellm is showing it using about 1.4Mbit down the hall
over our building's 100mbit ethernet, with percent=1 that falls away, but
the lag time from command to render is about 10 seconds.
A cpu limit set to 1% (integer only) probably is going to be too high
for dial up, but any less would create unacceptable lag times.
?

Hamish

9

Soeren Gebbert wrote:

i have created a simple graphical model of the r.external work-flow,
attached as PNG file. (for presentation purpose)
If this is correct, i can add this to the help-pages or Wiki?

To be honest, I found that diagram rather confusing.

I think that it would be better to explain in words what r.external
and r.external.out do.

--
Glynn Clements <glynn@gclements.plus.com>

10

Hamish wrote:

> Yes. Specifically, it continually refreshes the display from the
> contents of the image file (which must be written using
> GRASS_PNG_MAPPED=TRUE, so that the file is modified in place, never
> truncated), providing a rough substitute for XDRIVER (i.e. run d.*
> commands from the command line, observe the result in a window).

would it be possible to have it watch the file's timestamps and only
refresh if it sees that the file has been changed?

This is problematic, as the file can be modified without the timestamp
changing (the timestamp is typically only accurate to 1 second). It
can be done, but the logic is non-trivial (and it would still need to
poll the timestamps continually).

With continual refresh I worry that ximgview and wximgview will not be
much use ssh'ed from the near-dialup connection speeds from home.

You can use percent=0, in which case it will only update when it
receives SIGUSR1 (that doesn't work on Windows; suggestions?).

You can use e.g.

  GRASS_NOTIFY="killall -USR1 `pidof wximgview`"

to cause SIGUSR1 to be sent whenever a module calls D_close_driver().

OTOH, if the issue is X bandwidth rather than application CPU usage,
it would be possible to add an option to only perform a redraw if the
data changes.

BTW, there is also the wxpyimgview script, which is the same thing but
written in Python rather than C++. It didn't occur to me to use Python
until shortly after I'd finished writing wximgview. The original
reason for using C++ was that I assumed that I would have to perform
the RGBA->RGB conversion in Python, which would be insanely slow; then
I realised that I could use NumPy.

--
Glynn Clements <glynn@gclements.plus.com>

11

On Sun, Feb 14, 2010 at 5:18 AM, Michael Barton <Michael.Barton@asu.edu> wrote:

Thanks much for the explanations. A few responses below.
On Feb 13, 2010, at 7:25 PM, Glynn Clements wrote:

Michael Barton wrote:

Should r.statistics simply be replaced with r.statistics2?

You need both r.statistics2 (accumulator-based statistics) and
r.statistics3 (quantiles) to get most of the functionality of
r.statistics.

What is the difference between r.quantile and r.statistics3 (quantiles)?

...

It seems that r.statistics3 should be renamed to something more
descriptive as it differs functionally from r.statistics

It can be used as a substitute for r.statistics method=median, but can
calculate arbitrary quantiles. It's derived from r.quantile.

Is there a way to merge r.quantile and r.statistics3 since both do quantiles?

Markus

12

Markus Neteler wrote:

>>> Should r.statistics simply be replaced with r.statistics2?
>>
>> You need both r.statistics2 (accumulator-based statistics) and
>> r.statistics3 (quantiles) to get most of the functionality of
>> r.statistics.

What is the difference between r.quantile and r.statistics3 (quantiles)?

r.quantile prints quantiles for a specific map, similar to
"r.univar -e" (but vastly more efficient for large maps).

r.statistics3 calculates quantiles for a cover/base combination, and
optionally generates a reclass of the base map, similar to
"r.statistics method=median" (but more efficient, not limited to
integer data, and for arbitrary quantiles).

>>> It seems that r.statistics3 should be renamed to something more
>>> descriptive as it differs functionally from r.statistics
>>
>> It can be used as a substitute for r.statistics method=median, but can
>> calculate arbitrary quantiles. It's derived from r.quantile.

Is there a way to merge r.quantile and r.statistics3 since both do quantiles?

It should be possible to make r.statistics3 behave like r.quantile in
the absence of a base map.

--
Glynn Clements <glynn@gclements.plus.com>

13

Hamish:

> With continual refresh I worry that ximgview and wximgview will not
> be much use ssh'ed from the near-dialup connection speeds from home.

You can use percent=0, in which case it will only update
when it receives SIGUSR1 (that doesn't work on Windows;
suggestions?).

You can use e.g.

    GRASS_NOTIFY="killall -USR1 `pidof wximgview`"

to cause SIGUSR1 to be sent whenever a module calls
D_close_driver().

then I get:
G7> d.erase
7682: no process killed

but no refresh.

resizing the window slightly by dragging its window manager frame's
corner a little redraws it.

OTOH, if the issue is X bandwidth rather than application CPU usage,
it would be possible to add an option to only perform a redraw if the
data changes.

wximgview @ percent=100%, top shows:
sshd 20%
wximgview: 11%

the numbers fluctuate a little, but the ratio stays about the same.
(100mbit ssh -X)

BTW, there is also the wxpyimgview script, which is the same thing but
written in Python rather than C++. It didn't occur to me to use Python
until shortly after I'd finished writing wximgview. The original
reason for using C++ was that I assumed that I would have
to perform the RGBA->RGB conversion in Python, which would be
insanely slow; then I realised that I could use NumPy.

@ 100% top shows:
sshd: 16%
python: 15%

...and
ximgview @ 100% top shows:
sshd: 14%
python: 12%

Hamish

14

Hamish wrote:

> You can use e.g.
>
> GRASS_NOTIFY="killall -USR1 `pidof wximgview`"
>
> to cause SIGUSR1 to be sent whenever a module calls
> D_close_driver().

then I get:
G7> d.erase
7682: no process killed

but no refresh.

That suggests that wximgview wasn't running when GRASS_NOTIFY was set.
Backticks are evaluated within double quotes, so the pidof command is
run when GRASS_NOTIFY is set; if you want it run for each command, use
single quotes.

resizing the window slightly by dragging its window manager frame's
corner a little redraws it.

SIGUSR1 and timeout (determined by percent=) force a redraw. Resizing
also causes a redraw.

--
Glynn Clements <glynn@gclements.plus.com>