[GRASS5] some thoughts

GRASS GRASS developer
1

On Thu, 17 Jan 2002, Glynn Clements wrote:
  

In order to be able to implement a proper GUI (as well as things like
a web interface), the core programs have to cease using G_ask_*,
R_get_location_with_*, stdin, stdout, stderr, etc. Instead, they need
to just parse the command line, read data, perform computation, write
data, and exit.

I wasn't grumbling about the user interface, much less about the GUI. I
use the command line interface, because GUI's don't work well over a slow
network link and I often work from home over a 56K modem. Removing the
terminal interface from the commands would destroy much of the value in
GRASS. On the other hand, you could rewrite the parser to eliminate some
the interactive prompts and it wouldn't bother me at all.

My suggestion was that a layer (running as a user-space daemon) should be
inserted in the libraries. That layer would be responsible for managing
the user's run-time environment and coordinating between different tasks.
The user would no longer need to micro-manage his own environment, and
some variables (location, for instance) could be changed on the fly. Other
variables (like the region settings) could be altered by one user without
effecting other users, making simultaneous access a more realistic
possibility.

I think the new layer could be built into the exiting GRASS architecture
without breaking any of the existing functionality. The initial version
would be simple and have a minimal role. Additional library routines could
be rewritten to use the added layer as time goes by.

Roger Miller

2

Roger Miller wrote:

> In order to be able to implement a proper GUI (as well as things like
> a web interface), the core programs have to cease using G_ask_*,
> R_get_location_with_*, stdin, stdout, stderr, etc. Instead, they need
> to just parse the command line, read data, perform computation, write
> data, and exit.

I wasn't grumbling about the user interface, much less about the GUI. I
use the command line interface, because GUI's don't work well over a slow
network link and I often work from home over a 56K modem. Removing the
terminal interface from the commands would destroy much of the value in
GRASS. On the other hand, you could rewrite the parser to eliminate some
the interactive prompts and it wouldn't bother me at all.

The main problem with the terminal interface (by which I mean Vask,
G_ask_* etc) is that it gets in the way of any other kind of usage.
E.g. tcltkgrass has to run every command in an xterm because the
command might depend upon a terminal.

The other big problem is the "session" mechanism, and the way in which
it interacts with mapsets. This means that:

1. Concurrent sessions on a mapset aren't possible, because the
current region is stored in the mapset.

2. Concurrent sessions for a user aren't possible, because some of the
information is stored in $GISRC.

Now, some form of persistent state is certainly useful; I wouldn't
want to have to type:

  d.something gisdbase=... location=... mapset=... region=... monitor=...

for every command. OTOH, this might be preferable for a web interface,
rather than having to generate mapsets on the fly. It wouldn't be that
hard to modify G_parser to automatically add such options to every
command (in the same way the Xt programs all accept "-display", "-xrm"
etc), *except* that a few programs use these settings to initialise
the defaults before calling G_parser().

In principle, it shouldn't be hard to combine $LOCATION/WIND and parts
of $GISRC into a dedicated state file, which would be generated at
session startup based upon $GISRC (which would then be just a user
configuration file, like every other ~/.*rc file).

There would probably have to be some locking added to the
applications. I doubt that it would be sufficient to add locking to
libgis alone, as you would need to ensure atomic operation of
read-modify-write cycles and multi-file updates (e.g. cell, cellhd,
cats).

Also, the $GISDBASE/$LOCATION_NAME/$MAPSET directory layout doesn't
coexist particularly well with Unix file permissions. E.g. $GISDBASE
has to be writable for users to be able to create locations, and each
location has to be writable for users to be able to create mapsets. A
"mapset path" would be better, e.g.

  GISPATH=/opt/grass-data:$HOME/grass-data

with locations existing beneath each component.

My suggestion was that a layer (running as a user-space daemon) should be
inserted in the libraries. That layer would be responsible for managing
the user's run-time environment and coordinating between different tasks.
The user would no longer need to micro-manage his own environment, and
some variables (location, for instance) could be changed on the fly. Other
variables (like the region settings) could be altered by one user without
effecting other users, making simultaneous access a more realistic
possibility.

I think the new layer could be built into the exiting GRASS architecture
without breaking any of the existing functionality. The initial version
would be simple and have a minimal role. Additional library routines could
be rewritten to use the added layer as time goes by.

I'm not sure how a separate daemon would help. I think that most of
the problems stem from the way in which the settings are stored in
$GISRC and $LOCATION/WIND (and others; e.g. $LOCATION/GROUP, which
also conflicts with the $LOCATION/group directory if your filesystem
isn't case-sensitive, e.g. Cygwin, and maybe MacOS).

Also, regarding the monitor, I'm becoming more convinced that the
redraw-on-resize feature was a mistake. Whereas the monitor used to be
a standalone component, living at the bottom of the call graph, it's
now part of a loop which binds it to the session in which it was
started. It would have been better to extend the protocol to support a
d.resize command, which would basically run d.save, resize the
monitor, then redraw the contents using the d.save output.

Actually, I'm wondering whether it would be better for D_add_to_list
et al to store the data in a file instead of within the monitor. In
any case, a graphics terminal and a bunch of d.* commands aren't
really a substitute for a stateful viewer.

The thrust of my position is that much of what exists to support
interactive use doesn't do a particularly good job of it and, more
importantly, it's getting in the way of something more substantial.

--
Glynn Clements <glynn.clements@virgin.net>

3

Hi all,

I have been following this news group silently for some time, but now you
have finally got me writing.

My interest in GRASS have primarily been experimental. In general, I am
interested in free tools for geospatial computing. Currently I work as a
programmer and system designer in a large scale software project. I have
some user experience with generic GIS tools (mostly MapInfo some years
back). As a software user, most of my related experience have been using
applications centred around dynamic geospatial data, and presentation of
this type of data as an overlay on more static "map" data.

Well, back to the discussion,

I think both Roger and Glynn have some valuable opinions, I don't think they
are excluding each other in any way.

My experience with GRASS as a user tool concur with Roger's. The command
line tools are a blessing as far as giving flexibility, but a curse as far
as user friendliness for the less experienced user (like myself). Roger
wants some tools automating some of the small quirks the experienced user
does without even thinking of it... well I have also struggled with the
monitors in GRASS. Coming from the outside, with no GRASS experienced friend
to teach me, I find the hole concept rather odd. The fact that they don't
work right away - does not help.. I am afraid this discourages a lot of
potential GRASS users, even the ones that rather starts with the command
line interface than attempting the tcl/tk stuff.

Glynn is bringing up something that need to be addressed if GRASS is ever
are going to get a modern UI. The cost of a modern UI is almost guarantied
to be less flexibility than the UNIX like command line tools used today. I
don't think the community should even think of breaking what you have, to
get what you want. And that is why a more layered architecture of the
libraries and/or tools should be considered. I think the main thing a
modern UI can do for GRASS is to increase the user base, and therefore also
the incentive and award for being a GRASS developer.

Some thought from me:

layer 1:
basic current and future GRASS and other libraries

layer 2:
user level libraries e.g. equalent to functionality of command line tools,
e.g. v.in.mif would use a layer 2 routine called v_in_mif(...) which works
on generic streams set up by the caller to files, pipes, sockets, or memory.
Basically, the idea is to move most of the work-horse code into these
library routines and publish them as a library with a documented and
supported interface. UI developers may use layer 2 routines directly.
Additional layer 2 routines should also support multiple concurrent
environments.

layer 3:
executable command line tools, as today - but more rigid and less
interactive. Intended as a better option for software generated or managed
scripting. Calls level 2 equalents after setting up streams.

layer 4.
existing command line UI. Calls level 3 or 2.

layer 5.
Current an future GRASS supported GUIs. Calls level 2, 3 or 4.

layer 6.
not a GRASS supported level, but a category for other applications bases
on the lover layers in GRASS

Just in case someone feels like jumping on this, that is Ok - go ahead. I
don't write this based on knowledge of the current state of the source code,
I have been to busy to have time to dig into it. Neither do I promise to do
that on the future.

-- Bjorn Roald

4

On Fri, 18 Jan 2002 02:42:55 +0100, Bjorn Roald <bjroald@online.no> wrote:

[snip]

My experience with GRASS as a user tool concur with Roger's. The command
line tools are a blessing as far as giving flexibility, but a curse as far
as user friendliness for the less experienced user (like myself). Roger
wants some tools automating some of the small quirks the experienced user
does without even thinking of it... well I have also struggled with the
monitors in GRASS. Coming from the outside, with no GRASS experienced friend
to teach me, I find the hole concept rather odd. The fact that they don't
work right away - does not help.. I am afraid this discourages a lot of
potential GRASS users, even the ones that rather starts with the command
line interface than attempting the tcl/tk stuff.

For a little perspective of the monitor scenario, think about the fragmented
nature of proprietary *nix in the 1980's before the standardization on
the X Window System. It's my understanding, the display system was
developed to be a small simple set of commands for which drivers could
be written specific to the hardware (probably mostly 8-bit color without
any concept of "windows", hence d.frame).

TclTk GRASS is simply a menu systems written around previous/current
GRASS commands. It's meant to make things a little easier for user's,
but isn't really standalone GUI.

Glynn is bringing up something that need to be addressed if GRASS is ever
are going to get a modern UI. The cost of a modern UI is almost guarantied
to be less flexibility than the UNIX like command line tools used today. I
don't think the community should even think of breaking what you have, to
get what you want. And that is why a more layered architecture of the
libraries and/or tools should be considered. I think the main thing a
modern UI can do for GRASS is to increase the user base, and therefore also
the incentive and award for being a GRASS developer.

I don't think there's a question about wanting to ditch the current display
architechture (and modifying the session management functionality). It's
a big task to [re]write all the necessary parts to make the core of GRASS
really usable as a flexible backend for GUI's or WWW or whatever.

Some thought from me:

layer 1:
basic current and future GRASS and other libraries

In order for GRASS "libraries" to be even usable as shared object libraries
will probably take considerable rewriting. They rely heavily on a lot of
global data. Not to mention, their contents probably need a reorganization.

layer 2:
user level libraries e.g. equalent to functionality of command line tools,
e.g. v.in.mif would use a layer 2 routine called v_in_mif(...) which works
on generic streams set up by the caller to files, pipes, sockets, or memory.
Basically, the idea is to move most of the work-horse code into these
library routines and publish them as a library with a documented and
supported interface. UI developers may use layer 2 routines directly.
Additional layer 2 routines should also support multiple concurrent
environments.

I essentially agree with this scenario. My thought are, this part would
basically be broken into two or three related sets of things.

  1. You have a function entry point that does the task.
  2. You have a set of "generic" input validation routines.
  3. You have an interface specification, such that command line, GUI,
      whatever, can be generated.

Such a scenario wouldn't necessarily work with all types of commands, but
I think it'd work for most. It gets more complex with functionality that
needs to interact with the user more (and modifies its behavior based
on inputs). Such functionality would probably have to be written specific
for the interface type (at least partially).

--
Eric G. Miller <egm2@jps.net>

5

Eric G. Miller wrote:

> Some thought from me:
>
> layer 1:
> basic current and future GRASS and other libraries

In order for GRASS "libraries" to be even usable as shared object libraries
will probably take considerable rewriting. They rely heavily on a lot of
global data. Not to mention, their contents probably need a reorganization.

A simple ".a" -> ".so" conversion is simple enough; Linux-style shared
objects don't have the same sort of restrictions as e.g. Windows DLLs.

> layer 2:
> user level libraries e.g. equalent to functionality of command line tools,
> e.g. v.in.mif would use a layer 2 routine called v_in_mif(...) which works
> on generic streams set up by the caller to files, pipes, sockets, or memory.
> Basically, the idea is to move most of the work-horse code into these
> library routines and publish them as a library with a documented and
> supported interface. UI developers may use layer 2 routines directly.
> Additional layer 2 routines should also support multiple concurrent
> environments.

I essentially agree with this scenario. My thought are, this part would
basically be broken into two or three related sets of things.

  1. You have a function entry point that does the task.
  2. You have a set of "generic" input validation routines.
  3. You have an interface specification, such that command line, GUI,
      whatever, can be generated.

Such a scenario wouldn't necessarily work with all types of commands, but
I think it'd work for most. It gets more complex with functionality that
needs to interact with the user more (and modifies its behavior based
on inputs). Such functionality would probably have to be written specific
for the interface type (at least partially).

Also, I'd just like to make clear my preferred approach regarding a
user interfaces (graphical or otherwise).

I'm definitely *not* aiming for a monolithic program, but something
more akin to tcltkgrass, where "commands" (e.g. menu entries) would
spawn separate executables.

The main advantage of this approach is the robustness provided by
"insulating" the commands within separate processes. If a specific
command segfaults, exhausts memory, calls exit() etc, it doesn't take
down the entire session. Also, system resources (memory, file
descriptors, etc) are guaranteed to be freed.

This means that we don't have to get rid of every single crash bug,
memory leak, and exit() call before the resulting system becomes
usable.

What I am aiming for is to make the individual commands more "program
friendly" (c/f "user friendly"). This includes things like:

+ Not trying conduct question/answer sessions with the user on
stdin/stdout.

+ Not using modal loops with e.g. R_get_location_with_*.

+ Overall consistency regarding argument parsing (including --help and
--interface-description), error/warning handling etc.

+ Self-description (again including --help and --int-desc), e.g. with
more precise argument type information (i.e. more types than just
string/int/float, e.g. "existing raster layer name" rather than just
"string"), allowable argument combinations (e.g. option groups) etc.

+ Overall separation of functionality from interface. E.g. programs
like d.what.* might have coordinates passed as arguments or fed on
stdin, leaving it for the UI to obtain the coordinates from the user.

This would allow for a more integrated UI than is currently possible.

E.g. tcltkgrass prompts for arguments using fairly generic dialogs,
then runs the command and ... well, the command does whatever it does;
it's effectively detached from the UI at that point.

The command may obtain further input from the xterm in which it is run
(either in teletype-style prompt/answer form, or using Vask, or a
free-form curses interface) and/or from the monitor (either just
coordinates or "menu" selections in the style of d.menu).

Ultimately you can only get so much integration before separate
processes start to become a problem. But eliminating the exit() calls,
opportunities for fatal errors[1] and memory leaks isn't going to
happen overnight.

[1] By which I basically mean crashes. G_fatal_error() can be modified
to allow a "fatal" error handler to be installed which could call
setjmp() rather than exit(). This would allow "exception" style error
handling rather than having to pass NULL/-1/whatever all the way up
the call stack.

--
Glynn Clements <glynn.clements@virgin.net>

6

On Fri, 18 Jan 2002, Glynn Clements wrote:

The main advantage of this approach is the robustness provided by
"insulating" the commands within separate processes. If a specific
command segfaults, exhausts memory, calls exit() etc, it doesn't take
down the entire session. Also, system resources (memory, file
descriptors, etc) are guaranteed to be freed.

This means that we don't have to get rid of every single crash bug,
memory leak, and exit() call before the resulting system becomes
usable.

....

Ultimately you can only get so much integration before separate
processes start to become a problem. But eliminating the exit() calls,
opportunities for fatal errors[1] and memory leaks isn't going to
happen overnight.

[1] By which I basically mean crashes. G_fatal_error() can be modified
to allow a "fatal" error handler to be installed which could call
setjmp() rather than exit(). This would allow "exception" style error
handling rather than having to pass NULL/-1/whatever all the way up
the call stack.

The R interface "takes over" error handling from libgis.a as:

void R_G_init(char *name) {

  G_set_error_routine(R_handler);
  G_sleep_on_error(0);
  G_gisinit(name);
}

with R_handler() being a simple function that passes the error on within
R. If modules never use exit() but only G_fatal_error(), this works
smoothly in my experience.

Interfaces to other programmes also need clean command line style argument
sets which can be used through system calls - though in fact VASK-style
interfaces still work from within R in a terminal window.

On a different aspect, I've thought of some kind of daemon process too,
but as an auditing monitor for metadata, giving some kind of tag to
withdrawals and commits to the database, not least for external programmes
to talk to, so that metadata is kept clean for data objects being moved in
and out of the system. However, I'm sure there must be better ways of
doing this (in S/R with sticky attributes on objects, perhaps - a site
object might have a location/projection/... tag as an attribute). As it
is, lots of metadata is just in the database, but once data has been
extracted, it may easily be lost.

Roger

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no
and: Department of Geography and Regional Development, University of
Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.

7

Roger Bivand wrote:

> Ultimately you can only get so much integration before separate
> processes start to become a problem. But eliminating the exit() calls,
> opportunities for fatal errors[1] and memory leaks isn't going to
> happen overnight.
>
> [1] By which I basically mean crashes. G_fatal_error() can be modified
> to allow a "fatal" error handler to be installed which could call
> setjmp() rather than exit(). This would allow "exception" style error
> handling rather than having to pass NULL/-1/whatever all the way up
> the call stack.

Of course, I meant longjmp().

>
The R interface "takes over" error handling from libgis.a as:

void R_G_init(char *name) {

  G_set_error_routine(R_handler);
  G_sleep_on_error(0);
  G_gisinit(name);
}

with R_handler() being a simple function that passes the error on within
R.

Right; you've already done that bit.

I was thinking about supporting pushing/popping error handlers, along
with try/catch/throw macros. Given that most code just gives up and
quits as soon as something goes wrong, status returns mostly just make
the code ugly.

If modules never use exit() but only G_fatal_error(), this works
smoothly in my experience.

Unfortunately, a fair amount of code currently just calls exit(). But
that isn't hard to change, just a lot of donkey work.

--
Glynn Clements <glynn.clements@virgin.net>

8

Eric G. Miller, wrote Friday, January 18, 2002 4:08 AM

For a little perspective of the monitor scenario, think about the
fragmented
nature of proprietary *nix in the 1980's before the standardization on
the X Window System. It's my understanding, the display system was
developed to be a small simple set of commands for which drivers could
be written specific to the hardware (probably mostly 8-bit color without
any concept of "windows", hence d.frame).

TclTk GRASS is simply a menu systems written around previous/current
GRASS commands. It's meant to make things a little easier for user's,
but isn't really standalone GUI.

Thanks for the insigt on monitor scenario and TclTk GUI! I guess I already
had figured most of this out. But it is always good to have the history of
things in mind.

I don't think there's a question about wanting to ditch the
current display
architechture (and modifying the session management functionality). It's
a big task to [re]write all the necessary parts to make the core of GRASS
really usable as a flexible backend for GUI's or WWW or whatever.

I agree,
The first question to ask is:

Is there really any point?

Maybe what some of us want GRASS to become is better served by other "free"
software, existing or emerging.

If the answer to that is NO, then everybody may be served by a GRASS
development focusing on increased functionality, rather than better
usability as stand along application or building blocks. GRASS would then
continue to appeal to its limited user base, but never become/or be part of
any mainstream application. It is just not simple enough to use.

If the answer is YES: Well then I think the important thing is to get the
architectural issues in place as well as the tools the individual programmer
needs. As long as the existing functionality is not broken, the need for a
complete re-work of every module by any set deadline is not critical. If
the tools are easy to use (and understand), and they add something to the
existing system, such as better robustness, then I think re-writing the
modules for the new design as they come by becomes natural. Eventually at
some point, somebody will take the task of getting everything cleaned up.
So what are these tools, I think the code structure, make system and naming
conventions are central. But also stuff like error/exception handling,
error reporting system, debug tracing....

> layer 2:
> user level libraries e.g. equalent to functionality of command
line tools,
> e.g. v.in.mif would use a layer 2 routine called v_in_mif(...)
which works
> on generic streams set up by the caller to files, pipes,
sockets, or memory.
> Basically, the idea is to move most of the work-horse code into these
> library routines and publish them as a library with a documented and
> supported interface. UI developers may use layer 2 routines directly.
> Additional layer 2 routines should also support multiple concurrent
> environments.

I essentially agree with this scenario. My thought are, this part would
basically be broken into two or three related sets of things.

  1. You have a function entry point that does the task.
  2. You have a set of "generic" input validation routines.
  3. You have an interface specification, such that command line, GUI,
      whatever, can be generated.

Such a scenario wouldn't necessarily work with all types of commands, but
I think it'd work for most. It gets more complex with functionality that
needs to interact with the user more (and modifies its behavior based
on inputs). Such functionality would probably have to be written specific
for the interface type (at least partially).

I think functionality that need to interact with the user while it is
executing, such as v.digit, need to be split up at the library level. The
hole idea is then that this will facilitate new UI support for the same
tasks. The old UI may be completely scrapped in exchange with something new
and better.

As for GUI support, think it is vice to use a "strategy" pattern for support
of GUI interaction at the library level. Without knowing the details, I
don't think the GRASS sources nesseserially are that far of the mark. But I
have a feeling there are a "lack of feature" problem. The idea is to
possibly be able to support multiple GUI and graphical libraries. I am not
asking for everybody, or anybody, to jump into multi target development (in
the sense of GUI/and graphical libraries, not hardware/OS platform). The
point is to insulate the libraries from changes in the application code
utilizing it. If GRASS libraries are becoming mainstream building blocks,
then they will need to be adaptable into applications using various
graphical engines, existing and future such. Using a "strategy" pattern
will hopefully protect the libraries against major re-write every time new
technology arises.

----
Bjorn Roald, mailto://bjroald@online.no

9

Friday, January 18, 2002 6:36 AM, Glynn Clements wrote

Also, I'd just like to make clear my preferred approach regarding a
user interfaces (graphical or otherwise).

I'm definitely *not* aiming for a monolithic program, but something
more akin to tcltkgrass, where "commands" (e.g. menu entries) would
spawn separate executables.

My point is that layer 3 in my original posting should support this. While
layer 2 basically are called from the higher (or outer) layers. Whether
applications in layer 5, are monolithic or not - should be up to the
developer of such applications.

The goals for the GRASS team may then be to provide software in layers 1 to
4, and leave layer 5 class applications entirely up to others. Having said
that, the goal for the robustness of layer 1 and 2 should be sufficient to
support monolithic application building. This does not mean that routines
should never be able to give up. A clean way of passing the problem to the
caller is all that is needed. A system modelled after java/C++ exception
handling would do, but I am not familiar with any good way of doing this in
C.

----
Bjorn Roald