[GRASS5] Platform for next generation UI

Russell Nelson wrote:

> No, I'm saying that modules should, wherever possible, work with or
> without a terminal. No writing prompts on stdout and expecting a reply
> upon stdin; that won't work if the module is called from a GUI.

Why? It works just fine with SMTP, NNTP, FTP and probably a few other
protocols. But that's the point -- if it's really a protocol that can
also be used by a person, then that's what it is. The complete
opposite would be running every command in a pseudo-teletype and
parsing its output as a screen reader would, then forcing keystrokes
at it. The first is perfectly fine; the second is completely insane.

Engaging in an interactive dialog with a program is significantly more
involved than simply running it with specific arguments.

Networking protocols don't have command-line arguments, the examples
which you cited involve significantly more data than is communicated
to a GRASS module, the protocols were explicitly designed to be
machine-readable (e.g. three-digit result codes rather than
(localised) text messages), and are all client-driven (the client
sends a request, the server replies; the server doesn't ask the client
a question and wait for its response).

> If a module is inherently interactive, there needs to be a
> non-interactive alternative.

I think by interactive you mean the second of my two above, and by
"non-interactive" I think you mean the first.

By non-interactive, I mean that all operating parameters are passed as
command-line arguments.

But can I point to a different model? It seems as if currently, GRASS
GUIS invoke command-line programs from inside themselves. I think it
would be better to have the guts of each program actually reside in a
library, with the command-line interface consisting merely of
a non-interactive or interactive shell around the library call.

Re-working the guts of each module for use in a library which is to be
used by a long-lived application would be a lot of work, for several
reasons:

1. You need much greater robustness. A segfault in a module results in
the module crashing (i.e. the specific "command" failing); a segfault
in a library function will take down the entire application.

2. You need to make more effort to prevent memory leaks. Memory leaks
in modules don't matter unless they occur inside of loop with a high
iteration count. Memory which is allocated once per invocation can be
ignored; it will all get released when the module terminates.

3. You need to be able to re-initialise the state at each invocation.
Currently, the core GRASS libraries use a lot of static data. This
data is initialised automatically at startup, and is discarded at
termination. For a library, you would have to be able to explicitly
re-initialise all of that data before invoking each command.

In short, GRASS relies heavily upon the semantics of program
execution, and removing that reliance would involve a thorough
re-write.

--
Glynn Clements <glynn@gclements.plus.com>

On Tue, 3 Jan 2006 15:56:11 +0000
Glynn Clements <glynn@gclements.plus.com> wrote:

Trevor Wiens wrote:

> > > > + Minimising dependence upon global state (WIND, $GISRC, monitor pads,
> > > > etc).
> > >
> > > I would expect this would be highly desirable to addressing issues of
> > > having multiple users accessing a single location at one time.
> >
> > That situation is harder to solve.
>
> Yes, but wouldn't it be logical to try to solve that at the same time
> since this is a desirable feature and we only need to make the changes
> once?

No. By "harder", I mean "several orders of magnitude harder".

Minimising dependence upon global state is a relatively simple matter
of modifying the functions which query that state to allow the values
to be overriden, so that the global state simply becomes a default.

A concrete example: modifying the functions which read the WIND file
to first check for an environment variable (e.g. WIND_OVERRIDE) and,
if set, read the file to which that variable refers instead. That
would allow the GUI to run commands using a specific region without
having to modify the WIND file (which could interfere with other any
other commands which were running).

[The redraw-on-resize functionality built into XDRIVER involves
something similar; R_open_driver() first checks for the environment
variable MONITOR_OVERRIDE, and only uses the MONITOR setting in $GISRC
if that is unset. This allows a monitor to redraw itself even when it
isn't the active monitor.]

Making it so that multiple commands could safely modify a given mapset
concurrently would require adding fine-grained locking to every
module, including deadlock detection, inheritable locks etc. That is a
vast amount of work just to implement, let alone the design issues.

When I had asked the question I was thinking of read access and the
possibility of multiple views of the same location could be useful with
frames for cartographic purposes. However,...

--- thinking out loud follows ---

For modification I would expect that one wouldn't want more than one
individual modifying a given vector or raster file at the same time, no
matter how sophisticated your implementation. Nor can I see (at this
moment) any real practical benefit to doing so. However, for managing
the modification of individual layers, their creation, deletion, etc.,
such a scheme could be setup by managing all a locations data in a
database like PostgreSQL because modern databases have all the
facilities already written to manage these types of problems. What I'm
thinking about is a database of environment settings, vector and raster
file names, state, ownership, etc. Individual modules would have to be
modified to first check against the database if they had permission to
perform an action. Since the problem would be identical for each module
so this would only need to be written once. One potential roadblock is
this creates more dependencies for GRASS, but since there is talk of
integrating SQLite, it might be sufficient (I've never used it so I
don't know).

Clearly this is beyond the scope of current development
plans but if it isn't an entirely bad idea brought on by sleep
deprivation, it might be worth thinking about for the future.

T
--
Trevor Wiens
twiens@interbaun.com

The significant problems that we face cannot be solved at the same
level of thinking we were at when we created them.
(Albert Einstein)

Hello,

I have a problem with a size of integer (CELL) maps. It is strange that the size of compressed integer maps is bigger than the same maps in floats. Recently, we tried to import srtm files and floating point maps took much less disk space then integers :frowning:
I tried to use the GRASS_INT_ZLIB variable, but the compression flag in a header file is still set to 1 (should be 2?). Problably I missed something. I re-compiled GRASS using zlib.
Many thanks for any hints!

Jaro

Trevor Wiens wrote:

When I had asked the question I was thinking of read access and the
possibility of multiple views of the same location could be useful with
frames for cartographic purposes. However,...

OK. You can already have multiple sessions reading from the same
mapset, either by specifying an explicit mapset using the map@mapset
syntax or by adding the mapset to the mapset path with g.mapsets.

This is still technically unsafe; if you try to read a map which is
being created or updated, the results are undefined.

For a single user with multiple sessions reading a mapset and
(optionally) one session modifying it, it isn't that hard to ensure
that problems don't arise. The risk is more significant if one user is
reading another's mapset and the two aren't coordinating their usage.

--- thinking out loud follows ---

For modification I would expect that one wouldn't want more than one
individual modifying a given vector or raster file at the same time, no
matter how sophisticated your implementation. Nor can I see (at this
moment) any real practical benefit to doing so. However, for managing
the modification of individual layers, their creation, deletion, etc.,
such a scheme could be setup by managing all a locations data in a
database like PostgreSQL because modern databases have all the
facilities already written to manage these types of problems. What I'm
thinking about is a database of environment settings, vector and raster
file names, state, ownership, etc. Individual modules would have to be
modified to first check against the database if they had permission to
perform an action. Since the problem would be identical for each module
so this would only need to be written once. One potential roadblock is
this creates more dependencies for GRASS, but since there is talk of
integrating SQLite, it might be sufficient (I've never used it so I
don't know).

The main issue is with modules which read and/or write multiple maps.

Each module would need to acquire locks on all of the maps which it
will be using at the outset, and release them all upon completion.

This really needs to be done in the module itself, as the libraries
only get to "see" individual maps being opened and closed.

--
Glynn Clements <glynn@gclements.plus.com>

At least for SRTM rasters, it depends on whether there are negative elevations (common around coastal areas). I noticed this a while back and wondered about it. I don't remember the details, but it came down to what order of integer is needed - 1, 2 or 4 bytes, unsigned - to represent all the values in the region. BUT, if there were negative integers it always used a signed 8 byte integer (or something large like that). Compression came into it also I think, again I don't remember details.

So SRTM tiles in coastal areas are often much bigger than you'd expect, though with the new version 2 SRTM that shouldn't be as big a problem since water areas have been filled with a flat 0 elevation, or an appropriate elevation for lakes.

I don't know about the GRASS_INT_ZLIB variable (env? compile?), but I was under the impression that I couldn't do anything about it.

On Jan 4, 2006, at 8:44 AM, Jaro Hofierka wrote:

Hello,

I have a problem with a size of integer (CELL) maps. It is strange that the size of compressed integer maps is bigger than the same maps in floats. Recently, we tried to import srtm files and floating point maps took much less disk space then integers :frowning:
I tried to use the GRASS_INT_ZLIB variable, but the compression flag in a header file is still set to 1 (should be 2?). Problably I missed something. I re-compiled GRASS using zlib.
Many thanks for any hints!

Jaro

-----
William Kyngesburye <kyngchaos@kyngchaos.com>
http://www.kyngchaos.com/

"Oh, look, I seem to have fallen down a deep, dark hole. Now what does that remind me of? Ah, yes - life."

- Marvin

Jaro Hofierka wrote:

I have a problem with a size of integer (CELL) maps. It is strange that
the size of compressed integer maps is bigger than the same maps in
floats.

FP maps use zlib; integer maps use run-length encoding.

Recently, we tried to import srtm files and floating point maps
took much less disk space then integers :frowning:
I tried to use the GRASS_INT_ZLIB variable, but the compression flag in
a header file is still set to 1 (should be 2?). Problably I missed
something.

Did you export it, e.g.:

  GRASS_INT_ZLIB=1
  export GRASS_INT_ZLIB

or (bash-specific):

  export GRASS_INT_ZLIB=1
?

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

Jaro Hofierka wrote:

Recently, we tried to import srtm files and floating point maps took much less disk space then integers :frowning:
I tried to use the GRASS_INT_ZLIB variable, but the compression flag in a header file is still set to 1 (should be 2?). Problably I missed something.

Did you export it, e.g.:

  GRASS_INT_ZLIB=1
  export GRASS_INT_ZLIB

or (bash-specific):

  export GRASS_INT_ZLIB=1
?

I did it using g.gisenv set="GRASS_INT_ZLIB=1"
Then I checked using g.gisenv get=GRASS_INT_ZLIB.

Jaro

Jaro Hofierka wrote:

>>Recently, we tried to import srtm files and floating point maps
>>took much less disk space then integers :frowning:
>>I tried to use the GRASS_INT_ZLIB variable, but the compression flag in
>>a header file is still set to 1 (should be 2?). Problably I missed
>>something.
>
>
> Did you export it, e.g.:
>
> GRASS_INT_ZLIB=1
> export GRASS_INT_ZLIB
>
> or (bash-specific):
>
> export GRASS_INT_ZLIB=1
> ?

I did it using g.gisenv set="GRASS_INT_ZLIB=1"
Then I checked using g.gisenv get=GRASS_INT_ZLIB.

That won't work; GRASS_INT_ZLIB needs to be an environment variable,
not a GRASS variable.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

Jaro Hofierka wrote:

Recently, we tried to import srtm files and floating point maps took much less disk space then integers :frowning:
I tried to use the GRASS_INT_ZLIB variable, but the compression flag in a header file is still set to 1 (should be 2?). Problably I missed something.

Did you export it, e.g.:

GRASS_INT_ZLIB=1
export GRASS_INT_ZLIB

or (bash-specific):

export GRASS_INT_ZLIB=1
?

I did it using g.gisenv set="GRASS_INT_ZLIB=1"
Then I checked using g.gisenv get=GRASS_INT_ZLIB.

That won't work; GRASS_INT_ZLIB needs to be an environment variable,
not a GRASS variable.

Glynn,

You're right! Thanks!

Jaro

For non-programmer it seems to be trivial -
G_open_raster_new() - deadlock;
G_open_raster_old() - softlock.

Only big problem - what if module forgets to call G_close_cell() (i.e.
segfaults).

Anyway - if we create new UI, adding calls for map locking shouldn't
be big problem, if locking is done per mapset base - independent from
user/session.

Just to keep flame,
Maris.

Each module would need to acquire locks on all of the maps which it
will be using at the outset, and release them all upon completion.

This really needs to be done in the module itself, as the libraries
only get to "see" individual maps being opened and closed.

--
Glynn Clements <glynn@gclements.plus.com>

=?ISO-8859-13?Q?M=E2ris_Narti=F0s?= wrote:

> Each module would need to acquire locks on all of the maps which it
> will be using at the outset, and release them all upon completion.
>
> This really needs to be done in the module itself, as the libraries
> only get to "see" individual maps being opened and closed.

For non-programmer it seems to be trivial -
G_open_raster_new() - deadlock;
G_open_raster_old() - softlock.

I presume that you mean write-lock/read-lock.

The problem is that a module needs to obtain all of the locks it
requires before it starts doing anything. Otherwise, it could end up
in a situation where it discovers it cannot complete (due to deadlock)
after it has already made some of the changes to the mapset.

Consequently, the locking has to be done by the module; it can't be
done by individual libgis functions.

Only big problem - what if module forgets to call G_close_cell() (i.e.
segfaults).

Use kernel locks (flock/lockf/fcntl); those are released automatically
when the process which owns the lock terminates.

That still doesn't solve the problem of inheriting locks. If one
module invokes another to do its work, the caller must lock the
relevant maps at the outset, and the callee must be able to inherit
the lock.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

Trevor Wiens wrote:

I think that there is a tendency among programmers in general to apply
the tools they are most comfortable with beyond their optimal use.
Interpreted languages like Python are often quite suitable for GUI
development. So, I wonder if we should consider, using a hybrid
strategy especially if it will enable us to make the transition from
GRASS in its current state to a more GUI friendly GRASS. Further, it
that were to work, why rework the transitional GUI in C?

Using a minority language is likely to limit the number of people who
will work on the code. This is more of an issue for maintenance than
for the original development.

I completely agree with Glynn concerning the lack of developers (altough I think that using an "easy" language such as Python might actually allow more people to participate in developing...), but I do really like the idea of a scripted language for the GUI as it allows very easy and rapid changes to the GUI without any need for compilation. Just look at the way that Michael has been able to send us tcl/tk files for testing which we could just use as drop-in replacement for the existing installed GUI. I find this a fairly nice feature which allows non-programmers to participate more actively.

Moritz

Moritz Lennert wrote:

>>I think that there is a tendency among programmers in general to apply
>>the tools they are most comfortable with beyond their optimal use.
>>Interpreted languages like Python are often quite suitable for GUI
>>development. So, I wonder if we should consider, using a hybrid
>>strategy especially if it will enable us to make the transition from
>>GRASS in its current state to a more GUI friendly GRASS. Further, it
>>that were to work, why rework the transitional GUI in C?
>
>
> Using a minority language is likely to limit the number of people who
> will work on the code. This is more of an issue for maintenance than
> for the original development.

I completely agree with Glynn concerning the lack of developers (altough
I think that using an "easy" language such as Python might actually
allow more people to participate in developing...),

That's true. A lot depends upon the degree of motivation. Someone who
has a strong desire to participate in GUI development may be willing
to learn a new language to do so. In which case, the easier the
language, the better.

OTOH, someone who stumbles across a bug or wants a minor feature added
is more likely to do the work themself if they don't need to learn
another language in order to do so.

but I do really like
the idea of a scripted language for the GUI as it allows very easy and
rapid changes to the GUI without any need for compilation. Just look at
the way that Michael has been able to send us tcl/tk files for testing
which we could just use as drop-in replacement for the existing
installed GUI. I find this a fairly nice feature which allows
non-programmers to participate more actively.

Using an interpreted language is certainly a significant benefit when
the GUI is in a state of flux. Ultimately, I think that we will start
to run into limitations though. More so for Tcl/Tk than for a binding
to a "real" toolkit (e.g. wxPython).

The main issue there is likely to be the limitations of the "canvas"
used to display maps etc. While the Tcl/Tk canvas widget is quite
powerful, you can't get around its limitations (e.g. adding new object
types). For that, you need access to the base rendering primitives so
that you can redraw the canvas yourself.

--
Glynn Clements <glynn@gclements.plus.com>

I agree with all of this. We can make considerable headway in improving the
GRASS UI with a good plan and the tools we now have. But we probably also
will hit some limits, for example in integrating 3D GIS with 2D GIS in a way
that is apparently seamless to users.

So I'm moving ahead with some nice interim updates to the UI while others
are working on prototypes for more advanced interfaces. With respect to
interpreted vs. compiled, the latter may not be so much of a problem for
people to help with if we keep it in the cvs with source code that can be
worked on with standard, widely available tools (e.g., *.ui files for Qt
designer or gtk files for glade).

Michael
______________________________
Michael Barton, Professor of Anthropology
School of Human Evolution and Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262; fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

From: Glynn Clements <glynn@gclements.plus.com>
Date: Fri, 06 Jan 2006 19:53:39 +0000
To: Moritz Lennert <mlennert@club.worldonline.be>
Cc: Trevor Wiens <twiens@interbaun.com>, Michael Barton
<michael.barton@asu.edu>, grass5 <grass5@grass.itc.it>
Subject: Re: [GRASS5] Platform for next generation UI

Moritz Lennert wrote:

I think that there is a tendency among programmers in general to apply
the tools they are most comfortable with beyond their optimal use.
Interpreted languages like Python are often quite suitable for GUI
development. So, I wonder if we should consider, using a hybrid
strategy especially if it will enable us to make the transition from
GRASS in its current state to a more GUI friendly GRASS. Further, it
that were to work, why rework the transitional GUI in C?

Using a minority language is likely to limit the number of people who
will work on the code. This is more of an issue for maintenance than
for the original development.

I completely agree with Glynn concerning the lack of developers (altough
I think that using an "easy" language such as Python might actually
allow more people to participate in developing...),

That's true. A lot depends upon the degree of motivation. Someone who
has a strong desire to participate in GUI development may be willing
to learn a new language to do so. In which case, the easier the
language, the better.

OTOH, someone who stumbles across a bug or wants a minor feature added
is more likely to do the work themself if they don't need to learn
another language in order to do so.

but I do really like
the idea of a scripted language for the GUI as it allows very easy and
rapid changes to the GUI without any need for compilation. Just look at
the way that Michael has been able to send us tcl/tk files for testing
which we could just use as drop-in replacement for the existing
installed GUI. I find this a fairly nice feature which allows
non-programmers to participate more actively.

Using an interpreted language is certainly a significant benefit when
the GUI is in a state of flux. Ultimately, I think that we will start
to run into limitations though. More so for Tcl/Tk than for a binding
to a "real" toolkit (e.g. wxPython).

The main issue there is likely to be the limitations of the "canvas"
used to display maps etc. While the Tcl/Tk canvas widget is quite
powerful, you can't get around its limitations (e.g. adding new object
types). For that, you need access to the base rendering primitives so
that you can redraw the canvas yourself.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements writes:
> Russell Nelson wrote:
>
> > > No, I'm saying that modules should, wherever possible, work with or
> > > without a terminal. No writing prompts on stdout and expecting a reply
> > > upon stdin; that won't work if the module is called from a GUI.
> >
> > Why? It works just fine with SMTP, NNTP, FTP and probably a few other
> > protocols. But that's the point -- if it's really a protocol that can
> > also be used by a person, then that's what it is.

> Engaging in an interactive dialog with a program is significantly more
> involved than simply running it with specific arguments.

That's the point -- if it's really a protocol which can also be used
by a person typing commands at it, then it's been designed to be
operated by another program.

"Back in the day" a Unix program would ordinarily be written to accept
commands on its stdin and write the results to its stdout. Somewhere
along the way people lost the idea that a program could be driven by
another, and we ended up with big monolithic applications that expect
to receive keystrokes and write curses; or worse that require a
combination of keyboard and mouse input and write graphics output to a
GUI.

> Re-working the guts of each module for use in a library which is to be
> used by a long-lived application would be a lot of work, for several
> reasons:
>
> 1. You need much greater robustness.
> 2. You need to make more effort to prevent memory leaks.

Urgh. It sounds to me like you're saying that the GRASS is so poorly
written that one function needs to be protected from another by the
kernel's inter-process protection.

> 3. You need to be able to re-initialise the state at each invocation.
> Currently, the core GRASS libraries use a lot of static data. This
> data is initialised automatically at startup, and is discarded at
> termination. For a library, you would have to be able to explicitly
> re-initialise all of that data before invoking each command.

#1 and #2 are reasons to split functions between processes.
#3 is a reason NOT to split functions between processes.

I would argue against you, but you're doing a perfectly fine job of it
yourself. In principle, you're not disagreeing with me. You're just
pointing out that an artifact of the implementation would make
creation and use of a library difficult.

--
--my blog is at blog.russnelson.com | A computer without Python is
Crynwr sells support for free software | PGPok | like a CPU without memory:
521 Pleasant Valley Rd. | +1 315-323-1241 | it runs, but you can't do
Potsdam, NY 13676-3213 | Sheepdog | anything useful with it.

Russell Nelson wrote:

> Re-working the guts of each module for use in a library which is to be
> used by a long-lived application would be a lot of work, for several
> reasons:
>
> 1. You need much greater robustness.
> 2. You need to make more effort to prevent memory leaks.

Urgh. It sounds to me like you're saying that the GRASS is so poorly
written that one function needs to be protected from another by the
kernel's inter-process protection.

Well, that's the implication of 1, and in a sense it's true, although
your phrasing isn't entirely fair.

The problem is that reliability is subject to exponential decay. E.g.
if the probability of a module crashing is 0.01, the probability of it
not crashing is 0.99 and the probability of running 100 commands in
succession with none of them crashing is 0.99^100 = 0.36.

IOW, if you want reasonable robustness for the application as a whole,
the robustness of each command needs to be very much higher,
essentially "bulletproof". The code in the GRASS modules wasn't
originally written to such constraints; it's reliability is merely
reasonable for its intended use.

Regarding point 2, if you're writing a standalone executable, not
bothering to free memory isn't a bug. There is no reason to do this
when the the kernel is going to do it for you.

> 3. You need to be able to re-initialise the state at each invocation.
> Currently, the core GRASS libraries use a lot of static data. This
> data is initialised automatically at startup, and is discarded at
> termination. For a library, you would have to be able to explicitly
> re-initialise all of that data before invoking each command.

#1 and #2 are reasons to split functions between processes.
#3 is a reason NOT to split functions between processes.

Huh?

I would argue against you, but you're doing a perfectly fine job of it
yourself. In principle, you're not disagreeing with me. You're just
pointing out that an artifact of the implementation would make
creation and use of a library difficult.

Exactly. But that's the whole point. We aren't discussing creating a
GIS from scratch, we're discussing where GRASS should go from where it
is now. If you were writing a GIS from scratch, you would do it very
differently to how GRASS has been written.

--
Glynn Clements <glynn@gclements.plus.com>

> > Re-working the guts of each module for use in a library which is to be
> > used by a long-lived application would be a lot of work, for several
> > reasons:
> >
> > 1. You need much greater robustness.
> > 2. You need to make more effort to prevent memory leaks.
>
> Urgh. It sounds to me like you're saying that the GRASS is so poorly
> written that one function needs to be protected from another by the
> kernel's inter-process protection.

Well, that's the implication of 1, and in a sense it's true, although
your phrasing isn't entirely fair.

The problem is that reliability is subject to exponential decay. E.g.
if the probability of a module crashing is 0.01, the probability of it
not crashing is 0.99 and the probability of running 100 commands in
succession with none of them crashing is 0.99^100 = 0.36.

IOW, if you want reasonable robustness for the application as a whole,
the robustness of each command needs to be very much higher,
essentially "bulletproof". The code in the GRASS modules wasn't
originally written to such constraints; it's reliability is merely
reasonable for its intended use.

Regarding point 2, if you're writing a standalone executable, not
bothering to free memory isn't a bug. There is no reason to do this
when the the kernel is going to do it for you.

One point which I forgot until after I'd sent this.

There are some similarities between points 1 and 2, which ties in with
point 3. In the same way that standalone modules leave initialising
state to the loader, and deallocating memory to the kernel, they deal
with errors in a similar manner: call the fatal error routine,
G_fatal_error(), which terminates the program, deallocating all memory
and discarding the global state.

While it's possible to install your own handler for fatal errors, any
code which calls G_fatal_error() relies upon the fact that it doesn't
return, and won't make any attempt to clean up whatever global state
it has been using.

Again, this is an entirely reasonable approach for a standalone
module, but won't work in a library.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements writes:
> Russell Nelson wrote:
>
> > > Re-working the guts of each module for use in a library which is to be
> > > used by a long-lived application would be a lot of work, for several
> > > reasons:
> > >
> > > 1. You need much greater robustness.
> > > 2. You need to make more effort to prevent memory leaks.
> >
> > Urgh. It sounds to me like you're saying that the GRASS is so poorly
> > written that one function needs to be protected from another by the
> > kernel's inter-process protection.
>
> Well, that's the implication of 1, and in a sense it's true, although
> your phrasing isn't entirely fair.

Yeah, on second thought, you're right; I apologize. It *is* easier to
write code if it's okay if the code can exit. If you look at qmail's
source code (written in C), some programs simply exit immediately if
they run out of memory. There's nothing more to be done, so they may
as well indicate a temporary failure and be done with it. Other
programs (like qmail-send) CANNOT under any circumstances exit
prematurely, so for example if they cannot get more memory or write to
a file, they retry the operation until it succeeds.

> The problem is that reliability is subject to exponential decay. E.g.
> if the probability of a module crashing is 0.01,

No, that's not it. There are two sources of unreliability: repeatable
and unrepeatable. The first would include a request to do something
which always exceeds the available resources, or a bug which cannot
handle certain data. The second would include an unreliable CPU
(e.g. if the CPU fan has died), or a bug caused by an uncaught
resource starvation.

If the unreliability is repeatable, then it's NOT subject to
exponential decay.

If the unreliability is not repeatable, then it doesn't really matter
if the sequence of operations occurs in separate programs or in one;
either way the sequence is broken and needs to be restarted.

> Regarding point 2, if you're writing a standalone executable, not
> bothering to free memory isn't a bug. There is no reason to do this
> when the the kernel is going to do it for you.

There's two kinds of memory leaks: not freeing memory when you exit,
and losing memory on an ongoing basis. The first is not a problem.
The second is a problem no matter what; if it doesn't bite you with a
small data set, it will with a large one. Data sets only get larger.

> > #3 is a reason NOT to split functions between processes.
>
> Huh?

If you need data loaded, it makes more sense to load it once and share
it between library calls, rather than to have to load it every time a
program starts up.

> Exactly. But that's the whole point. We aren't discussing creating a
> GIS from scratch, we're discussing where GRASS should go from where it
> is now. If you were writing a GIS from scratch, you would do it very
> differently to how GRASS has been written.

Maybe it's time to write a GIS from scratch, then, liberally borrowing
from GRASS as appropriate?

--
--my blog is at blog.russnelson.com | A computer without Python is
Crynwr sells support for free software | PGPok | like a CPU without memory:
521 Pleasant Valley Rd. | +1 315-323-1241 | it runs, but you can't do
Potsdam, NY 13676-3213 | Sheepdog | anything useful with it.

Russell Nelson wrote:

> The problem is that reliability is subject to exponential decay. E.g.
> if the probability of a module crashing is 0.01,

No, that's not it. There are two sources of unreliability: repeatable
and unrepeatable. The first would include a request to do something
which always exceeds the available resources, or a bug which cannot
handle certain data. The second would include an unreliable CPU
(e.g. if the CPU fan has died), or a bug caused by an uncaught
resource starvation.

If the unreliability is repeatable, then it's NOT subject to
exponential decay.

If the unreliability is not repeatable, then it doesn't really matter
if the sequence of operations occurs in separate programs or in one;
either way the sequence is broken and needs to be restarted.

That's a completely separate issue.

I'm not talking about reliability in terms of whether or not an
operation succeeds, but in terms of how it fails.

If a standalone module fails, it doesn't really matter how. There
isn't a great deal of difference between segfaulting, calling exit()
straight away, or calling exit() after having deallocated memory,
restored global state etc.

OTOH, for a library function, it makes a great deal of difference how
failure is handled.

> Regarding point 2, if you're writing a standalone executable, not
> bothering to free memory isn't a bug. There is no reason to do this
> when the the kernel is going to do it for you.

There's two kinds of memory leaks: not freeing memory when you exit,
and losing memory on an ongoing basis. The first is not a problem.
The second is a problem no matter what; if it doesn't bite you with a
small data set, it will with a large one. Data sets only get larger.

Standalone modules normally handle the latter (or at least try to),
but there's no reason for them to handle the former, so they don't.
OTOH, library functions need to handle both.

> > #3 is a reason NOT to split functions between processes.
>
> Huh?

If you need data loaded, it makes more sense to load it once and share
it between library calls, rather than to have to load it every time a
program starts up.

OK, I see your point.

> Exactly. But that's the whole point. We aren't discussing creating a
> GIS from scratch, we're discussing where GRASS should go from where it
> is now. If you were writing a GIS from scratch, you would do it very
> differently to how GRASS has been written.

Maybe it's time to write a GIS from scratch, then, liberally borrowing
from GRASS as appropriate?

My main point is that you aren't going to be able to borrow as much as
you might initially suspect, due to the mechanisms used for:

1. State initialisation (the loader initialises global variables from
the data and BSS segments at startup).

2. Resource deallocation (call exit()).

3. Error handling (again, call exit()).

--
Glynn Clements <glynn@gclements.plus.com>

On Wed, Jan 04, 2006 at 08:20:12PM +0100, Jaro Hofierka wrote:

Glynn Clements wrote:
>Jaro Hofierka wrote:
>
>
>>>>Recently, we tried to import srtm files and floating point maps
>>>>took much less disk space then integers :frowning:
>>>>I tried to use the GRASS_INT_ZLIB variable, but the compression flag in
>>>>a header file is still set to 1 (should be 2?). Problably I missed
>>>>something.
>>>
>>>
>>>Did you export it, e.g.:
>>>
>>> GRASS_INT_ZLIB=1
>>> export GRASS_INT_ZLIB
>>>
>>>or (bash-specific):
>>>
>>> export GRASS_INT_ZLIB=1
>>>?
>>
>>I did it using g.gisenv set="GRASS_INT_ZLIB=1"
>>Then I checked using g.gisenv get=GRASS_INT_ZLIB.
>
>
>That won't work; GRASS_INT_ZLIB needs to be an environment variable,
>not a GRASS variable.
>

Glynn,

You're right! Thanks!

Jaro,

the ZLIB compression is broken in GRASS 6.
Please don't use it.

GRASS 6.1.cvs (spearfish60):~ > g.region s=0 n=15 w=0 e=15 res=1
GRASS 6.1.cvs (spearfish60):~ > r.mapcalc mymap=3 ; r.out.ascii mymap
100%
north: 15
south: 0
east: 15
west: 0
rows: 15
cols: 15
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

GRASS 6.1.cvs (spearfish60):~ > g.region s=0 n=13 w=0 e=13 res=1
GRASS 6.1.cvs (spearfish60):~ > r.mapcalc mymap=3 ; r.out.ascii mymap
100%
north: 13
south: 0
east: 13
west: 0
rows: 13
cols: 13
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3

GRASS 6.1.cvs (spearfish60):~ > g.region s=0 n=11 w=0 e=11 res=1
GRASS 6.1.cvs (spearfish60):~ > r.mapcalc mymap=3 ; r.out.ascii mymap
100%
north: 11
south: 0
east: 11
west: 0
rows: 11
cols: 11
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34
120 156 99 102 134 3 0 0 209 0 34

At least small map are not handled correctly.

Markus