[GRASS-dev] another not quiet GRASS command

cmbarton · September 2, 2007, 6:43pm

I'm trying to use g.mapset in the georectifier module for wxgrass. I have to
switch back and forth between locations and mapsets. For TclTk, I used
g.gisenv, but thought it would be more convenient to use g.mapset here.

BUT, I've hit the perennial problem of a GRASS command that won't be quiet.
Regardless of the --q switch, g.mapset insists on dumping a warning to
stderr.

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using subprocess.Popen)
to report an error when there isn't one.

(Martin, this seems to be a general issue with cmd.Command. Is there a way
to work around this for commands that insist on dumping extraneous stuff to
stderr?)

This might be a useful warning for command line users to get. But for
scripting, I don¹t want to see ANY warnings or other extraneous output. It
does nothing except lock up scripts that don't make special arrangements to
parse such stuff. This is a real problem for TclTk, leaving us with the
necessity of sending all stderr to dev/null so that we only get back error
messages sent by TclTk. Even for languages like wxPython that can separate
stderr from stdout, it means that attempts to gracefully trap and display
errors will incorrectly show an error when something like this happens. It
would make life so much easier for scripting if we could optionally turn off
ALL returned message except stdout for commands that are supposed to return
a value and stderr for real errors.

Michael
__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

H_B · September 3, 2007, 10:58am

Michael Barton wrote:

I'm trying to use g.mapset in the georectifier module for wxgrass. I
have to switch back and forth between locations and mapsets. For TclTk,
I used g.gisenv, but thought it would be more convenient to use
g.mapset here.

BUT, I've hit the perennial problem of a GRASS command that won't be
quiet. Regardless of the --q switch, g.mapset insists on dumping a
warning to stderr.

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using
subprocess.Popen) to report an error when there isn't one.

..

This might be a useful warning for command line users to get. But for
scripting, I don¹t want to see ANY warnings or other extraneous output.

Warnings and fatal error messages are not affected by --quiet by design;
they are not optional and need to be passed on to the user.

With any module you can and will get warnings or fatal error messages,
and any GUI wrapper will need to deal with them somehow.

In the above case of georect.tcl, we may ask 1) if switching between
mapsets in the script is really the best solution; 2) if that warning
should be a warning. One might argue that in this case it should use
G_important_message(), not G_warning(), but the warning message reads
quite well IMO. G_important_message("WARNING: ..."); ? Seems kinda
circuitous to me.

It does nothing except lock up scripts that don't make special
arrangements to parse such stuff.

Because the calling wrapper is deficient, not the concept of warnings and
error messages. All scripts must make special arrangements to parse
such stuff. This can be aided by a wrapper fn, in the same way as the Tcl
GUI menu has different 'spawn' module launchers. The cure is to fix the
wrapper -- not hide the errors. It is fundamentally wrong to assume that
anything written to stderr automatically means a problem is fatal, and
the GUI code needs to be written with that in mind, even if it is a pain.

It would make life so much easier for scripting if we could optionally
turn off ALL returned message except stdout for commands that are
supposed to return a value and stderr for real errors.

Warnings and fatal errors are "real errors". I guess one problem lies in
the concept of the warning. Is it a warning that the user is sailing in
dangerous waters? or is it a warning that the binary executable module
did something unexpected, but non-fatal? (I'd argue both are needed). Is
it a line item to be logged in GIS_ERROR_LOG or a console? or is it a
standalone pop-up GUI message? (I'd argue that the GUI needs a console).

Note in the C API there is G_suppress_warnings() (in lib/gis/error.c),
/*!
* \fn int G_suppress_warnings (int flag)
*
* \brief Suppress printing a warning message to stderr
*
* \param[in] flag a warning message will be suppressed if non-zero value
is given
* \return previous flag
*/

Use that at your own peril, it is likely that it will accidentally cause
more trouble than it solves.

... so instead of lots of open and catch statements, I'd suggest that any
call to a module be done using a standard wrapping function, and that
standard wrapping function take care of robust open/close & error
handling, once & correctly. I don't think there's another good way.
Duplicating robust error handling for all external calls is wasted
duplicated effort and as you point out, a pain.

Hamish

martinl · September 3, 2007, 1:40pm

Hi Michael,

2007/9/2, Michael Barton <Michael.Barton@asu.edu>:

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using subprocess.Popen)
to report an error when there isn't one.

mapset = "PERMANENT"
location = "spearfish60"

mapsetCmd = cmd.Command(['g.mapset', 'mapset=%s' % mapset,
'location=%s' % location, '--q'])

print "%d" % mapsetCmd.returncode

# 0 success
# 1 failure

?

Martin
--
Martin Landa <landa.martin@gmail.com> * http://gama.fsv.cvut.cz/~landa *

Glynn_Clements1 · September 3, 2007, 2:22pm

Hamish wrote:

In the above case of georect.tcl, we may ask 1) if switching between
mapsets in the script is really the best solution; 2) if that warning
should be a warning.

3) If g.mapset is the right way to switch mapsets, or whether gis.m
should generate a temporary GISRC and "set env(GISRC) $tmp_gisrc".

If you're going to modify the existing $GISRC with g.mapset, the user
had better not be running GRASS commands on the terminal in the
meantime.

--
Glynn Clements <glynn@gclements.plus.com>

cmbarton · September 3, 2007, 4:52pm

The problem in this and other cases is that the warning is sent to stderr
when the command operates and exits properly. Error messages are sent to
stderr when the command does not operate properly and/or exits with an
error. There is no easy way (maybe no way in many cases) for a script to
tell the difference.

The result is that an "error" is generated both when the command operates
correctly AND incorrectly. The choice is to display an error message every
time the command runs (ranging from annoying to incapacitating) OR to send
them off to devnull limbo where a user will rarely if ever see a REAL error
message.

While admitedly useful for command line use, an informational message (i.e.,
not a value returned) that is sent by a module when it operates correctly is
completely pointless in a scripting context. When the message is sent to the
output channel that is also used to inform users about a real error, it is a
problem too. Why can't quiet mean quiet since it is an option? Or can there
be a 'super quiet' that really means quiet?

For scripting purposes, the ideal behavior of a module would be to 1) return
a value in an easily parsable form (e.g., key=val\nkey=val\n...) when asked
to do so, 2) return a clear and easily understandable error message that
could help a user/developer when an error occurs; and 3) completely silently
go about its business in all other circumstances.

Michael

On 9/3/07 3:58 AM, "Hamish" <hamish_nospam@yahoo.com> wrote:

Michael Barton wrote:

I'm trying to use g.mapset in the georectifier module for wxgrass. I
have to switch back and forth between locations and mapsets. For TclTk,
I used g.gisenv, but thought it would be more convenient to use
g.mapset here.

BUT, I've hit the perennial problem of a GRASS command that won't be
quiet. Regardless of the --q switch, g.mapset insists on dumping a
warning to stderr.

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using
subprocess.Popen) to report an error when there isn't one.

..

This might be a useful warning for command line users to get. But for
scripting, I don¹t want to see ANY warnings or other extraneous output.

Warnings and fatal error messages are not affected by --quiet by design;
they are not optional and need to be passed on to the user.

With any module you can and will get warnings or fatal error messages,
and any GUI wrapper will need to deal with them somehow.

In the above case of georect.tcl, we may ask 1) if switching between
mapsets in the script is really the best solution; 2) if that warning
should be a warning. One might argue that in this case it should use
G_important_message(), not G_warning(), but the warning message reads
quite well IMO. G_important_message("WARNING: ..."); ? Seems kinda
circuitous to me.

It does nothing except lock up scripts that don't make special
arrangements to parse such stuff.

Because the calling wrapper is deficient, not the concept of warnings and
error messages. All scripts must make special arrangements to parse
such stuff. This can be aided by a wrapper fn, in the same way as the Tcl
GUI menu has different 'spawn' module launchers. The cure is to fix the
wrapper -- not hide the errors. It is fundamentally wrong to assume that
anything written to stderr automatically means a problem is fatal, and
the GUI code needs to be written with that in mind, even if it is a pain.

It would make life so much easier for scripting if we could optionally
turn off ALL returned message except stdout for commands that are
supposed to return a value and stderr for real errors.

Warnings and fatal errors are "real errors". I guess one problem lies in
the concept of the warning. Is it a warning that the user is sailing in
dangerous waters? or is it a warning that the binary executable module
did something unexpected, but non-fatal? (I'd argue both are needed). Is
it a line item to be logged in GIS_ERROR_LOG or a console? or is it a
standalone pop-up GUI message? (I'd argue that the GUI needs a console).

Note in the C API there is G_suppress_warnings() (in lib/gis/error.c),
/*!
* \fn int G_suppress_warnings (int flag)
*
* \brief Suppress printing a warning message to stderr
*
* \param[in] flag a warning message will be suppressed if non-zero value
is given
* \return previous flag
*/

Use that at your own peril, it is likely that it will accidentally cause
more trouble than it solves.

... so instead of lots of open and catch statements, I'd suggest that any
call to a module be done using a standard wrapping function, and that
standard wrapping function take care of robust open/close & error
handling, once & correctly. I don't think there's another good way.
Duplicating robust error handling for all external calls is wasted
duplicated effort and as you point out, a pain.

Hamish

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

cmbarton · September 3, 2007, 5:01pm

On 9/3/07 7:22 AM, "Glynn Clements" <glynn@gclements.plus.com> wrote:

Hamish wrote:

In the above case of georect.tcl, we may ask 1) if switching between
mapsets in the script is really the best solution; 2) if that warning
should be a warning.

3) If g.mapset is the right way to switch mapsets, or whether gis.m
should generate a temporary GISRC and "set env(GISRC) $tmp_gisrc".

If you're going to modify the existing $GISRC with g.mapset, the user
had better not be running GRASS commands on the terminal in the
meantime.

Because of the way GRASS works, it is necessary to switch between mapsets.
i.e., a display command will only work with the current environment and an
accessible mapset. Any operation that changes something (e.g., i.group) will
only work in the current mapset.

However, with more experimentation, it looks like g.mapset won't work as a
way to do this. I've had to revert to using g.gisenv
set=LOCATION_NAME=[xy_location].

Michael

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Paul_Kelly · September 3, 2007, 5:06pm

On Mon, 3 Sep 2007, Michael Barton wrote:

The problem in this and other cases is that the warning is sent to stderr
when the command operates and exits properly. Error messages are sent to
stderr when the command does not operate properly and/or exits with an
error. There is no easy way (maybe no way in many cases) for a script to
tell the difference.

I think Tcl's deficiencies are confusing you. Tcl cannot tell the difference and this is IMHO a major bug in Tcl. But it is a single exception. Operating systems and probably almost all other scripting languages use the exit code of the program: a zero exit code indicates it exited successfully; a non-zero exit code indicates it exited with an error.

In any case Maris seems to have found a workaround to Tcl's deficiency: check the global variable errorcode after running an external command. I'm waiting to see if anybody comes up with a problem with this approach.

cmbarton · September 3, 2007, 5:19pm

On 9/3/07 3:58 AM, "Hamish" <hamish_nospam@yahoo.com> wrote:

It does nothing except lock up scripts that don't make special
arrangements to parse such stuff.

Because the calling wrapper is deficient, not the concept of warnings and
error messages. All scripts must make special arrangements to parse
such stuff. This can be aided by a wrapper fn, in the same way as the Tcl
GUI menu has different 'spawn' module launchers. The cure is to fix the
wrapper -- not hide the errors. It is fundamentally wrong to assume that
anything written to stderr automatically means a problem is fatal, and
the GUI code needs to be written with that in mind, even if it is a pain.

It would make life so much easier for scripting if we could optionally
turn off ALL returned message except stdout for commands that are
supposed to return a value and stderr for real errors.

Warnings and fatal errors are "real errors". I guess one problem lies in
the concept of the warning. Is it a warning that the user is sailing in
dangerous waters? or is it a warning that the binary executable module
did something unexpected, but non-fatal? (I'd argue both are needed). Is
it a line item to be logged in GIS_ERROR_LOG or a console? or is it a
standalone pop-up GUI message? (I'd argue that the GUI needs a console).

The GUI has a console. Errors were being routed to the console, but overall
it is a better user experience to have real errors showing up in
GUI-appropriate message boxes than expect a user to search for it in a
terminal or other console. This also doesn't solve all problems of warnings
being mistaken for real errors.

Note in the C API there is G_suppress_warnings() (in lib/gis/error.c),
/*!
* \fn int G_suppress_warnings (int flag)
*
* \brief Suppress printing a warning message to stderr
*
* \param[in] flag a warning message will be suppressed if non-zero value
is given
* \return previous flag
*/

Use that at your own peril, it is likely that it will accidentally cause
more trouble than it solves.

I don't disagree with these concepts. However, the difference is between a
message that is returned when a module operates as designed (regardless of
whether a user might be venturing into dangerous territory or not) or does
not operated as designed. When buried inside a complex script that chains
together multiple commands, something that causes a module to NOT operate as
designed (an error to me at least) can result in unexpected or undesired
behavior or crash the whole script. This needs to be reported 'outside' the
script so that the user can learn what is going wrong 'inside'. But when the
module is working as designed, sending a message is pointless. If it is a
message that triggers an error trap that requires the user to press an OK
button the message itself can bring down the whole script.

... so instead of lots of open and catch statements, I'd suggest that any
call to a module be done using a standard wrapping function, and that
standard wrapping function take care of robust open/close & error
handling, once & correctly. I don't think there's another good way.
Duplicating robust error handling for all external calls is wasted
duplicated effort and as you point out, a pain.

I definitely agree. This is what has been done with wxgrass. Martin created
a cmd.Command function that will execute a GRASS command using
subprocess.Popen and do a lot of this housekeeping. It doesn't completely
get rid of the need to individually trap potential errors with the try
function, but it helps a lot.

However, this STILL hits the same problem with warnings. Since they come in
through stderr, they apparently produce a non-zero return value for Popen.
This triggers the error management routines--as it should--raising a message
box that says g.mapset failed when it in fact did not.

Michael

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Glynn_Clements1 · September 3, 2007, 7:21pm

Michael Barton wrote:

>> In the above case of georect.tcl, we may ask 1) if switching between
>> mapsets in the script is really the best solution; 2) if that warning
>> should be a warning.
>
> 3) If g.mapset is the right way to switch mapsets, or whether gis.m
> should generate a temporary GISRC and "set env(GISRC) $tmp_gisrc".
>
> If you're going to modify the existing $GISRC with g.mapset, the user
> had better not be running GRASS commands on the terminal in the
> meantime.

Because of the way GRASS works, it is necessary to switch between mapsets.
i.e., a display command will only work with the current environment and an
accessible mapset. Any operation that changes something (e.g., i.group) will
only work in the current mapset.

However, with more experimentation, it looks like g.mapset won't work as a
way to do this. I've had to revert to using g.gisenv
set=LOCATION_NAME=[xy_location].

I'm not questioning whether it's necessary to switch mapsets. I'm
questioning whether modifying the existing $GISRC is the right
approach, or whether gis.m should create a new $GISRC solely for use
by the georectifier.

Personally, I'd suggest the latter.

In general, global state (e.g. $GISRC, WIND) is undesirable. For
command-line use, it's a necessary compromise, as having to manually
specify database/location/mapset (and others, e.g. monitor and
database) for each command would be a major nuisance.

A GUI doesn't have this problem; it can maintain its own state.
Moreover, it can maintain as many different states as it wishes, and
use the appropriate one for each individual command.

See also: past discussions of $WIND_OVERRIDE and $GRASS_REGION vs the
WIND file.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · September 3, 2007, 7:36pm

Michael Barton wrote:

I'm trying to use g.mapset in the georectifier module for wxgrass. I have to
switch back and forth between locations and mapsets. For TclTk, I used
g.gisenv, but thought it would be more convenient to use g.mapset here.

BUT, I've hit the perennial problem of a GRASS command that won't be quiet.
Regardless of the --q switch, g.mapset insists on dumping a warning to
stderr.

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using subprocess.Popen)
to report an error when there isn't one.

(Martin, this seems to be a general issue with cmd.Command. Is there a way
to work around this for commands that insist on dumping extraneous stuff to
stderr?)

If cmd.Command considers writing to stderr to constitute an error,
that's a bug in cmd.Command.

That behaviour of Tcl's "exec" is a significant design flaw; it should
not be intentionally replicated by wxGRASS.

This might be a useful warning for command line users to get. But for
scripting, I don¹t want to see ANY warnings or other extraneous output.

Many of the warnings which commands generate are quite important, and
shouldn't be hidden for the sake of convenience.

The g.mapset warning is of debatable usefulness (in general; it's
entirely meaningless for the GUI), but there are plenty of others
which are useful.

It does nothing except lock up scripts that don't make special arrangements to
parse such stuff.

Scripts should not be attempting to parse stderr. The only reasonable
ways to handle stderr output are:

1. Show it to the user.
2. Log it to a file.
3. Discard it, or otherwise ignore it.

This is a real problem for TclTk, leaving us with the
necessity of sending all stderr to dev/null so that we only get back error
messages sent by TclTk.

That's a deficiency of Tcl's process management primitives
("primitive" being the operative term).

Even for languages like wxPython that can separate
stderr from stdout, it means that attempts to gracefully trap and display
errors will incorrectly show an error when something like this happens.

Only if the code mimic's Tcl's bogus concept of what constitutes an
error (i.e. stderr output).

Writing to stderr is *not* an error. Returning a non-zero exit code is
an error.

It
would make life so much easier for scripting if we could optionally turn off
ALL returned message except stdout for commands that are supposed to return
a value and stderr for real errors.

That isn't going to happen.

Apart from the amount of work involved in modifying GRASS, many
library functions can write warnings (and other messages) to stderr,
and don't necessarily provide any mechanism to control whether this
happens, or even to detect that it has happened.

In general terms, stderr is a mechanism for programs to communicate
arbitrary information to the *user*. Programs should not get involved
with reading stderr unless they are doing so in order to act as a
"common carrier" between the program and the user (i.e. to simply pass
the information along).

--
Glynn Clements <glynn@gclements.plus.com>

cmbarton · September 3, 2007, 7:38pm

On 9/3/07 12:21 PM, "Glynn Clements" <glynn@gclements.plus.com> wrote:

Michael Barton wrote:

In the above case of georect.tcl, we may ask 1) if switching between
mapsets in the script is really the best solution; 2) if that warning
should be a warning.

3) If g.mapset is the right way to switch mapsets, or whether gis.m
should generate a temporary GISRC and "set env(GISRC) $tmp_gisrc".

If you're going to modify the existing $GISRC with g.mapset, the user
had better not be running GRASS commands on the terminal in the
meantime.

Because of the way GRASS works, it is necessary to switch between mapsets.
i.e., a display command will only work with the current environment and an
accessible mapset. Any operation that changes something (e.g., i.group) will
only work in the current mapset.

However, with more experimentation, it looks like g.mapset won't work as a
way to do this. I've had to revert to using g.gisenv
set=LOCATION_NAME=[xy_location].

I'm not questioning whether it's necessary to switch mapsets. I'm
questioning whether modifying the existing $GISRC is the right
approach, or whether gis.m should create a new $GISRC solely for use
by the georectifier.

Personally, I'd suggest the latter.

OK. I see what you're getting at now. I'm not sure how to accomplish it.

The active GISRC is a tempfile ( /tmp/grass6-cmbarton-6824/gisrc on my
machine).

Will simply creating a new temporary file with different values for
LOCATION_NAME and MAPSET have the effect of 'switching' to a different
working location/mapset? If so, how to get the current GRASS session to
recognize this?

Does it even have to be a text file on disk somewhere or can it simply be
set to a variable (e.g., Python data object) with the appropriate values?

Can I just make a minimal GISRC with only location and mapset, or is it
safer to copy whatever is in the currently active one and then modify
location and mapset.

My active one currently contains...

DIGITIZER: none
GISDBASE: /Users/Shared/grassdata
GRASS_DB_ENCODING: utf-8
GRASS_ADDON_PATH:
/Applications/Grass/GRASS-6.3.app/Contents/Resources/etc/gm/script
MONITOR: x0
LOCATION_NAME: xy
MAPSET: test
GRASS_RENDER_IMMEDIATE: TRUE
GRASS_GUI: wx

Michael

In general, global state (e.g. $GISRC, WIND) is undesirable. For
command-line use, it's a necessary compromise, as having to manually
specify database/location/mapset (and others, e.g. monitor and
database) for each command would be a major nuisance.

A GUI doesn't have this problem; it can maintain its own state.
Moreover, it can maintain as many different states as it wishes, and
use the appropriate one for each individual command.

See also: past discussions of $WIND_OVERRIDE and $GRASS_REGION vs the
WIND file.

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

cmbarton · September 3, 2007, 7:52pm

So what is *supposed* to happen in a scripting environment is...

1) non-zero exit code returned
2) show whatever is in stderr to the user

OR

1) 0 exit code returned
2) do not report with whatever is in stderr (not because it is unimportant,
but because it is not an error)

Correct?

If so, then there may be a bug somewhere in cmd.Comand since it is showing
non-meaningful error messages sometimes when the command operates fine.

An aside: g.mapset treats as an error (It says "ERROR", so I assume that it
generates a non-zero exit code) a situation where the location and mapset
are the current location and mapset. IMHO, this should not be an error. I
guess it could be a "warning", but the module is operating correctly in that
it is 'changing' to the location and mapset specified; these just happen to
be the current one.

Michael

On 9/3/07 12:36 PM, "Glynn Clements" <glynn@gclements.plus.com> wrote:

Michael Barton wrote:

I'm trying to use g.mapset in the georectifier module for wxgrass. I have to
switch back and forth between locations and mapsets. For TclTk, I used
g.gisenv, but thought it would be more convenient to use g.mapset here.

BUT, I've hit the perennial problem of a GRASS command that won't be quiet.
Regardless of the --q switch, g.mapset insists on dumping a warning to
stderr.

g.mapset mapset=PERMANENT location=Spearfish60_test --q
WARNING: Your shell continues to use the history for the old mapset.

This causes cmd.Command (wxgrass command processor using subprocess.Popen)
to report an error when there isn't one.

(Martin, this seems to be a general issue with cmd.Command. Is there a way
to work around this for commands that insist on dumping extraneous stuff to
stderr?)

If cmd.Command considers writing to stderr to constitute an error,
that's a bug in cmd.Command.

That behaviour of Tcl's "exec" is a significant design flaw; it should
not be intentionally replicated by wxGRASS.

This might be a useful warning for command line users to get. But for
scripting, I don¹t want to see ANY warnings or other extraneous output.

Many of the warnings which commands generate are quite important, and
shouldn't be hidden for the sake of convenience.

The g.mapset warning is of debatable usefulness (in general; it's
entirely meaningless for the GUI), but there are plenty of others
which are useful.

It does nothing except lock up scripts that don't make special arrangements
to
parse such stuff.

Scripts should not be attempting to parse stderr. The only reasonable
ways to handle stderr output are:

1. Show it to the user.
2. Log it to a file.
3. Discard it, or otherwise ignore it.

This is a real problem for TclTk, leaving us with the
necessity of sending all stderr to dev/null so that we only get back error
messages sent by TclTk.

That's a deficiency of Tcl's process management primitives
("primitive" being the operative term).

Even for languages like wxPython that can separate
stderr from stdout, it means that attempts to gracefully trap and display
errors will incorrectly show an error when something like this happens.

Only if the code mimic's Tcl's bogus concept of what constitutes an
error (i.e. stderr output).

Writing to stderr is *not* an error. Returning a non-zero exit code is
an error.

It
would make life so much easier for scripting if we could optionally turn off
ALL returned message except stdout for commands that are supposed to return
a value and stderr for real errors.

That isn't going to happen.

Apart from the amount of work involved in modifying GRASS, many
library functions can write warnings (and other messages) to stderr,
and don't necessarily provide any mechanism to control whether this
happens, or even to detect that it has happened.

In general terms, stderr is a mechanism for programs to communicate
arbitrary information to the *user*. Programs should not get involved
with reading stderr unless they are doing so in order to act as a
"common carrier" between the program and the user (i.e. to simply pass
the information along).

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

H_B · September 4, 2007, 2:00am

Michael Barton wrote:

An aside: g.mapset treats as an error (It says "ERROR", so I assume
that it generates a non-zero exit code)

it does: general/g.mapset/main.c
    /* TODO: this should be checked better (repeated '/' etc.) */
    if ( strcmp(mapset_old_path, mapset_new_path) == 0 )
        G_fatal_error ( _("%s is already the current mapset"),
mapset_new );

a situation where the location and mapset are the current location and
mapset. IMHO, this should not be an error. I guess it could be a
"warning", but the module is operating correctly in that it is
'changing' to the location and mapset specified; these just happen to
be the current one.

when the rectifying target is the current mapset, can you test for that
and just not run g.mapset at all? The fewer times g.mapset needs to be
called the better..

Hamish

cmbarton · September 4, 2007, 2:36am

On 9/3/07 7:00 PM, "Hamish" <hamish_nospam@yahoo.com> wrote:

when the rectifying target is the current mapset, can you test for that
and just not run g.mapset at all? The fewer times g.mapset needs to be
called the better..

I can test for this, but it's easier and cleaner to just make the change to
whatever the user wants.

For the georectifier, this is moot now, since g.mapset won't work. But I
guess you could consider it a general question/suggestion. It doesn't seem
to be an error if g.mapset does what it is supposed to and simply stays in
the current location/mapset because that is what the user instructed it to
do.

FWIW, here is what is going on with g.mapset (error/warning reporting
aside).

There is a custom selection tool that uses g.mapsets -p and g.mlist (I'd
prefer g.list, but need to have it output a simple, single column list) to
create a pull-down combo box with a hierarchical list of selectable maps
within each accessible mapset.

However, in spite of calling g.mapset before repopulating the combo box list
of values, it only shows maps in the old location/mapset. I don't know
whether this is a function of the object-oriented nature of Python (initial
instantiation of the control is somehow locked into the GISRC values that it
is called with) or something about the GRASS modules. I suspect the latter,
however, because when I use g.gisenv to reset location and mapset it all
works fine.

Glynn is suggesting creating a temporary GISRC file for all this. I'm not
yet sure how to go about this, but it should be pretty easy to plug in since
the location/mapset switch routines are all within a method that is called
as needed.

Thanks for the comments and thoughts

Michael
__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Glynn_Clements1 · September 4, 2007, 10:32am

Michael Barton wrote:

> I'm not questioning whether it's necessary to switch mapsets. I'm
> questioning whether modifying the existing $GISRC is the right
> approach, or whether gis.m should create a new $GISRC solely for use
> by the georectifier.
>
> Personally, I'd suggest the latter.

OK. I see what you're getting at now. I'm not sure how to accomplish it.

The active GISRC is a tempfile ( /tmp/grass6-cmbarton-6824/gisrc on my
machine).

Will simply creating a new temporary file with different values for
LOCATION_NAME and MAPSET have the effect of 'switching' to a different
working location/mapset?

You also need to change $GISRC to refer to the new file.

If so, how to get the current GRASS session to
recognize this?

You don't want the "current GRASS session" to recognise it. What you
want is for the commands run from the georectifier (and nothing else)
to recognise it.

In gis.m, "set env(GISRC) $tmp_gisrc" before calling such commands,
then set it back afterwards.

In Python, change GISRC in the environment passed to Popen() for such
commands, e.g.:

  env = os.environ.copy()
  env["GISRC"] = tmp_gisrc
  child = subprocess.Popen(cmd, env = env, ...)

Does it even have to be a text file on disk somewhere or can it simply be
set to a variable (e.g., Python data object) with the appropriate values?

It needs to be a text file.

Can I just make a minimal GISRC with only location and mapset, or is it
safer to copy whatever is in the currently active one and then modify
location and mapset.

I suggest the latter. If creating a new one, you need at least
GISDBASE, LOCATION_NAME and MAPSET for all commands. The others should
be optional.

My active one currently contains...

DIGITIZER: none
GISDBASE: /Users/Shared/grassdata
GRASS_DB_ENCODING: utf-8
GRASS_ADDON_PATH: /Applications/Grass/GRASS-6.3.app/Contents/Resources/etc/gm/script
MONITOR: x0
LOCATION_NAME: xy
MAPSET: test
GRASS_RENDER_IMMEDIATE: TRUE
GRASS_GUI: wx

[I don't know how the GRASS_RENDER_IMMEDIATE setting got there; that's
an environment variable.]

I have:

GISDBASE: /opt/grass-data
LOCATION_NAME: spearfish57
MAPSET: glynn
GRASS_DB_ENCODING: ascii
GRASS_GUI: text

GRASS_DB_ENCODING is only used by the form library. GRASS_GUI is used
to remember the -text/-gui/etc setting between sessions and to
determine whether --ui uses Tcl/Tk or wxPython. Neither of these are
relevant for non-interactive commands.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · September 4, 2007, 10:48am

Michael Barton wrote:

So what is *supposed* to happen in a scripting environment is...

1) non-zero exit code returned
2) show whatever is in stderr to the user

OR

1) 0 exit code returned
2) do not report with whatever is in stderr (not because it is unimportant,
but because it is not an error)

Correct?

A GUI for GRASS should generally show everything written to both
stderr and stdout (unless explicitly redirected) to the user,
regardless of whether or not the command succeeds. IOW, you need a log
window.

If a command succeeds but generates warnings, the user should be aware
of those warnings. It's not uncommon for a command to be able to
produce results yet have doubts as to whether the result is what the
user actually wanted.

In the case of of a non-zero exit code, the output from stderr is
likely to be useful as part of any error message. If you're going to
display an error in a message box, the stderr output may be useful.

If so, then there may be a bug somewhere in cmd.Comand since it is showing
non-meaningful error messages sometimes when the command operates fine.

It isn't beyond the bounds of possibility that some commands return
incorrect exit codes.

In particular, if a script completes by reaching EOF (rather than
using "exit"), its exit code will be that of the last command, which
doesn't necessarily reflect the success or failure of the script as a
whole.

[Also, some commands have a notion of failure which isn't limited to
errors. E.g. "grep" fails if nothing matches; 0 => match(es) found,
1 => no matches found, 2 => actual error (file not found, etc).]

An aside: g.mapset treats as an error (It says "ERROR", so I assume that it
generates a non-zero exit code) a situation where the location and mapset
are the current location and mapset. IMHO, this should not be an error.

Agreed.

I guess it could be a "warning", but the module is operating correctly in that
it is 'changing' to the location and mapset specified; these just happen to
be the current one.

This is similar to the discussions over whether g.remove "fails" if
the entity (map etc) doesn't exist. For comparison, "rm" fails if the
specified file doesn't exist whereas "rm -f" doesn't (it fails if the
file exists but cannot be removed, e.g. due to permissions).

--
Glynn Clements <glynn@gclements.plus.com>

cmbarton · September 4, 2007, 3:19pm

Thanks much for the explanation. I'll try this out with the wxPython
georectifier I'm working on now.

Michael

On 9/4/07 3:32 AM, "Glynn Clements" <glynn@gclements.plus.com> wrote:

Michael Barton wrote:

I'm not questioning whether it's necessary to switch mapsets. I'm
questioning whether modifying the existing $GISRC is the right
approach, or whether gis.m should create a new $GISRC solely for use
by the georectifier.

Personally, I'd suggest the latter.

OK. I see what you're getting at now. I'm not sure how to accomplish it.

The active GISRC is a tempfile ( /tmp/grass6-cmbarton-6824/gisrc on my
machine).

Will simply creating a new temporary file with different values for
LOCATION_NAME and MAPSET have the effect of 'switching' to a different
working location/mapset?

You also need to change $GISRC to refer to the new file.

If so, how to get the current GRASS session to
recognize this?

You don't want the "current GRASS session" to recognise it. What you
want is for the commands run from the georectifier (and nothing else)
to recognise it.

In gis.m, "set env(GISRC) $tmp_gisrc" before calling such commands,
then set it back afterwards.

In Python, change GISRC in the environment passed to Popen() for such
commands, e.g.:

env = os.environ.copy()
env["GISRC"] = tmp_gisrc
child = subprocess.Popen(cmd, env = env, ...)

Does it even have to be a text file on disk somewhere or can it simply be
set to a variable (e.g., Python data object) with the appropriate values?

It needs to be a text file.

Can I just make a minimal GISRC with only location and mapset, or is it
safer to copy whatever is in the currently active one and then modify
location and mapset.

I suggest the latter. If creating a new one, you need at least
GISDBASE, LOCATION_NAME and MAPSET for all commands. The others should
be optional.

My active one currently contains...

DIGITIZER: none
GISDBASE: /Users/Shared/grassdata
GRASS_DB_ENCODING: utf-8
GRASS_ADDON_PATH:
/Applications/Grass/GRASS-6.3.app/Contents/Resources/etc/gm/script
MONITOR: x0
LOCATION_NAME: xy
MAPSET: test
GRASS_RENDER_IMMEDIATE: TRUE
GRASS_GUI: wx

[I don't know how the GRASS_RENDER_IMMEDIATE setting got there; that's
an environment variable.]

I have:

GISDBASE: /opt/grass-data
LOCATION_NAME: spearfish57
MAPSET: glynn
GRASS_DB_ENCODING: ascii
GRASS_GUI: text

GRASS_DB_ENCODING is only used by the form library. GRASS_GUI is used
to remember the -text/-gui/etc setting between sessions and to
determine whether --ui uses Tcl/Tk or wxPython. Neither of these are
relevant for non-interactive commands.

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

cmbarton · September 4, 2007, 3:25pm

Thanks for the clarification. Hopefully NOW I understand how this is
SUPPOSED to work. It will be particularly helpful with wxPython. At least
now I know to check for an incorrect exit code if gcmd.Command doesn't work
as expected. It is also not producing useful error messages. Perhaps it is
not parsing stderror correctly.

Michael

On 9/4/07 3:48 AM, "Glynn Clements" <glynn@gclements.plus.com> wrote:

Michael Barton wrote:

So what is *supposed* to happen in a scripting environment is...

1) non-zero exit code returned
2) show whatever is in stderr to the user

OR

1) 0 exit code returned
2) do not report with whatever is in stderr (not because it is unimportant,
but because it is not an error)

Correct?

A GUI for GRASS should generally show everything written to both
stderr and stdout (unless explicitly redirected) to the user,
regardless of whether or not the command succeeds. IOW, you need a log
window.

If a command succeeds but generates warnings, the user should be aware
of those warnings. It's not uncommon for a command to be able to
produce results yet have doubts as to whether the result is what the
user actually wanted.

In the case of of a non-zero exit code, the output from stderr is
likely to be useful as part of any error message. If you're going to
display an error in a message box, the stderr output may be useful.

If so, then there may be a bug somewhere in cmd.Comand since it is showing
non-meaningful error messages sometimes when the command operates fine.

It isn't beyond the bounds of possibility that some commands return
incorrect exit codes.

In particular, if a script completes by reaching EOF (rather than
using "exit"), its exit code will be that of the last command, which
doesn't necessarily reflect the success or failure of the script as a
whole.

[Also, some commands have a notion of failure which isn't limited to
errors. E.g. "grep" fails if nothing matches; 0 => match(es) found,
1 => no matches found, 2 => actual error (file not found, etc).]

An aside: g.mapset treats as an error (It says "ERROR", so I assume that it
generates a non-zero exit code) a situation where the location and mapset
are the current location and mapset. IMHO, this should not be an error.

Agreed.

I guess it could be a "warning", but the module is operating correctly in
that
it is 'changing' to the location and mapset specified; these just happen to
be the current one.

This is similar to the discussions over whether g.remove "fails" if
the entity (map etc) doesn't exist. For comparison, "rm" fails if the
specified file doesn't exist whereas "rm -f" doesn't (it fails if the
file exists but cannot be removed, e.g. due to permissions).

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

cmbarton · September 5, 2007, 3:55am

On 9/3/07 10:06 AM, "Paul Kelly" <paul-grass@stjohnspoint.co.uk> wrote:

On Mon, 3 Sep 2007, Michael Barton wrote:

The problem in this and other cases is that the warning is sent to stderr
when the command operates and exits properly. Error messages are sent to
stderr when the command does not operate properly and/or exits with an
error. There is no easy way (maybe no way in many cases) for a script to
tell the difference.

I think Tcl's deficiencies are confusing you. Tcl cannot tell the
difference and this is IMHO a major bug in Tcl. But it is a single
exception. Operating systems and probably almost all other scripting
languages use the exit code of the program: a zero exit code indicates it
exited successfully; a non-zero exit code indicates it exited with an
error.

You're probably right. I'm trying to work out a reasonable error trap
function. Not very difficult in general, though replacing all the individual
error traps will be tedious.

In any case Maris seems to have found a workaround to Tcl's deficiency:
check the global variable errorcode after running an external command. I'm
waiting to see if anybody comes up with a problem with this approach.

I looked at this and tried it out. I found that it doesn't work as expected.
The reason is that all or most of the error codes returned by Tcl seem to
potentially be ones we'd need to trap. See docs at
<ActiveState Community - Boosting coder and team productivity with ready-to-use open source languages and tools.
M18>.

For example, I tried giving an incorrect module name in a menu to get a file
not found error. In this case, Tcl_errorCode returns POSIX rather than
CHILDSTATUS

The generic error trap will allow for global formatting change in the
future, but I don't see any way around the confusion of real error messages
and other messages sent to stderr in Tcltk.

Michael

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton