[GRASS-dev] Incorrect stuff coming through stderr?

One of my colleagues just noticed that the information below is being
written to stderr from r.in.ascii.

It seems to us that this is incorrect, and should be directed to stdout or
something else.

Michael

___________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

WWW - http://www.public.asu.edu/~cmbarton
Phone: 480-965-6262
Fax: 480-965-7671

------ Forwarded Message
From: <Gary.Mayer@asu.edu>
Date: Thu, 31 Aug 2006 01:28:47 +0000
To: <Michael.Barton@asu.edu>
Subject: Std err

I don't know it it will paste correctly, but this is what I was talking
about:

0% 4% 8% 12% 16% 20% 24%
28% 32% 36% 40% 44% 48% 52%
56% 60% 64% 68% 72% 76% 80%
84% 88% 92% 96% 100%
CREATING SUPPORT FILES FOR managed.2.6

It occurs when I create the new maps using r.in.ascii and is sent via
standard error.

- Gary

------ End of Forwarded Message

Can't confirm on 6.3cvs. GUI shows nice progress bar but CLI prints x% as it
should be.

Only place in question for r.in.ascii is at file gethead.c:301

  fprintf (stderr, " %s\n", msg);

As this is some kind of error printing routine, this could be converted to
G_warning atleast.

Oh, btw, why r.in.ascii lacks license information? No author, no GPL?

Just my 0.01,
Maris.

On Thursday 31 August 2006 05:44, Michael Barton wrote:

One of my colleagues just noticed that the information below is being
written to stderr from r.in.ascii.

It seems to us that this is incorrect, and should be directed to stdout or
something else.

Michael

___________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

WWW - http://www.public.asu.edu/~cmbarton
Phone: 480-965-6262
Fax: 480-965-7671

------ Forwarded Message
From: <Gary.Mayer@asu.edu>
Date: Thu, 31 Aug 2006 01:28:47 +0000
To: <Michael.Barton@asu.edu>
Subject: Std err

I don't know it it will paste correctly, but this is what I was talking
about:

0% 4% 8% 12% 16% 20% 24%
28% 32% 36% 40% 44% 48% 52%
56% 60% 64% 68% 72% 76% 80%
84% 88% 92% 96% 100%
CREATING SUPPORT FILES FOR managed.2.6

It occurs when I create the new maps using r.in.ascii and is sent via
standard error.

- Gary

------ End of Forwarded Message

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

Ma-ris Nartišs wrote on 08/31/2006 08:53 AM:
...

Oh, btw, why r.in.ascii lacks license information? No author, no GPL?

Just my 0.01,
Maris.
  

Many files lack that. Unfortunately some developers even commit *new* code
without adding the agreed header (as described in SUBMITTING). This is
strange
since there were strong opinions against a potential MIT/X11 or BSD
"takeover" of
the code or whatever... At least the main.c should carry such header.

To help: there is a nice tool available:
tools/copywrite.pl

## Script to easily add missing copyright statements in
## GRASS source code files
##
## The script searches and opens sequentially all files
## which lack the word "copyright". Additionally the local
## ChangeLog is fetched and contributor names are extracted.
## Then a new header is inserted containing the GPL statement,
## purpose template and authors.
##
## Please run from top source code directory.
## For usage details, see below.
##
## 2006, written by Schuyler Erle <schuyler , nocat net>

Run this script as (have a look at the instructions within the script
first):
perl copywrite.pl contributors.csv

It is needed to check 'description.html' for the first author which
doesn't appear in the CVS Changelog in case the module is older
than 31 Dec. 1999 (the day when we started to maintain the code in CVS).
Better to always check 'description.html'.

Markus

PS: r.in.ascii was originally written by Michael Shapiro/CERL.

Maris:

> Oh, btw, why r.in.ascii lacks license information? No author, no
> GPL?

Markus:

Many files lack that. Unfortunately some developers even commit *new*
code without adding the agreed header (as described in SUBMITTING).
This is strange since there were strong opinions against a potential
MIT/X11 or BSD "takeover" of the code or whatever... At least the
main.c should carry such header.

To help: there is a nice tool available:
tools/copywrite.pl

..

Run this script as (have a look at the instructions within the script
first):
perl copywrite.pl contributors.csv

It is needed to check 'description.html' for the first author which
doesn't appear in the CVS Changelog in case the module is older
than 31 Dec. 1999 (the day when we started to maintain the code in
CVS). Better to always check 'description.html'.

And check the GRASS 5.4 version of 'description.html' !
ie html/html/g.module.html

In addition to historical info there is a lot of useful help content
there which has never been ported to GRASS 6 along with the modules.

Hamish

Markus Neteler wrote:

It is needed to check 'description.html' for the first author which
doesn't appear in the CVS Changelog in case the module is older
than 31 Dec. 1999 (the day when we started to maintain the code in
CVS). Better to always check 'description.html'.

btw, if anyone isn't familiar with it, the web interface to the CVS is
great for following development history over the last 6.5 years:

http://freegis.org/cgi-bin/viewcvs.cgi/#dirlist

Can we arbitrarily add GPL notices to [public domain] code from pre-GPL
versions of GRASS? (I guess so, we can do whatever we want with it....)

Hamish

Hamish wrote on 08/31/2006 12:17 PM:

Markus Neteler wrote:
  

It is needed to check 'description.html' for the first author which
doesn't appear in the CVS Changelog in case the module is older
than 31 Dec. 1999 (the day when we started to maintain the code in
CVS). Better to always check 'description.html'.
    
btw, if anyone isn't familiar with it, the web interface to the CVS is
great for following development history over the last 6.5 years:

http://freegis.org/cgi-bin/viewcvs.cgi/#dirlist

Can we arbitrarily add GPL notices to [public domain] code from pre-GPL
versions of GRASS? (I guess so, we can do whatever we want with it....)
  

... as far as no 3rd party rights are infringed. But we had the major
pre-GPL cleanup
where lots of code was removed. So: arbitrarily I would not say, but
"at best knowledge" we have to do this code vetting. After 10/1999 people
contributed knowingly to a GPL project, so that could should be covered.
If in doubt, contact the author(s).

Markus

Michael Barton wrote:

One of my colleagues just noticed that the information below is being
written to stderr from r.in.ascii.

It seems to us that this is incorrect, and should be directed to stdout or
something else.

I don't know it it will paste correctly, but this is what I was talking
about:

0% 4% 8% 12% 16% 20% 24%
28% 32% 36% 40% 44% 48% 52%
56% 60% 64% 68% 72% 76% 80%
84% 88% 92% 96% 100%
CREATING SUPPORT FILES FOR managed.2.6

It occurs when I create the new maps using r.in.ascii and is sent via
standard error.

What's the problem? Progress messages are normal, and are supposed to
be written to stderr.

--
Glynn Clements <glynn@gclements.plus.com>

We are linking GRASS modules with a Java agent-based modeling environment.
The lack of consistency in GRASS output is an issue here, as it is with
trying to link GRASS with an interface building platform.

Because GRASS is a suite of command-line tools, it is an ideal platform to
be used by other platform for sophisticated modeling (Python, Java, etc) as
well as be wrapped in an interface platform to serve as a full-featured GIS.

However, GRASS was originally designed to be used by a person sitting at a
terminal, typing commands, and reading the output off the screen. That's why
there is output of the type produced by r.info and r.report. It's also why
it didn't matter about inconsistencies between the information provided by
r.display in different kinds of maps or between g.region -p and g.region -g

However, these become issues that range from annoyances (parsing r.info) to
impossibilities (parsing r.describe) when you try to use GRASS from another
environment than a user at a terminal.

There is nothing wrong with nicely formatted output (though it usually
depends on fixed-width fonts, and looks ugly otherwise). However, it would
be very useful (maybe even increasingly essential) that we also have a
standard, easy to parse output format that is consistent across all modules.
Standardized input is equally important, but we are in much better shape
there.

We already have a good start one kind of standardized output format, usually
called "shell-script style", but sometimes also called "terse" or other
things. This takes the format of "key=value" followed by a new line. It
would also be very helpful, however, to simply have commands return a list
of values in an order given in the command docs. So I'd like to propose the
following as a goal in future version of GRASS.

All commands that return values of some kind (i.e, that do not create or
modify maps) have the option to return ALL of their values (not just a
subset as is the case with g.region) in "shell-script" or "terse" format of
"key=value".

This option should be controlled by a standard flag for all modules for
consistency.

All commands that return values should also be able to return values as a
list of values "value value value...".

Again, this should be controlled by a standard flag for all modules.

... Snip snip ...

56% 60% 64% 68% 72% 76% 80%
84% 88% 92% 96% 100%
CREATING SUPPORT FILES FOR managed.2.6

It occurs when I create the new maps using r.in.ascii and is sent via
standard error.

What's the problem? Progress messages are normal, and are supposed to
be written to stderr.

It seems odd to me that progress messages are written to stderr. However, if
that is the norm across C programs, then I guess that's were they belong.
But does " CREATING SUPPORT FILES FOR managed.2.6" also go there?

We also need a "quiet" option for all commands to suppress all stdout except
the actual output values. There is a lot of verbage produced some GRASS
commands that might be nice for a user at a terminal to read, but that is
problematic when GRASS modules are being used in other environments. This is
available for some modules, but has variable effects (i.e., sometimes it
only suppresses some of the extraneous verbage).

I'm open to suggestion for how to modify this proposal. The objective is to
offer standardized output that would make it easier to use GRASS in
non-interactive environments.

Michael
__________________________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Michael, you may want to put this into wiki for GRASS7, it will get lost in the email list.
http://grass.gdf-hannover.de/wiki/GRASS_7_ideas_collection

Helena

On Sep 2, 2006, at 1:26 PM, Michael Barton wrote:

We are linking GRASS modules with a Java agent-based modeling environment.
The lack of consistency in GRASS output is an issue here, as it is with
trying to link GRASS with an interface building platform.

Because GRASS is a suite of command-line tools, it is an ideal platform to
be used by other platform for sophisticated modeling (Python, Java, etc) as
well as be wrapped in an interface platform to serve as a full-featured GIS.

However, GRASS was originally designed to be used by a person sitting at a
terminal, typing commands, and reading the output off the screen. That's why
there is output of the type produced by r.info and r.report. It's also why
it didn't matter about inconsistencies between the information provided by
r.display in different kinds of maps or between g.region -p and g.region -g

However, these become issues that range from annoyances (parsing r.info) to
impossibilities (parsing r.describe) when you try to use GRASS from another
environment than a user at a terminal.

There is nothing wrong with nicely formatted output (though it usually
depends on fixed-width fonts, and looks ugly otherwise). However, it would
be very useful (maybe even increasingly essential) that we also have a
standard, easy to parse output format that is consistent across all modules.
Standardized input is equally important, but we are in much better shape
there.

We already have a good start one kind of standardized output format, usually
called "shell-script style", but sometimes also called "terse" or other
things. This takes the format of "key=value" followed by a new line. It
would also be very helpful, however, to simply have commands return a list
of values in an order given in the command docs. So I'd like to propose the
following as a goal in future version of GRASS.

All commands that return values of some kind (i.e, that do not create or
modify maps) have the option to return ALL of their values (not just a
subset as is the case with g.region) in "shell-script" or "terse" format of
"key=value".

This option should be controlled by a standard flag for all modules for
consistency.

All commands that return values should also be able to return values as a
list of values "value value value...".

Again, this should be controlled by a standard flag for all modules.

... Snip snip ...

56% 60% 64% 68% 72% 76% 80%
84% 88% 92% 96% 100%
CREATING SUPPORT FILES FOR managed.2.6

It occurs when I create the new maps using r.in.ascii and is sent via
standard error.

What's the problem? Progress messages are normal, and are supposed to
be written to stderr.

It seems odd to me that progress messages are written to stderr. However, if
that is the norm across C programs, then I guess that's were they belong.
But does " CREATING SUPPORT FILES FOR managed.2.6" also go there?

We also need a "quiet" option for all commands to suppress all stdout except
the actual output values. There is a lot of verbage produced some GRASS
commands that might be nice for a user at a terminal to read, but that is
problematic when GRASS modules are being used in other environments. This is
available for some modules, but has variable effects (i.e., sometimes it
only suppresses some of the extraneous verbage).

I'm open to suggestion for how to modify this proposal. The objective is to
offer standardized output that would make it easier to use GRASS in
non-interactive environments.

Michael
__________________________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

_______________________________________________
grass-dev mailing list
grass-dev@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass-dev

Michael Barton wrote:

>> 56% 60% 64% 68% 72% 76% 80%
>> 84% 88% 92% 96% 100%
>> CREATING SUPPORT FILES FOR managed.2.6
>>
>> It occurs when I create the new maps using r.in.ascii and is sent via
>> standard error.
>
> What's the problem? Progress messages are normal, and are supposed to
> be written to stderr.

It seems odd to me that progress messages are written to stderr. However, if
that is the norm across C programs, then I guess that's were they belong.
But does " CREATING SUPPORT FILES FOR managed.2.6" also go there?

It should.

As a general rule, human-readable output goes to stderr while
machine-readable output goes to stdout. There are two main reasons for
this approach:

1. If you have a pipleine such as:

  foo | bar | baz > out.txt

the data piped between commands and written to the file consists
solely of machine-readable data. Writing diagnostics or status message
to stdout would confuse programs further down the pipeline, or
programs reading the resulting text file.

In the above example, none of the redirections affect stderr, so it
would continue to refer to the terminal.

2. At program startup, stdout is line-buffered if it refers to a
terminal and fully-buffered otherwise (e.g. for files, pipes or
sockets), while stderr is unbuffered.

Writing to a fully-buffered stream won't produce any
externally-visible results until either the buffer is full or an
explicit fflush() is used.

If stdout is a terminal, you will only see complete lines of output,
so output where lines are composed over time (e.g. percentage output,
or "doing something ..... <long delay> ... done") will appear all at
once upon completion.

Worse, if stdout is a pipe (or file being monitored with "tail -f ..."
from another terminal), you will only see output in complete blocks
(typically 4KiB).

We also need a "quiet" option for all commands to suppress all stdout except
the actual output values. There is a lot of verbage produced some GRASS
commands that might be nice for a user at a terminal to read, but that is
problematic when GRASS modules are being used in other environments. This is
available for some modules, but has variable effects (i.e., sometimes it
only suppresses some of the extraneous verbage).

That isn't hard to implement, although it's a fair amount of (rather
tedious) work.

There are two parts:

1. Change individual modules to use G_message() rather than
[f]printf() for all progress messages.

2. Add a G_INFO_FORMAT_QUIET option to G_info_format() and change
G_message() to take note of it.

--
Glynn Clements <glynn@gclements.plus.com>

Michael Barton wrote:

We are linking GRASS modules with a Java agent-based modeling
environment. The lack of consistency in GRASS output is an issue here,
as it is with trying to link GRASS with an interface building
platform.

Because GRASS is a suite of command-line tools, it is an ideal
platform to be used by other platform for sophisticated modeling
(Python, Java, etc) as well as be wrapped in an interface platform to
serve as a full-featured GIS.

Implement Java bindings for the SWIG interface?
See QGIS<->GRASS module glue?

However, these become issues that range from annoyances (parsing
r.info) to impossibilities (parsing r.describe) when you try to use
GRASS from another environment than a user at a terminal.

With r.info you have a point (although extending the -g flag as
suggested by Martin could help that). For me r.info has all the flags I
need (because we've added them as needed...), but v.info could use a
parsable version of lines= boundaries= etc.

What's so hard about parsing "r.describe -q1"?

It seems odd to me that progress messages are written to stderr.

If you redirect stderr with "2> /dev/null" all that should be left is
parsable output from stdout.

We also need a "quiet" option for all commands to suppress all stdout
except the actual output values.

or otherwise stated, only parsable output should be sent to stdout.

I agree with your basic tenet that as much functionalily as possible
should be able to be dealt with by scripts or wrapper functions (i.e.
generalized interoperability). I think we are well on our way to this
already, certainly more so than the competition.

Hamish

From: Hamish <hamish_nospam@yahoo.com>
Date: Mon, 4 Sep 2006 14:16:42 +1200
To: Michael Barton <michael.barton@asu.edu>
Cc: <grass-dev@grass.itc.it>, <Gary.Mayer@asu.edu>
Subject: Re: [GRASS-dev] Incorrect stuff coming through stderr?

Implement Java bindings for the SWIG interface?
See QGIS<->GRASS module glue?

Actually, we don't even need SWIG.

However, these become issues that range from annoyances (parsing
r.info) to impossibilities (parsing r.describe) when you try to use
GRASS from another environment than a user at a terminal.

With r.info you have a point (although extending the -g flag as
suggested by Martin could help that). For me r.info has all the flags I
need (because we've added them as needed...), but v.info could use a
parsable version of lines= boundaries= etc.

But you can just get all the r.info data at once, and one flag seems to
cancel out the others, so you can't accumulate them either.

What's so hard about parsing "r.describe -q1"?

Because the output varies depending on whether the map is latlon or
projected, all positive or negative values, or some positive and some
negative values.

It seems odd to me that progress messages are written to stderr.

If you redirect stderr with "2> /dev/null" all that should be left is
parsable output from stdout.

We also need a "quiet" option for all commands to suppress all stdout
except the actual output values.

or otherwise stated, only parsable output should be sent to stdout.

I agree with your basic tenet that as much functionalily as possible
should be able to be dealt with by scripts or wrapper functions (i.e.
generalized interoperability). I think we are well on our way to this
already, certainly more so than the competition.

I agree. It seems an opportune time to try to regularize across all of GRASS

Michael

__________________________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton