[GRASSLIST:155] running grass commands from CGI script

GRASS GRASS user
1

I am putting together a system using Mapserver to render maps online
that are derived from GRASS raster grids. I am using CGI to POST to a
cgi-bin executable which then invokes a series of GRASS commands to
generate the raster before returning to Mapserver to display the
results. This process works GREAT except for the following:

* r.mapcalc will "freeze" causing an apache timeout error. To be
precise, actually, invoking r.mapcalc a large (N) number of times will
cause the server to timeout. I'm not exactly sure what N is the
critical threshold that causes problems (somewhere around 10 or so). I
have a simple process that calls r.mapcalc about five times and doesn't
cause any problems. I have another complex function that can
potentially call r.mapcalc dozens of times. It is this complex function
that fails.

* This process fails ONLY when running through the CGI process and the
apache webserver. As a shell script from the command line...no
problems, no errors.

* I can run any number of other grass commands (e.g. g.rename, g.copy,
r.report, whatever). It seems to be only the r.mapcalc command that
offends apache.

Has anyone done anything like this before and succeeded? Obviously the
problem lies BOTH in Apache and r.mapcalc. Any suggestions?

Matt

Matt Doggett
Spatial Climate Analysis Service
Oregon State University
316 Strand Ag Hall
Corvallis, OR 97331
(541)737-9153
mdoggett@coas.oregonstate.edu

2

Upon further investigation, I discovered that if I redirect STDERR and
STDOUT when I call r.mapcalc, this problem goes away (e.g. `r.mapcalc
newgrid=A*B*C >somefile 2>&1`).

  Maybe someone can shed some light as to what r.mapcalc is doing to
STDOUT and/or STDERR that might be causing problems with apache?

matt

-----Original Message-----
From: owner-GRASSLIST@baylor.edu [mailto:owner-GRASSLIST@baylor.edu]

On

Behalf Of Matt Doggett
Sent: Friday, May 23, 2003 12:51 PM
To: GRASSLIST@baylor.edu
Subject: [GRASSLIST:155] running grass commands from CGI script

I am putting together a system using Mapserver to render maps online
that are derived from GRASS raster grids. I am using CGI to POST to a
cgi-bin executable which then invokes a series of GRASS commands to
generate the raster before returning to Mapserver to display the
results. This process works GREAT except for the following:

* r.mapcalc will "freeze" causing an apache timeout error. To be
precise, actually, invoking r.mapcalc a large (N) number of times will
cause the server to timeout. I'm not exactly sure what N is the
critical threshold that causes problems (somewhere around 10 or so).

I

have a simple process that calls r.mapcalc about five times and

doesn't

cause any problems. I have another complex function that can
potentially call r.mapcalc dozens of times. It is this complex

function

that fails.

* This process fails ONLY when running through the CGI process and the
apache webserver. As a shell script from the command line...no
problems, no errors.

* I can run any number of other grass commands (e.g. g.rename, g.copy,
r.report, whatever). It seems to be only the r.mapcalc command that
offends apache.

Has anyone done anything like this before and succeeded? Obviously

the

problem lies BOTH in Apache and r.mapcalc. Any suggestions?

Matt

Matt Doggett
Spatial Climate Analysis Service
Oregon State University
316 Strand Ag Hall
Corvallis, OR 97331
(541)737-9153
mdoggett@coas.oregonstate.edu

3

Matt Doggett wrote:

Upon further investigation, I discovered that if I redirect STDERR and
STDOUT when I call r.mapcalc, this problem goes away (e.g. `r.mapcalc
newgrid=A*B*C >somefile 2>&1`).

  Maybe someone can shed some light as to what r.mapcalc is doing to
STDOUT and/or STDERR that might be causing problems with apache?

[What follows assumes that you're using GRASS 5.0.0 or later, which
has a completely re-written r.mapcalc; the r.mapcalc in 4.3 and early
5.0 betas is substantially different.]

r.mapcalc writes error messages to stderr, but this shouldn't
interfere with Apache; it's not uncommon for CGI scripts (or the
programs they call) to write to stderr; any such output should end up
in Apache's error log.

AFAICT, r.mapcalc itself shouldn't write anything to stdout; however,
I can't entirely rule out the possibility that some lower-level
function might write to stdout.

In the CGI script, are you generating the CGI header (Content-Type
etc) early on, or are you leaving it until after r.mapcalc has been
run? If a program writes to stdout before you've written the header,
Apache will try to parse it as a CGI header, with unpredictable
results.

--
Glynn Clements <glynn.clements@virgin.net>

4

Some more information...

* I am using grass5.0.2 and have tried both the new r.mapcalc and the
original r.mapcalc (4.3 version) with the same results.
* My CGI headers are formulated correctly prior to running the grass
script. Therefore, any STDOUT output would simply appear as text in my
browser.
* r.mapcalc prints a progression status to STDERR as the command runs.
i.e. 1%...2%...3%...etc. (I find it interesting that it goes to STDERR
not STDOUT.) Also, it must be sending special cursor control characters
to the terminal (like backspaces or beginning of line) since it
continuously prints this % completion in the same place (hmmm, special
control-characters have been known to cause problems in some
environments).
* While these output go to STDERR, they do NOT show up the the Apache
error_log
* If I do NOT redirect STDERR to either a file or /dev/null, apache will
timeout on the CGI execution.

My conclusion...
* I can't say for sure whether there is anything *wrong* with either
Grass or Apache configuration--it just appears to me a mis-handling of
the behavior that r.mapcalc exhibits. It looks to be tied up in the
output that r.mapcalc sends to STDERR that reports the progress of the
r.mapcalc command.

My solution...
* If running CGI scripts that call the r.mapcalc command, redirect
STDERR to either a file or /dev/null. This is not a big deal as the
progression indicator (which seems to be the culprit) isn't very useful
in the CGI environment anyway.

My question...
* Is there a way to disable this status reporting of r.mapcalc command?

Hopefully this thread may help anyone attempting to do this in the
future.

-Matt

-----Original Message-----
From: owner-GRASSLIST@baylor.edu [mailto:owner-GRASSLIST@baylor.edu]

On

Behalf Of Glynn Clements
Sent: Friday, May 23, 2003 3:57 PM
To: mdoggett@coas.oregonstate.edu
Cc: GRASSLIST@baylor.edu
Subject: [GRASSLIST:161] RE: running grass commands from CGI script

Matt Doggett wrote:

> Upon further investigation, I discovered that if I redirect STDERR

and

> STDOUT when I call r.mapcalc, this problem goes away (e.g.

`r.mapcalc

> newgrid=A*B*C >somefile 2>&1`).
>
> Maybe someone can shed some light as to what r.mapcalc is doing to
> STDOUT and/or STDERR that might be causing problems with apache?

[What follows assumes that you're using GRASS 5.0.0 or later, which
has a completely re-written r.mapcalc; the r.mapcalc in 4.3 and early
5.0 betas is substantially different.]

r.mapcalc writes error messages to stderr, but this shouldn't
interfere with Apache; it's not uncommon for CGI scripts (or the
programs they call) to write to stderr; any such output should end up
in Apache's error log.

AFAICT, r.mapcalc itself shouldn't write anything to stdout; however,
I can't entirely rule out the possibility that some lower-level
function might write to stdout.

In the CGI script, are you generating the CGI header (Content-Type
etc) early on, or are you leaving it until after r.mapcalc has been
run? If a program writes to stdout before you've written the header,
Apache will try to parse it as a CGI header, with unpredictable
results.

--
Glynn Clements <glynn.clements@virgin.net>

5

Matt Doggett wrote:

* I am using grass5.0.2 and have tried both the new r.mapcalc and the
original r.mapcalc (4.3 version) with the same results.
* My CGI headers are formulated correctly prior to running the grass
script. Therefore, any STDOUT output would simply appear as text in my
browser.

OK.

* r.mapcalc prints a progression status to STDERR as the command runs.
i.e. 1%...2%...3%...etc.

r.mapcalc should display 2% increments, i.e. 0%...2%...4%...etc.

(I find it interesting that it goes to STDERR
not STDOUT.)

This is standard practice; output which is only intended for a human
operator (progress indication, prompts etc) is written to stderr, so
that it's still displayed on the terminal if stdout is redirected to a
file.

As r.mapcalc doesn't (intentionally) write to stderr, this isn't
applicable here. However, the progress output is generated by the
G_percent() function, which is used by many GRASS programs, some of
which may write to stderr data which might reasonably be redirected to
a file.

Also, it must be sending special cursor control characters
to the terminal (like backspaces or beginning of line) since it
continuously prints this % completion in the same place

Backspace codes (ASCII code 8) are written to return the cursor to the
beginning of the line:

  fprintf (stderr,"%4d%%\b\b\b\b\b",x);

(hmmm, special
control-characters have been known to cause problems in some
environments).
* While these output go to STDERR, they do NOT show up the the Apache
error_log
* If I do NOT redirect STDERR to either a file or /dev/null, apache will
timeout on the CGI execution.

My conclusion...
* I can't say for sure whether there is anything *wrong* with either
Grass or Apache configuration--it just appears to me a mis-handling of
the behavior that r.mapcalc exhibits. It looks to be tied up in the
output that r.mapcalc sends to STDERR that reports the progress of the
r.mapcalc command.

My solution...
* If running CGI scripts that call the r.mapcalc command, redirect
STDERR to either a file or /dev/null. This is not a big deal as the
progression indicator (which seems to be the culprit) isn't very useful
in the CGI environment anyway.

My question...
* Is there a way to disable this status reporting of r.mapcalc command?

No. Many (most?) commands which display a progress percentage either
only do so if a -v (verbose) switch is used, or allow it to be
disabled via a -q (quiet) switch. However, r.mapcalc doesn't have any
options or switches; the entire command line is treated as an
expression.

It should probably check whether stderr is a tty, and only display the
percentages if it is. It alread checks stdin (the "mapcalc>" prompt is
only displayed if stdin is a tty).

--
Glynn Clements <glynn.clements@virgin.net>