[GRASS-dev] GRASS_BATCH_JOB vs GUI

GRASS GRASS developer
1

I was discussing with someone running GRASS in batch mode, and it's not clear what is possible with respect to X11, display windows, TclTk or Python.

I see that in init.sh, a batch script is forced to run in text mode. Still, a module run without options (intentionally or not) would try to bring up a TclTk window and require user interaction, thus potentially disrupting the flow of the script.

Certainly running either the TclTk or Python GUIs in a batch script is not desirable, but maybe there are possibilities I don't see?

Display windows may be useful in a batch script - or not?

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"Those people who most want to rule people are, ipso-facto, those least suited to do it."

- A rule of the universe, from the HitchHiker's Guide to the Galaxy

2

William Kyngesburye wrote:

I was discussing with someone running GRASS in batch mode, and it's
not clear what is possible with respect to X11, display windows, TclTk
or Python.

I see that in init.sh, a batch script is forced to run in text mode.
Still, a module run without options (intentionally or not) would try
to bring up a TclTk window and require user interaction,

just because GRASS is started in text mode doesn't mean that module GUIs
won't work (this is how I use GRASS), just that the main GUI isn't
automatically started.

thus potentially disrupting the flow of the script.

that only matters if *you* expect the script to run without interaction.
GRASS itself doesn't care if the script pauses while a user fills out a
form.

Certainly running either the TclTk or Python GUIs in a batch script is
not desirable, but maybe there are possibilities I don't see?

the full GUI isn't really useful here, but theoretically you could launch
it if you really wanted to.

You can pre-populate a module's GUI by setting options and flags on the
command line then forcing a GUI with the --ui flag.

Display windows may be useful in a batch script - or not?

sure, it's a slightly different concept, but for custom apps, why not?
See the d.ask and d.menu modules for more interactive scripting fun
(those won't survive into GRASS 7 in their current state, but could be
rewritten to launch custom wx|tcl GUI windows).

FYI, AFAIR the latest batch mode changes to init.sh have not been
backported to the 6.3.0 branch- but I think they are just cosmetic.

Hamish

      ____________________________________________________________________________________
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost.
http://tc.deals.yahoo.com/tc/blockbuster/text5.com

3

William Kyngesburye wrote:

I was discussing with someone running GRASS in batch mode, and it's
not clear what is possible with respect to X11, display windows, TclTk
or Python.

I see that in init.sh, a batch script is forced to run in text mode.

As well as setting GRASS_GUI, the -text, -tcltk etc options also
determine how the database/location/mapset settings are obtained.
-text uses the curses interface (set_data), -tcltk uses Tcl/Tk
(gis_set.tcl) etc.

Back when I used to actually start GRASS sessions via the grass63
(etc) script, I invariably passed the full path to the mapset
directory as an argument to avoid the interactive startup altogether
(my preferred "terminal" is XEmacs' shell-mode, which doesn't support
curses).

Nowadays, I just set all the relevant variables from my bash startup
scripts, so I never explicitly start GRASS sessions (my bash startup
scripts are sourced from ~/.xsession, so I can use M-! in XEmacs to
run GRASS commands).

Personally, I would support skipping the database/location/mapset
dialog altogether if ~/.grassrc6 already exists and contains valid
data. You would only be prompted for this information the first time
you used GRASS. Thereafter, you would get a session using the stored
settings; you can use g.mapset (or the GUI) if you want to change
them.

Still, a module run without options (intentionally or not) would try
to bring up a TclTk window and require user interaction, thus
potentially disrupting the flow of the script.

The GUI form should[*] only be displayed if a module is run without
options but has at least one required option. If the module was called
from a script, getting a dialog would normally indicate a bug in the
script.

[*] However, the parser doesn't actually enforce this. It's normally
done by the the module skipping the G_parser() call altogether if
argc==1, i.e.:

  if (argc > 1 && G_parser(argc, argv))
      exit(1);

That's potentially problematic, e.g. if an option has ->multiple=YES
and a default ->answer setting, it won't get parsed into the ->answers
array if G_parser() isn't called. But that hasn't been an issue so
far.

Certainly running either the TclTk or Python GUIs in a batch script is
not desirable, but maybe there are possibilities I don't see?
Display windows may be useful in a batch script - or not?

There's no inherent reason why a script shouldn't be interactive. But
most of the time, you would be better off prompting the user for all
necessary options at the outset.

--
Glynn Clements <glynn@gclements.plus.com>

4

I was the one corresponding with William about this situation. What I was
after was what is provided in Linux. On Linux systems, I run grass scripts
embedded within bash scripts and have those run off the clock or from the
command line. I have no need for a display nor do I want one. When the
process is executed from the command line in Linux it dutifully sits in the
background, redirecting output to a log file with no interruptions to
whatever else I may be doing in other windows. When run off the clock, it
processes properly as well.

When I first tried to execute the scripts in OSX both an X term and a
Terminal app window would pop up stealing focus from whatever I was doing in
another window. The looping in the script would fire off grass repeatedly
and every time grass would keep the window I was working in at the top of
the heap but the cursor/mouse focus would be taken away by the X term
window.

I realize all the possibilities that are capable with grass scripting to use
a display even when in GRASS_BATCH_MODE. I understand the X term is needed
on the Mac since Grass requires it for the tcltk gui. However, I know I am
not going to ask for the display in my scripts. All the processing I will
do in this background mode is vector and raster manipulations and
calculations. So, what I would like, with caveat emptor, is a switch/shell
env't var to tell the grass execution not to pop up any other windows/terms,
especially in Mac OSX. This switch would put the burden on the programmer
to ensure their code didn't ask for the X capabilities, but that's a
responsibility I would be willing to take on to get the quiet, non-focus
stealing behavior I'm after.

I tinkered with the grass.sh script in /Apps.../Grass-6.3.app/Contents/MacOS
to bypass any of the GUI setup (X and Terminal windows) when GRASS_BATCH_JOB
is non-empty. That provided the behavior I was after and didn't interfere
with interactive use, so setting and acting upon a switch to indicate
background execution will work.

Joe

Glynn Clements wrote:

William Kyngesburye wrote:

I was discussing with someone running GRASS in batch mode, and it's
not clear what is possible with respect to X11, display windows, TclTk
or Python.

I see that in init.sh, a batch script is forced to run in text mode.

As well as setting GRASS_GUI, the -text, -tcltk etc options also
determine how the database/location/mapset settings are obtained.
-text uses the curses interface (set_data), -tcltk uses Tcl/Tk
(gis_set.tcl) etc.

Back when I used to actually start GRASS sessions via the grass63
(etc) script, I invariably passed the full path to the mapset
directory as an argument to avoid the interactive startup altogether
(my preferred "terminal" is XEmacs' shell-mode, which doesn't support
curses).

Nowadays, I just set all the relevant variables from my bash startup
scripts, so I never explicitly start GRASS sessions (my bash startup
scripts are sourced from ~/.xsession, so I can use M-! in XEmacs to
run GRASS commands).

Personally, I would support skipping the database/location/mapset
dialog altogether if ~/.grassrc6 already exists and contains valid
data. You would only be prompted for this information the first time
you used GRASS. Thereafter, you would get a session using the stored
settings; you can use g.mapset (or the GUI) if you want to change
them.

Still, a module run without options (intentionally or not) would try
to bring up a TclTk window and require user interaction, thus
potentially disrupting the flow of the script.

The GUI form should[*] only be displayed if a module is run without
options but has at least one required option. If the module was called
from a script, getting a dialog would normally indicate a bug in the
script.

[*] However, the parser doesn't actually enforce this. It's normally
done by the the module skipping the G_parser() call altogether if
argc==1, i.e.:

  if (argc > 1 && G_parser(argc, argv))
      exit(1);

That's potentially problematic, e.g. if an option has ->multiple=YES
and a default ->answer setting, it won't get parsed into the ->answers
array if G_parser() isn't called. But that hasn't been an issue so
far.

Certainly running either the TclTk or Python GUIs in a batch script is
not desirable, but maybe there are possibilities I don't see?
Display windows may be useful in a batch script - or not?

There's no inherent reason why a script shouldn't be interactive. But
most of the time, you would be better off prompting the user for all
necessary options at the outset.

--
Glynn Clements <glynn@gclements.plus.com>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

--
View this message in context: http://www.nabble.com/GRASS_BATCH_JOB-vs-GUI-tp16508266p16519926.html
Sent from the Grass - Dev mailing list archive at Nabble.com.

5

Joe O. wrote:

I was the one corresponding with William about this situation. What I was
after was what is provided in Linux. On Linux systems, I run grass scripts
embedded within bash scripts and have those run off the clock or from the
command line. I have no need for a display nor do I want one. When the
process is executed from the command line in Linux it dutifully sits in the
background, redirecting output to a log file with no interruptions to
whatever else I may be doing in other windows. When run off the clock, it
processes properly as well.

When I first tried to execute the scripts in OSX both an X term and a
Terminal app window would pop up stealing focus from whatever I was doing in
another window. The looping in the script would fire off grass repeatedly
and every time grass would keep the window I was working in at the top of
the heap but the cursor/mouse focus would be taken away by the X term
window.

I realize all the possibilities that are capable with grass scripting to use
a display even when in GRASS_BATCH_MODE. I understand the X term is needed
on the Mac since Grass requires it for the tcltk gui. However, I know I am
not going to ask for the display in my scripts. All the processing I will
do in this background mode is vector and raster manipulations and
calculations. So, what I would like, with caveat emptor, is a switch/shell
env't var to tell the grass execution not to pop up any other windows/terms,
especially in Mac OSX. This switch would put the burden on the programmer
to ensure their code didn't ask for the X capabilities, but that's a
responsibility I would be willing to take on to get the quiet, non-focus
stealing behavior I'm after.

I tinkered with the grass.sh script in /Apps.../Grass-6.3.app/Contents/MacOS
to bypass any of the GUI setup (X and Terminal windows) when GRASS_BATCH_JOB
is non-empty. That provided the behavior I was after and didn't interfere
with interactive use, so setting and acting upon a switch to indicate
background execution will work.

GRASS_BATCH_JOB is a bit of a hack. You would be better off bypassing
Init.sh altogether.

FWIW, I just set up the GRASS environment from my ~/.bash_profile,
with:

  export GISBASE=/opt/grass-6.3.svn
  export GRASS_GNUPLOT='gnuplot -persist'
  export GRASS_WIDTH=640
  export GRASS_HEIGHT=480
  export GRASS_HTML_BROWSER=firefox
  export GRASS_PAGER=cat
  export GRASS_PERL=perl
  export GRASS_TCLSH=tclsh
  export GRASS_WISH=wish
  export GRASS_MESSAGE_FORMAT=silent
  export GRASS_TRUECOLOR=TRUE
  
  export PATH="$GISBASE/bin:$GISBASE/scripts:$PATH"
  export LD_LIBRARY_PATH="$GISBASE/lib"
  export GRASS_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
  
  export GIS_LOCK=$$
  export GRASS_VERSION="6.3.svn"
  
  tmp=/tmp/grass6-"`whoami`"-$GIS_LOCK
  export GISRC="$tmp/gisrc"
  mkdir "$tmp"
  cp ~/.grassrc6 "$GISRC"

AFAICT, this sets all of the environment variables which Init.sh would
set, even the ones which aren't really necessary (e.g. GRASS_PERL
isn't actually used by any GRASS modules or script; it's only used to
generate the manpages at compile time).

The above would need some minor changes for any given system, e.g.
setting GISBASE appropriately, using DYLD_LIBRARY_PATH instead of
LD_LIBRARY_PATH on MacOSX, etc).

--
Glynn Clements <glynn@gclements.plus.com>

6

Joe O. wrote:
> When I first tried to execute the scripts in OSX both an X term and a
> Terminal app window would pop up stealing focus from whatever I was
> doing in another window.

the first thing I do after installing X11 on Mac OSX is to stop it from
launching a xterm upon X11 startup.

go to a terminal
$ sudo su
# cd /etc/X11/xinit
# chmod u+w xinitrc

then edit the xinitrc file and #comment out the line at the end which
calls xterm

that stops some of the mess, but GRASS in text mode should never even
call X11, or pop up a new xterm, something else might be doing that.
(binary packager's startup script?)

FWIW, ISTR that OSX 10.5 launches X11 on process-demand, whereas in
earlier versions you have to launch in manually.

> The looping in the script would fire off grass repeatedly and every
> time grass would keep the window I was working in at the top of the
> heap but the cursor/mouse focus would be taken away by the X term
> window.

check your ~/.grassrc6 file has "GRASS_GUI: text". If not, start grass
from the terminal prompt with "grass63 -text" once, then text mode should
be set as the default. (but I think GRASS_BATCH_JOB should be overriding
whatever is there anyway)

to confirm: the gui is not popping up, just extra terminal windows?

Can you add something to your scripts that launch grass to test for an
existing session?

if [ -z "$GISBASE" ] ; then
  grass63 /path/to/mapset # to run batch job
else
  echo "ERROR: already in a grass session"
  # shrug, run it anyway
  . /path/to/batchjob.sh
fi

is your shell csh or bash? ("echo $SHELL") If the machine was upgraded
from older versions of OSX, enduring user accounts may still be using
csh.

> I tinkered with the grass.sh script

...

> to bypass any of the GUI setup (X and Terminal windows) when
> GRASS_BATCH_JOB is non-empty. That provided the behavior I was
> after and didn't interfere with interactive use, so setting and
> acting upon a switch to indicate background execution will work.

can you islotate the problematic part of the script?

is this a 6.3.0RC or 6.3svn version of GRASS?
self compiled or one of the binary packages? who's binary package?

by starting scripts "by the clock", do you mean using OSX's replacement
for cron jobs? (I forget what they called that)

Glynn:

GRASS_BATCH_JOB is a bit of a hack.

perhaps that is true, but it should work. And if it doesn't work it
should be fixed.

would a command line option for batch job be better? init.sh could be
written to act in a similar way to GRASS's G_parser():

grass63 mapset=/path/to/mapset batch=/path/to/batchjob.sh
?

Hamish

      ____________________________________________________________________________________
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost.
http://tc.deals.yahoo.com/tc/blockbuster/text5.com

7

On Apr 5, 2008, at 9:52 PM, Hamish wrote:

the first thing I do after installing X11 on Mac OSX is to stop it from
launching a xterm upon X11 startup.

We got that straightened out - Joe forgot to do that (it's mentioned in the OSX readme).

I also worked out some improvements in the app startup script to minimize the X11 focus switching. Now it doesn't "open" (*) X11 if it's already running, thus avoiding a focus switch. It also records which term application the script is running in (**) and returns focus to that.

So now, starting X11 before GRASS (ie on login), or just leaving it running after the first run of GRASS, seems to work well.

(*) 'open' runs an OSX application, and since an OSX application normally can only have one instance running, it only activates it when it's already running. I used this as a shortcut to checking if it's running in the startup script.

(**) If run from a double-click or drag-n-drop, GRASS.app will always use Terminal.app, but the grass script, as Joe is doing, could be run directly from a different term app such as xterm or iTerm.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"I ache, therefore I am. Or in my case - I am, therefore I ache."

- Marvin

8

Glynn Clements <glynn@gclements.plus.com> writes:

[...]

>> I tinkered with the grass.sh script in
>> /Apps.../Grass-6.3.app/Contents/MacOS to bypass any of the GUI setup
>> (X and Terminal windows) when GRASS_BATCH_JOB is non-empty. That
>> provided the behavior I was after and didn't interfere with
>> interactive use, so setting and acting upon a switch to indicate
>> background execution will work.

> GRASS_BATCH_JOB is a bit of a hack. You would be better off
> bypassing Init.sh altogether.

  I wonder, is there a way to create locations non-interactively?

  It may be quite useful in, e. g., the test suite's scripts.
  I've in mind the following scheme:

  * create a fresh location;

  * prepare inputs and produce outputs;

  * check checksums (SHA1 or MD5) of the files in the locations to
    match those contained in the test suite; check for unwanted or
    missing files as well;

  * if all went good, remove the location; otherwise, leave it in
    order for the developer to examine it.

  Or, does GRASS use architecture-dependent formats within its
  locations?

> FWIW, I just set up the GRASS environment from my ~/.bash_profile,
> with:

> export GISBASE=/opt/grass-6.3.svn

[...]

  BTW, it may worth to split the `export's off ~/.bash_profile
  into, say, ~/.session_common to `source' from both
  ~/.bash_profile and ~/.xsession:

[ -f "$HOME"/.session_common ] && . "$HOME"/.session_common

  instead of processing ~/.bash_profile itself as part of the X
  client-side init sequence. At least, I use some
  terminal-related bits in ~/.bash_profile, which weren't meant
  for X.

9

Ivan Shmakov wrote:

>> I tinkered with the grass.sh script in
>> /Apps.../Grass-6.3.app/Contents/MacOS to bypass any of the GUI setup
>> (X and Terminal windows) when GRASS_BATCH_JOB is non-empty. That
>> provided the behavior I was after and didn't interfere with
>> interactive use, so setting and acting upon a switch to indicate
>> background execution will work.

> GRASS_BATCH_JOB is a bit of a hack. You would be better off
> bypassing Init.sh altogether.

  I wonder, is there a way to create locations non-interactively?

  g.proj -c location=...

  Or, does GRASS use architecture-dependent formats within its
  locations?

AFAICT, the core library functions use portable formats; however:

1. Individual modules can write whatever they want under cell_misc.
E.g. the fftreal and fftimag files written by i.fft are in host byte
order.

2. Even when portable formats are used, the precise choice of format
may be platform-specific.

E.g. the row indices in the raster files can be 4 or 8 bytes,
depending up the size of off_t in the code which creates them. The
actual size is stored in the file, and the code which reads them can
handle the case where the size used in the file doesn't match the
off_t used in the code, so long as the actual values fit within an
off_t.

For text formats, there's the issue that the MSVCRT version of
printf's %e and %g always use a 3-digit exponent, while other
implementations follow ANSI and only use 3 digits where necessary.

--
Glynn Clements <glynn@gclements.plus.com>

10

On Apr 6, 2008, at 10:55 AM, Glynn Clements wrote:

1. Individual modules can write whatever they want under cell_misc.
E.g. the fftreal and fftimag files written by i.fft are in host byte
order.

Ugh, didn't know about that one (we took care of the stroke fonts and portable.h already). This would make it impossible to work with the same fft map data on PPC and Intel OSX, or sharing fft map data between different platforms (most Mac users consider "OSX" the platform, not the "processor" type).

2. Even when portable formats are used, the precise choice of format
may be platform-specific.

Maybe we need to specify that anything written must be in a portable format. And provide some library functions (maybe already there?) for writing data in a portable way.

Another item for GRASS 7, I suppose...

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"Those people who most want to rule people are, ipso-facto, those least suited to do it."

- A rule of the universe, from the HitchHiker's Guide to the Galaxy

11

Glynn Clements <glynn@gclements.plus.com> writes:

[...]

>>> GRASS_BATCH_JOB is a bit of a hack. You would be better off
>>> bypassing Init.sh altogether.

>> I wonder, is there a way to create locations non-interactively?

> g.proj -c location=...

  ACK, thanks.

>> Or, does GRASS use architecture-dependent formats within its
>> locations?

> AFAICT, the core library functions use portable formats; however:

> 1. Individual modules can write whatever they want under cell_misc.
> E.g. the fftreal and fftimag files written by i.fft are in host byte
> order.

  Shouldn't these be fixed?

> 2. Even when portable formats are used, the precise choice of format
> may be platform-specific.

> E.g. the row indices in the raster files can be 4 or 8 bytes,
> depending up the size of off_t in the code which creates them. The
> actual size is stored in the file, and the code which reads them can
> handle the case where the size used in the file doesn't match the
> off_t used in the code, so long as the actual values fit within an
> off_t.

  Why not to use 8-byte values unconditionally when writing a
  raster? (Retaining the ability to read 4-byte values for the
  sake of compatibility.)

  If several formats need to be supported, there should probably
  be a way to choose one at the run time. (Say, an environment
  variable.) In particular, this will allow the reading of all of
  these formats to be tested with a single GRASS build.

> For text formats, there's the issue that the MSVCRT version of
> printf's %e and %g always use a 3-digit exponent, while other
> implementations follow ANSI and only use 3 digits where necessary.

  If there's only a small number of ways to store the result,
  several checksums may be specified for the file.

  If the result is largely unpredicable, then it shouldn't
  probably be checked with the test suite in the first place.
  E. g., there seems to be a little point of checking the vector
  created by the current version of `v.random'.

12

Ivan Shmakov wrote:

> 1. Individual modules can write whatever they want under cell_misc.
> E.g. the fftreal and fftimag files written by i.fft are in host byte
> order.

  Shouldn't these be fixed?

First, you need to identify which modules are doing this sort of
thing.

As for i.fft/i.ifft, the fact that it's storing the FFT data as a
"map" (where the actual map data is only used for e.g. d.rast, and is
ignored by i.ifft) is arguably a more significant flaw than the fact
that the files aren't portable.

> 2. Even when portable formats are used, the precise choice of format
> may be platform-specific.

> E.g. the row indices in the raster files can be 4 or 8 bytes,
> depending up the size of off_t in the code which creates them. The
> actual size is stored in the file, and the code which reads them can
> handle the case where the size used in the file doesn't match the
> off_t used in the code, so long as the actual values fit within an
> off_t.

  Why not to use 8-byte values unconditionally when writing a
  raster? (Retaining the ability to read 4-byte values for the
  sake of compatibility.)

Both formats already exist in the wild, so the code which reads raster
data has to be able to cope with both. In that situation, there
doesn't seem to be much point in modifying the writing code.

> For text formats, there's the issue that the MSVCRT version of
> printf's %e and %g always use a 3-digit exponent, while other
> implementations follow ANSI and only use 3 digits where necessary.

  If there's only a small number of ways to store the result,
  several checksums may be specified for the file.

With floating-point, there's always the possibility of differences in
rounding depending upon the architecture and which optimisations were
used. Ensuring that floating-point calculations are repeatable at the
bit-pattern level is typically a lot of work for negligible benefit
(simplifying the test suite is usually the *only* benefit).

--
Glynn Clements <glynn@gclements.plus.com>

13

Glynn Clements <glynn@gclements.plus.com> writes:

[...]

>>> 2. Even when portable formats are used, the precise choice of
>>> format may be platform-specific.

>>> E.g. the row indices in the raster files can be 4 or 8 bytes,
>>> depending up the size of off_t in the code which creates them. The
>>> actual size is stored in the file, and the code which reads them
>>> can handle the case where the size used in the file doesn't match
>>> the off_t used in the code, so long as the actual values fit within
>>> an off_t.

>> Why not to use 8-byte values unconditionally when writing a raster?
>> (Retaining the ability to read 4-byte values for the sake of
>> compatibility.)

> Both formats already exist in the wild, so the code which reads
> raster data has to be able to cope with both. In that situation,
> there doesn't seem to be much point in modifying the writing code.

  If not to allow for repeatable results. And this may be useful
  for purposes other than the test suite's ones.

  (Actually, I've found it quite annoying to check for the results
  to match with any tool other than cmp(1).)

>>> For text formats, there's the issue that the MSVCRT version of
>>> printf's %e and %g always use a 3-digit exponent, while other
>>> implementations follow ANSI and only use 3 digits where necessary.

>> If there's only a small number of ways to store the result, several
>> checksums may be specified for the file.

> With floating-point, there's always the possibility of differences in
> rounding depending upon the architecture and which optimisations were
> used. Ensuring that floating-point calculations are repeatable at the
> bit-pattern level is typically a lot of work for negligible benefit

  Agreed. For now, I'd focus in making the test suite work for
  the most basic modules, and defer the testing of FP ones until a
  later time.

  A somewhat cumbersome solution for the FP rasters would be to
  import a raster map, find the difference with `r.mapcalc' and
  then check for it to be acceptable with `r.univar'.

> (simplifying the test suite is usually the *only* benefit).