[GRASS-dev] Python Scripting

cmbarton · July 16, 2008, 6:06am

On Jul 15, 2008, at 10:42 PM, <grass-dev-request@lists.osgeo.org> <grass-dev-request@lists.osgeo.org > wrote:

Message: 2
Date: Wed, 16 Jul 2008 04:14:44 +0100
From: Glynn Clements <glynn@gclements.plus.com>
Subject: Re: [GRASS-dev] Python Scripting
To: "W. Chris Carleton" <w_chris_carleton@hotmail.com>
Cc: grass-dev@lists.osgeo.org
Message-ID: <18557.26532.411242.947615@cerise.gclements.plus.com>
Content-Type: text/plain; charset=us-ascii

W. Chris Carleton wrote:

I'm trying to automate some modules in GRASS 6.3.2 using Python scripts
and g.parser. I've had a look at the examples in /Scripts, but I'm
having some trouble. I want to pass arguments to a GRASS module by
iterating over a list of values. Here's what I have;

  v.extract input="$GIS_OPT_INPUT" output="$GIS_OPT_OUTPUT""_"i \
type="point" layer=1 new=-1 list=i

The above is shell syntax.

In Python, it would look something like:

  import subprocess

  ...

  subprocess.call([
    "v.extract",
    "input=%s" % os.getenv("GIS_OPT_INPUT"),
    "output=%s_%s" % (os.getenv("GIS_OPT_OUTPUT"), i),
    "type=point", "layer=1", "new=-1", "list=%s" % i])

Glynn,

How do subprocess.call and subprocess.Popen compare for running GRASS commands from inside Python scripts? Is call easier than Popen in this context?

Michael

The main differences between Python and the shell:

1. All strings must be quoted.

2. Environment variables have to be accessed via os.getenv().

3. String manipulation requires explicit operators, e.g. +
(concatenation) or % (substitution).

4. Executing external programs requires the functions in the
subprocess module.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · July 16, 2008, 8:28am

Michael Barton wrote:

> subprocess.call([
> "v.extract",
> "input=%s" % os.getenv("GIS_OPT_INPUT"),
> "output=%s_%s" % (os.getenv("GIS_OPT_OUTPUT"), i),
> "type=point", "layer=1", "new=-1", "list=%s" % i])

Glynn,

How do subprocess.call and subprocess.Popen compare for running GRASS
commands from inside Python scripts? Is call easier than Popen in this
context?

subprocess.call is a convenience function for the simplest case where
you don't need to interact with the child process beyond waiting for
it to finish; it's defined as:

  def call(*args, **kwargs):
      """Run command with arguments. Wait for command to complete, then
      return the returncode attribute.

      The arguments are the same as for the Popen constructor. Example:

      retcode = call(["ls", "-l"])
      """
      return Popen(*args, **kwargs).wait()

It behaves like system(), but without the shell (so you don't have to
deal with /bin/sh syntax vs cmd.exe syntax, spaces and other shell
metacharacters in argument values, etc).

It's roughly equivalent to os.spawnvp(P_WAIT,...), which is deprecated
in favour of the subprocess module. Most of the other os.* functions
are similarly deprecated, except for the os.exec* family (for which
the main use is executing g.parser).

--
Glynn Clements <glynn@gclements.plus.com>

cmbarton · July 16, 2008, 3:32pm

On Jul 16, 2008, at 1:28 AM, Glynn Clements wrote:

Michael Barton wrote:

  subprocess.call([
    "v.extract",
    "input=%s" % os.getenv("GIS_OPT_INPUT"),
    "output=%s_%s" % (os.getenv("GIS_OPT_OUTPUT"), i),
    "type=point", "layer=1", "new=-1", "list=%s" % i])

Glynn,

How do subprocess.call and subprocess.Popen compare for running GRASS
commands from inside Python scripts? Is call easier than Popen in this
context?

subprocess.call is a convenience function for the simplest case where
you don't need to interact with the child process beyond waiting for
it to finish; it's defined as:

  def call(*args, **kwargs):
      """Run command with arguments. Wait for command to complete, then
      return the returncode attribute.

      The arguments are the same as for the Popen constructor. Example:

      retcode = call(["ls", "-l"])
      """
      return Popen(*args, **kwargs).wait()

It behaves like system(), but without the shell (so you don't have to
deal with /bin/sh syntax vs cmd.exe syntax, spaces and other shell
metacharacters in argument values, etc).

It's roughly equivalent to os.spawnvp(P_WAIT,...), which is deprecated
in favour of the subprocess module. Most of the other os.* functions
are similarly deprecated, except for the os.exec* family (for which
the main use is executing g.parser).

--
Glynn Clements <glynn@gclements.plus.com>

This seems very handy for a lot of basic GRASS scipting, including replacing bash scripts with Python ones--something that I'd like to start on soon and would encourage anyone wanting to get experience in Python to try.

Michael

Glynn_Clements1 · July 17, 2008, 6:53am

Michael Barton wrote:

This seems very handy for a lot of basic GRASS scipting, including
replacing bash scripts with Python ones--something that I'd like to
start on soon

Before we start, we need to figure out a suitable template for Python
scripts. I've recently fixed some bugs in the g.parser/test.py
example, but I've just discovered another issue:

g.parser needs to both open the script as a text file (to parse the
option definitions), and to execute it. For the former, the script
needs to be able to obtain its own absolute path.

However; the Python documentation for sys.argv says:

  The list of command line arguments passed to a Python script.
  argv[0] is the script name (it is operating system dependent
  whether this is a full pathname or not).

If it isn't a full path, sys.path[0] should be the directory
containing the script. So, the code which invokes g.parser needs to
force argv[0] to an absolute path before passing it to g.parser.

Even more fundamentally, we should question whether the existing
g.parser interface is the right way to go for Python. I'm thinking
that it would be better to give g.parser an option to write the
option/value pairs to stdout as a Python dictionary, e.g.:

output = Popen(["g.parser"] + sys.argv, stdout=PIPE).communicate()[0]
options = eval(output)

[The Popen(..., stdout=PIPE).communicate()[0] idiom is equivalent to
shell backticks; it returns a string containing the data written to
stdout.]

Actually; we probably want some error handling, e.g.:

  child = Popen(["g.parser"] + sys.argv, stdout=PIPE)
  output = child.communicate()[0]
  if child.returncode != 0:
      raise OSError("Error executing g.parser")
  options = eval(output)

Rather than having to put this in every script, we should provide a
module for it, and have the GRASS startup set PYTHONPATH.

--
Glynn Clements <glynn@gclements.plus.com>

cmbarton · July 18, 2008, 5:02am

These sound promising, though I admit to only partly following it.

Michael
____________________
C. Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

On Jul 16, 2008, at 11:53 PM, Glynn Clements wrote:

Michael Barton wrote:

This seems very handy for a lot of basic GRASS scipting, including
replacing bash scripts with Python ones--something that I'd like to
start on soon

Before we start, we need to figure out a suitable template for Python
scripts. I've recently fixed some bugs in the g.parser/test.py
example, but I've just discovered another issue:

g.parser needs to both open the script as a text file (to parse the
option definitions), and to execute it. For the former, the script
needs to be able to obtain its own absolute path.

However; the Python documentation for sys.argv says:

  The list of command line arguments passed to a Python script.
  argv[0] is the script name (it is operating system dependent
  whether this is a full pathname or not).

If it isn't a full path, sys.path[0] should be the directory
containing the script. So, the code which invokes g.parser needs to
force argv[0] to an absolute path before passing it to g.parser.

Even more fundamentally, we should question whether the existing
g.parser interface is the right way to go for Python. I'm thinking
that it would be better to give g.parser an option to write the
option/value pairs to stdout as a Python dictionary, e.g.:

  output = Popen(["g.parser"] + sys.argv, stdout=PIPE).communicate()[0]
  options = eval(output)

[The Popen(..., stdout=PIPE).communicate()[0] idiom is equivalent to
shell backticks; it returns a string containing the data written to
stdout.]

Actually; we probably want some error handling, e.g.:

  child = Popen(["g.parser"] + sys.argv, stdout=PIPE)
  output = child.communicate()[0]
  if child.returncode != 0:
      raise OSError("Error executing g.parser")
  options = eval(output)

Rather than having to put this in every script, we should provide a
module for it, and have the GRASS startup set PYTHONPATH.

--
Glynn Clements <glynn@gclements.plus.com>