[GRASS-dev] Python Scripting

GRASS GRASS developer
1

On Jul 18, 2008, at 11:03 PM, <grass-dev-request@lists.osgeo.org> wrote:

Date: Fri, 18 Jul 2008 18:20:46 +0100
From: Glynn Clements <glynn@gclements.plus.com>
Subject: Re: [GRASS-dev] Python Scripting
To: "Dan D'Alimonte" <dan@dalimonte.ca>, grass-dev@lists.osgeo.org
Message-ID: <18560.53486.897323.701173@cerise.gclements.plus.com>
Content-Type: text/plain; charset="us-ascii"

Glynn Clements wrote:

As to existing modules, what about a helper function to access then?

module.executeModule( name="r.stats", options={ "input":
"elevation.dem,slope,aspect", "fs": ",", "output": "elev.csv"},
flags=["q", "1", "n", "g"] )

This idea has occurred to me. Some comments:

Pass argument values as Python values, e.g. passing multiple values as
lists, passing numeric types directly, etc, and have the interface
convert them to strings. Pass the flags as a single string.

module.execute( "r.stats",
               options = { "input": ["elevation.dem", "slope", "aspect"],
                           "fs": ",",
                           "output": "elev.csv" },
               flags = "q1ng" )

Provide a lower-level function which simply generates the command to
pass to Popen(), for cases where you want to interact with the child
process.

I have attached a first draft of such a module, which also includes a
wrapper function for g.parser (for which an example script is also
attached).

--
Glynn Clements <glynn@gclements.plus.com>

-------------- next part --------------
import os
import os.path
import sys
import types
import subprocess

def _make_val(val):
   if isinstance(val, types.StringType):
  return val
   if isinstance(val, types.ListType):
  return ",".join(map(_make_val, val))
   if isinstance(val, types.TupleType):
  return _make_val(list(val))
   return str(val)

def make_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False):
   args = [prog]
   if overwrite:
  args.append("--o")
   if quiet:
  args.append("--q")
   if verbose:
  args.append("--v")
   if flags:
  args.append("-%s" % flags)
   for opt, val in options.iteritems():
  args.append("%s=%s" % (opt, _make_val(val)))
   return args

def start_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False, **kwargs):
   args = make_command(prog, options, flags, overwrite, quiet, verbose)
   return subprocess.Popen(args, **kwargs)

def run_command(*args, **kwargs):
   ps = start_command(*args, **kwargs)
   return ps.wait()

def _parse_env():
   options = {}
   flags = {}
   for var, val in os.environ.iteritems():
  if var.startswith("GIS_OPT_"):
      opt = var.replace("GIS_OPT_", "", 1).lower()
      options[opt] = val;
  if var.startswith("GIS_FLAG_"):
      flg = var.replace("GIS_FLAG_", "", 1).lower()
      flags[flg] = bool(int(val));
   return (options, flags)

def parser():
   if not os.getenv("GISBASE"):
       print >> sys.stderr, "You must be in GRASS GIS to run this program."
       sys.exit(1)

   if len(sys.argv) > 1 and sys.argv[1] == "@ARGS_PARSED@":
  return _parse_env()

   argv = sys.argv[:]
   name = argv[0]
   if not os.path.isabs(name):
  argv[0] = os.path.normpath(os.path.join(sys.path[0], name))

   os.execvp("g.parser", [name] + argv)
   raise OSError("error executing g.parser")
-------------- next part --------------
#!/usr/bin/env python

# g.parser demo script for python programing

import grass

#%Module
#% description: g.parser test script (python)
#%End
#%flag
#% key: f
#% description: a flag
#%END
#%option
#% key: raster
#% type: string
#% gisprompt: old,cell,raster
#% description: raster input map
#% required : yes
#%end
#%option
#% key: vector
#% type: string
#% gisprompt: old,vector,vector
#% description: vector input map
#% required : yes
#%end
#%option
#% key: option1
#% type: string
#% description: an option
#% required : no
#%end

if __name__ == "__main__":
   (options, flags) = grass.parser()

   print ""
   if flags['f']:
       print "Flag -f set"
   else:
       print "Flag -f not set"

   #test if parameter present:
   if options['option1']:
       print "Value of option1: '%s'" % options['option1']

   print "Value of raster= option: '%s'" % options['raster']
   print "Value of vector= option: '%s'" % options['vector']

Do you still run a GRASS command as subprocess.call([command...])? Or is there another syntax with your wrapper script (e.g., as in the module.execute() above)?

Michael

2

Michael Barton wrote:

On Jul 18, 2008, at 11:03 PM, <grass-dev-request@lists.osgeo.org> wrote:

> Date: Fri, 18 Jul 2008 18:20:46 +0100
> From: Glynn Clements <glynn@gclements.plus.com>
> Subject: Re: [GRASS-dev] Python Scripting
> To: "Dan D'Alimonte" <dan@dalimonte.ca>, grass-dev@lists.osgeo.org
> Message-ID: <18560.53486.897323.701173@cerise.gclements.plus.com>
> Content-Type: text/plain; charset="us-ascii"
>
>
> Glynn Clements wrote:
>
>>> As to existing modules, what about a helper function to access then?
>>>
>>> module.executeModule( name="r.stats", options={ "input":
>>> "elevation.dem,slope,aspect", "fs": ",", "output": "elev.csv"},
>>> flags=["q", "1", "n", "g"] )
>>
>> This idea has occurred to me. Some comments:
>>
>> Pass argument values as Python values, e.g. passing multiple values
>> as
>> lists, passing numeric types directly, etc, and have the interface
>> convert them to strings. Pass the flags as a single string.
>>
>> module.execute( "r.stats",
>> options = { "input": ["elevation.dem", "slope", "aspect"],
>> "fs": ",",
>> "output": "elev.csv" },
>> flags = "q1ng" )
>>
>> Provide a lower-level function which simply generates the command to
>> pass to Popen(), for cases where you want to interact with the child
>> process.
>
> I have attached a first draft of such a module, which also includes a
> wrapper function for g.parser (for which an example script is also
> attached).

Do you still run a GRASS command as subprocess.call([command...])? Or
is there another syntax with your wrapper script (e.g., as in the
module.execute() above)?

Yes, although it's actually called grass.run_command().

Specifically, this:

  def make_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False):
    ...

constructs a list suitable for use as the "args" argument to the
Popen() constructor or to call(). E.g.:

  >>> import grass
  >>> grass.make_command( "r.stats",
                  options = { "input": ["elevation.dem", "slope", "aspect"],
                              "fs": ",",
                              "output": "elev.csv" },
                  flags = "1ng" )
  ['r.stats', '-1ng', 'input=elevation.dem,slope,aspect', 'fs=,', 'output=elev.csv']

This:

  def start_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False, **kwargs):
      args = make_command(prog, options, flags, overwrite, quiet, verbose)
      return subprocess.Popen(args, **kwargs)

does just that: constructs the argument list then passes it to the
Popen() constructor, along with any additional keyword arguments (so
you can set stdin, stdout, etc), and returns the Popen() object. E.g.:

  >>> import sys
  >>> import subprocess
  >>> import grass
  >>> p = grass.start_command( "g.list", options = { "type": "rast" }, stdout = subprocess.PIPE )
  >>> txt = p.communicate()[0]
  >>> sys.stdout.write(txt)
  ----------------------------------------------
  raster files available in mapset <PERMANENT>:
[snip]

Finally, this:

  def run_command(*args, **kwargs):
      ps = start_command(*args, **kwargs)
      return ps.wait()

is analogous to call(), but with the GRASS-oriented interface of
make_command and start_command, e.g.:

  >>> import grass
  >>> grass.run_command( "g.list", options = { "type": "rast" } )
  ----------------------------------------------
  raster files available in mapset <PERMANENT>:
[snip]

The option values can be strings, numbers, tuples, or lists, and are
converted appropriately; numbers (well, anything except for strings,
tuples and lists) are converted with str(), strings are taken
literally (i.e. they aren't quoted), tuples and lists have their
components converted and separated by commas, e.g.:

  >>> import grass
  >>> grass.make_command( "prog", options = { "arg": [(1,10),(2,20)] } )
  ['prog', 'arg=1,10,2,20']

It has just occurred to me that it might be better to take the options
as keyword arguments, rather than an explicit dictionary, e.g.:

  grass.make_command( "r.stats",
    flags = "1ng",
    input = ["elevation.dem", "slope", "aspect"],
    fs = ",",
    output = "elev.csv" )

You could still pass a dictionary using the ** syntax:

  opts = {"input": ["elevation.dem", "slope", "aspect"],
    "fs": ",",
    "output": "elev.csv" }
  grass.make_command( "r.stats", flags = "1ng", **opts)

This would be trivial to implement:

-def make_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False):
+def make_command(prog, flags = "", overwrite = False, quiet = False, verbose = False, **options):

but there is the (remote) possibility that a module option could
conflict with one of the predefined arguments (prog, flags, overwrite,
quiet, or verbose), or with one of Popen()'s arguments (which
run_command and start_command would have to handle explicitly, rather
than using **kwargs).

--
Glynn Clements <glynn@gclements.plus.com>

3

On Jul 19, 2008, at 12:01 PM, Glynn Clements wrote:

Michael Barton wrote:

On Jul 18, 2008, at 11:03 PM, <grass-dev-request@lists.osgeo.org> wrote:

Date: Fri, 18 Jul 2008 18:20:46 +0100
From: Glynn Clements <glynn@gclements.plus.com>
Subject: Re: [GRASS-dev] Python Scripting
To: "Dan D'Alimonte" <dan@dalimonte.ca>, grass-dev@lists.osgeo.org
Message-ID: <18560.53486.897323.701173@cerise.gclements.plus.com>
Content-Type: text/plain; charset="us-ascii"

Glynn Clements wrote:

As to existing modules, what about a helper function to access then?

module.executeModule( name="r.stats", options={ "input":
"elevation.dem,slope,aspect", "fs": ",", "output": "elev.csv"},
flags=["q", "1", "n", "g"] )

This idea has occurred to me. Some comments:

Pass argument values as Python values, e.g. passing multiple values
as
lists, passing numeric types directly, etc, and have the interface
convert them to strings. Pass the flags as a single string.

module.execute( "r.stats",
              options = { "input": ["elevation.dem", "slope", "aspect"],
                          "fs": ",",
                          "output": "elev.csv" },
              flags = "q1ng" )

Provide a lower-level function which simply generates the command to
pass to Popen(), for cases where you want to interact with the child
process.

I have attached a first draft of such a module, which also includes a
wrapper function for g.parser (for which an example script is also
attached).

Do you still run a GRASS command as subprocess.call([command...])? Or
is there another syntax with your wrapper script (e.g., as in the
module.execute() above)?

Yes, although it's actually called grass.run_command().

Specifically, this:

  def make_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False):
    ...

constructs a list suitable for use as the "args" argument to the
Popen() constructor or to call(). E.g.:

  >>> import grass
  >>> grass.make_command( "r.stats",
                  options = { "input": ["elevation.dem", "slope", "aspect"],
                              "fs": ",",
                              "output": "elev.csv" },
                  flags = "1ng" )
  ['r.stats', '-1ng', 'input=elevation.dem,slope,aspect', 'fs=,', 'output=elev.csv']

This:

  def start_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False, **kwargs):
      args = make_command(prog, options, flags, overwrite, quiet, verbose)
      return subprocess.Popen(args, **kwargs)

does just that: constructs the argument list then passes it to the
Popen() constructor, along with any additional keyword arguments (so
you can set stdin, stdout, etc), and returns the Popen() object. E.g.:

  >>> import sys
  >>> import subprocess
  >>> import grass
  >>> p = grass.start_command( "g.list", options = { "type": "rast" }, stdout = subprocess.PIPE )
  >>> txt = p.communicate()[0]
  >>> sys.stdout.write(txt)
  ----------------------------------------------
  raster files available in mapset <PERMANENT>:
[snip]

Finally, this:

  def run_command(*args, **kwargs):
      ps = start_command(*args, **kwargs)
      return ps.wait()

is analogous to call(), but with the GRASS-oriented interface of
make_command and start_command, e.g.:

  >>> import grass
  >>> grass.run_command( "g.list", options = { "type": "rast" } )
  ----------------------------------------------
  raster files available in mapset <PERMANENT>:
[snip]

The option values can be strings, numbers, tuples, or lists, and are
converted appropriately; numbers (well, anything except for strings,
tuples and lists) are converted with str(), strings are taken
literally (i.e. they aren't quoted), tuples and lists have their
components converted and separated by commas, e.g.:

  >>> import grass
  >>> grass.make_command( "prog", options = { "arg": [(1,10),(2,20)] } )
  ['prog', 'arg=1,10,2,20']

It has just occurred to me that it might be better to take the options
as keyword arguments, rather than an explicit dictionary, e.g.:

  grass.make_command( "r.stats",
    flags = "1ng",
    input = ["elevation.dem", "slope", "aspect"],
    fs = ",",
    output = "elev.csv" )

You could still pass a dictionary using the ** syntax:

  opts = {"input": ["elevation.dem", "slope", "aspect"],
    "fs": ",",
    "output": "elev.csv" }
  grass.make_command( "r.stats", flags = "1ng", **opts)

This would be trivial to implement:

-def make_command(prog, options = , flags = "", overwrite = False, quiet = False, verbose = False):
+def make_command(prog, flags = "", overwrite = False, quiet = False, verbose = False, **options):

but there is the (remote) possibility that a module option could
conflict with one of the predefined arguments (prog, flags, overwrite,
quiet, or verbose), or with one of Popen()'s arguments (which
run_command and start_command would have to handle explicitly, rather
than using **kwargs).

Glynn,

I am probably missing something. But I guess I don't understand the advantage of using the modules available from this grass library over using normal Python functions to run a GRASS command. That is, the syntax for running a command here doesn't seem any easier--and maybe a shade more complicated than simply using subprocess.call for simple one-shot commands and subprocess.Popen for commands where you need to return stdout or stderr.

Are there other benefits to using a grass library that I'm not understanding?

Michael

4

Michael Barton wrote:

I am probably missing something. But I guess I don't understand the
advantage of using the modules available from this grass library over
using normal Python functions to run a GRASS command. That is, the
syntax for running a command here doesn't seem any easier--and maybe a
shade more complicated than simply using subprocess.call for simple
one-shot commands and subprocess.Popen for commands where you need to
return stdout or stderr.

Are there other benefits to using a grass library that I'm not
understanding?

The idea is to avoid scripts all needing to include similar code.

The code examples aren't particularly good examples, as they all have
hard-coded literal option values. If you're running a command with
fixed options, then subprocess.call is straightforward. But most
practical scripts will be using Python expressions as option values,
so they will end up needing the equivalent of make_command. E.g.:

  grass.make_command( "r.category",
    map = map,
    cats = cats )

versus:

  subprocess.call(["r.category",
    "map=%s" % map,
    "cats=%s" % ",".join(map(str,cats))])

--
Glynn Clements <glynn@gclements.plus.com>