[GRASS-dev] porting r.in.wms to python

Hi,

before anyone spends any time porting r.in.wms/r.tileset to python, I
would suggest that it be substantially rewritten from the ground up,
requiring GDAL 1.5+'s WMS import filter instead of our own brittle
wget/curl solution. This requires writing a small XML template with the
pararms: http://www.gdal.org/frmt_wms.html

The tile mosaic nightmare for paletted images might be outsourced to
GDAL's gdal_merge.py:
  http://www.gdal.org/gdal_merge.html
(ie merge then import, not the other way around)

I assume python has some great built in libexpat stuff for managing XML.
This could really help with the download list available capabilities,
again the current self-parsing works, but is pushing the boundaries of
what a bash script should be asked to do (and it shows).

Hopefully the thing can be greatly simplified in the process.

The current solution mostly works, but maintaining it feels a bit like a
game of whack-a-mole.

Perhaps there is some overlap with v.in.wfs as well? (currently relies on
lynx, which I guess is not commonly installed with MS-Windows..)

Hamish

Hamish wrote:

before anyone spends any time porting r.in.wms/r.tileset to python,

I was planning on leaving those until last :wink:

Are there any other scripts which don't need to be converted?

g.mlist/g.mremove are low priority now that there is a C version.

Do {r,r3}.mapcalculator need to be kept? ISTR that the original
motivation was the fact that {r,r3}.mapcalc couldn't be run from the
GUI due to not using G_parser(), but that's no longer the case.

I assume python has some great built in libexpat stuff for managing XML.
This could really help with the download list available capabilities,
again the current self-parsing works, but is pushing the boundaries of
what a bash script should be asked to do (and it shows).

Processing XML using xml.dom.minidom.parse() is very straightforward.
I wish that there was something similar for HTML/SGML; it would have
made replacing g.html2man much simpler.

Perhaps there is some overlap with v.in.wfs as well? (currently relies on
lynx, which I guess is not commonly installed with MS-Windows..)

In 7.0, it has been converted to Python, and uses Python's
urllib.urlopen() rather than lynx.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn Clements wrote:

> before anyone spends any time porting r.in.wms/r.tileset to python,

I was planning on leaving those until last :wink:

Well, I'm now planning on just leaving them, period :wink:

Here's the current status on conversion of scripts to Python:

The following scripts are disabled, and haven't been converted:

  d.out.gpsdrive (d.mon)
  d.rast.leg (d.frame)
  d.vect.thematic (d.mon)
  i.spectral (d.mon, d.where)
  r.mapcalculator (redundant)
  r3.mapcalculator (redundant)

We need to decide whether they are to be fixed or removed.

r.out.gdal.sh is still installed, but hasn't been converted. Is this
worth keeping around as a backup? If it is, I'll convert it, otherwise
it should be removed.

These:
  r.in.wms/r.in.wms
  r.tileset/r.tileset
  v.in.gpsbabel/v.in.gpsbabel
  v.in.garmin/v.in.garmin

are too complex to replace using "rote" conversion (i.e. simply
replacing each chunk of code with equivalent Python code).

The limitations of the Bourne shell mean that complex shell scripts
are invariably structured in a manner which doesn't make sense in any
other language.

The above scripts really need to be rewritten by someone who
understands the overall purpose of the script.

The remaining scripts have all been converted to Python (including
d.rast.edit, which was previously Tcl/Tk). The existing versions
remain for comparison, but are no longer installed.

--
Glynn Clements <glynn@gclements.plus.com>

On Sun, Oct 5, 2008 at 9:39 PM, Glynn Clements <glynn@gclements.plus.com> wrote:

r.out.gdal.sh is still installed, but hasn't been converted. Is this
worth keeping around as a backup? If it is, I'll convert it, otherwise
it should be removed.

It should be removed as incomplete + causing troubles.

Markus

Glynn Clements wrote:

Here's the current status on conversion of scripts to Python:

Apart from the scripts in the "scripts" directory, various parts of
GRASS install shell scripts. After filtering out the "shell" scripts
which are actually Tcl/Tk scripts using the "exec $GRASS_WISH" trick,
I see:

bin/nviz
etc/Init.sh
etc/gui/scripts/d.colors.sh
etc/gui/scripts/d.path.sh
etc/gui/scripts/r.colors.rules
etc/gui/scripts/r.reclass.file
etc/gui/scripts/r.reclass.rules
etc/gui/scripts/r.recode.file
etc/gui/scripts/r.recode.rules
etc/gui/scripts/r.support.sh
etc/prompt.sh
etc/r.li.setup/masked_area_selection
etc/r.li.setup/sample_area_vector.sh
etc/r.li.setup/square_mouse_selection.sh
etc/wxpython/scripts/p.cmd
etc/wxpython/scripts/p.db
etc/wxpython/scripts/p.mon
etc/wxpython/scripts/p.rast
etc/wxpython/scripts/p.vect
etc/wxpython/scripts/wxgui
scripts/gis.m
scripts/r.li.setup

Of these, Init.sh and prompt.sh should probably stay as shell scripts.
The former has an equivalent batch file for Windows users, and isn't
essential. The latter is only probably only useful if you already have
a Bourne shell.

r.li.setup and the scripts relating to nviz and gis.m all invoke
Tcl/Tk scripts. These should probably be re-written to use either
tclsh or wish.

Also, the three etc/gui/scripts/r.*.rules scripts all use
grass-xterm-wrapper, which no longer exists.
etc/gui/scripts/r.support.sh also uses grass-xterm-wrapper, but
r.support's curses-based interactive mode no longer exists.

The scripts relating to the wxPython GUI should be re-written in
Python (what do those scripts do, anyhow?).

That just leaves the three scripts in etc/r.li.setup. These are all
shell scripts which rely upon d.mon, so they won't work at present.

Once these issues are dealt with, there should no longer be any need
to distribute MSys as part of the WinGRASS packages.

Also, should we change windows_launch.bat to invoke Python on the
scripts, or just install the scripts with a .py extension and add .PY
to PATHEXT?

--
Glynn Clements <glynn@gclements.plus.com>

On Sun, 5 Oct 2008, Glynn Clements wrote:

Also, should we change windows_launch.bat to invoke Python on the
scripts, or just install the scripts with a .py extension and add .PY
to PATHEXT?

windows_launch.bat was only needed because there was no way to persuade Tcl on Windows to execute any files that did not have a .exe, .com or .bat extension. It didn't respect PATHEXT or anything similar. But windows_launch.bat turned out to be a neat way of handling things. However if
(1) Python respects PATHEXT and
(2) PATHEXT as a method of executing scripts is reliable
then I'm sure that would a good way of doing it and would be one less thing to go wrong. As long as they could be installed without the .py extension on Unix platforms I suppose.

On the other hand, for the PATHEXT method to work does Python have to be officially "installed" on Windows in some manner? If it does that might preclude just including the Python interpreter as part of a GRASS binary distribution, which someone might want to do.

I suppose I should really install Python on Windows and see if I can give a better answer.

Paul

before anyone spends any time porting r.in.wms/r.tileset to python,

Glynn wrote:

> I was planning on leaving those until last :wink:

Well, I'm now planning on just leaving them, period :wink:

Here's the current status on conversion of scripts to
Python:

The following scripts are disabled, and haven't been converted:

  d.out.gpsdrive (d.mon)

(me)
like the very useful d.out.file module, it dumps current (composed) map
display to the PNG driver via d.save. I suppose replacing this will
be a clone/modification of how d.out.file's functionality has been
replicated.

  d.rast.leg (d.frame)

(any thoughts Markus?)

perhaps replaced by some wxGUI "load cartographic template" tool?
(users could contribute their own etc..)

  d.vect.thematic (d.mon)

(somewhat replaced d.thematic.area,v.class? any thoughts Moritz/Michael?)

  i.spectral (d.mon, d.where)

It would be very easy to add a coord=x,y[,x2,y2,...] option to feed
r.what instead of using d.where. The d.mon check is just to ensure
that d.where will work.
More work would be to replace gnuplot with d.graph or whatever, like
is done for d.polar.

  r.mapcalculator (redundant)
  r3.mapcalculator (redundant)

and probably belonged in etc/gui/scripts/ all along.

We need to decide whether they are to be fixed or removed.

These:
  r.in.wms/r.in.wms
  r.tileset/r.tileset

I think it is ok for the replacement version to depend on GDAL > 1.5.0.
A first step will be adding XML+GDAL option to the bash/sh version
for testing of that method.

  v.in.gpsbabel/v.in.gpsbabel
  v.in.garmin/v.in.garmin

Certainly v.in.gpsbabel should survive in some form. v.in.garmin is
useful as the gpsbabel garmin filter does not pass through all fields.

the XML stuff in v.in.gpsbabel is in desperate need of a python
replacement, otherwise it shouldn't be /too/ different from v.in.mapgen.

are too complex to replace using "rote" conversion (i.e. simply
replacing each chunk of code with equivalent Python code).

The above scripts really need to be rewritten by someone
who understands the overall purpose of the script.

For v.in.gps I guess that means me, but my python+xml is rather weak.

For a Python version r.in.wms, I defer the larger project to someone
else, but can try to add GDAL support to the existing sh/bash version
to demonstrate/verify the method.

[...]
etc/gui/scripts/d.colors.sh
etc/gui/scripts/d.path.sh
etc/gui/scripts/r.colors.rules
etc/gui/scripts/r.reclass.file
etc/gui/scripts/r.reclass.rules
etc/gui/scripts/r.recode.file
etc/gui/scripts/r.recode.rules

these are all there as wrapper code between the GUI and command line
modules, usually WRT piping info from stdin. All should be replaced by
integrated wxGUI wizards not simply python scripts AFAICT.

etc/gui/scripts/r.support.sh was partially needed because
1) non-interactive version did not in the past support all interactive
functionality, and
2) interactive version weirdly exits prematurely when called directly
from Tcl/Tk.

Hamish

> d.out.gpsdrive (d.mon)

(me)
like the very useful d.out.file module, it dumps current (composed) map
display to the PNG driver via d.save. I suppose replacing this will
be a clone/modification of how d.out.file's functionality has been
replicated.

I should mention that while a little pre-alpha work has gone into building
GpsDrive for Win32 lately, it is essentially a *NIX program. It relies on
Gpsd, which states on their homepage "No, we don't support Windows — get
a better operating system." [RMS]

While not directly affecting the running of d.out.gpsdrive (to create
backdrop tiles), it does mean that if it was excluded from the MS-Windows
build I doubt anyone would notice much.

semi-related-- does anyone know of a libgis function that will return
the current projection+coordinate's convergence angle WRT true north?
A method to find that with the `proj` program is given here:
http://grass.osgeo.org/wiki/Ps.map_scripts#Creating_a_fancy_North_Arrow

Hamish

Paul Kelly wrote:

> Also, should we change windows_launch.bat to invoke Python on the
> scripts, or just install the scripts with a .py extension and add .PY
> to PATHEXT?

windows_launch.bat was only needed because there was no way to persuade
Tcl on Windows to execute any files that did not have a .exe, .com or .bat
extension. It didn't respect PATHEXT or anything similar. But
windows_launch.bat turned out to be a neat way of handling things. However if
(1) Python respects PATHEXT and
(2) PATHEXT as a method of executing scripts is reliable
then I'm sure that would a good way of doing it and would be one less
thing to go wrong. As long as they could be installed without the .py
extension on Unix platforms I suppose.

Oops. It turns out that Python doesn't understand %PATHEXT% (but it
least it's consistent; it doesn't work with .bat or .cmd files
either).

It appears that %PATHEXT%, assoc and ftype are features of cmd.exe.
CreateProcess() itself doesn't understand them:

  To run a batch file, you must start the command interpreter;
  set lpApplicationName to cmd.exe and set lpCommandLine to the
  following arguments: /c plus the name of the batch file.

[ http://msdn.microsoft.com/en-us/library/ms682425.aspx ]

It works if you use os.system(), or subprocess.Popen() with
shell=True, which isn't surprising. But none of the os.spawn* or
os.exec* functions work, nor subprocess.Popen() with shell=False.

Having said that, it should be simple enough to create a wrapper
around the Windows version of subprocess.Popen._execute_child() which
searches for the "executable" using PATH and PATHEXT, then
automatically prepends the interpreter where necessary. Or which uses
subprocess.list2cmdline() to convert the command to a string, then use
['cmd', '/c', cmdstring].

In any case, it appears that windows_launch.bat isn't an option, as we
would need to jump through the same hoops to be able to run .bat files
directly.

On the other hand, for the PATHEXT method to work does Python have to be
officially "installed" on Windows in some manner? If it does that might
preclude just including the Python interpreter as part of a GRASS binary
distribution, which someone might want to do.

In order for PATHEXT to work, ".py" has to be added to PATHEXT
somehow. We can do that in init.bat.

Also, in order to be able to execute a file without explicitly
specifying the interpreter, the extension and type have to be known to
"assoc" and "ftype". AFAICT, the "assoc" information is stored in:

  HKEY_LOCAL_MACHINE\SOFTWARE\Classes\.py

while the "ftype" information is stored in:

  HKEY_LOCAL_MACHINE\SOFTWARE\Classes\python.file\Shell\Open\Command

At least, deleting those keys causes assoc and ftype to immediately
"forget" the association.

However, these keys weren't set by the Python installer, although it
did set the keys under both HKEY_CURRENT_USER\Software\Classes and
HKEY_CLASSES_ROOT. AFAIK, those are the ones which Explorer uses.

If we want to handle this ourselves from within GRASS' Python scripts,
we can use the _winreg module (note: leading underscore), e.g.:

  >>> import _winreg
  >>> h = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Classes\.py')
  >>> (val, type) = _winreg.QueryValueEx(h, None)
  >>> print (val, type)
  (u'python.file', 1)
  >>> h = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Classes\python.file\Shell\Open\Command')
  >>> (val, type) = _winreg.QueryValueEx(h, None)
  >>> print (val, type)
  (u'"C:\\Program Files\\Python25\\python.exe" "%1"', 2)

1 is REG_SZ (NUL-terminated string), while 2 is REG_EXPAND_SZ
(NUL-terminated string containing environment substitutions).

--
Glynn Clements <glynn@gclements.plus.com>

Hamish wrote:

> > I was planning on leaving those until last :wink:
>
> Well, I'm now planning on just leaving them, period :wink:
>
> Here's the current status on conversion of scripts to
> Python:
>
> The following scripts are disabled, and haven't been converted:
>
> d.out.gpsdrive (d.mon)

(me)
like the very useful d.out.file module, it dumps current (composed) map
display to the PNG driver via d.save. I suppose replacing this will
be a clone/modification of how d.out.file's functionality has been
replicated.

IOW, it needs to be built into the GUI; now that monitors no longer
exist, nothing else has any notion of a "current" map.

> d.rast.leg (d.frame)

(any thoughts Markus?)

perhaps replaced by some wxGUI "load cartographic template" tool?
(users could contribute their own etc..)

Note that it is possible to set a drawing frame by setting
GRASS_FRAME=t,b,l,r (see d.polar for an example).

Actually, this should be salvageable without too much effort. I just
saw the d.frame calls and didn't look much further.

> i.spectral (d.mon, d.where)

It would be very easy to add a coord=x,y[,x2,y2,...] option to feed
r.what instead of using d.where. The d.mon check is just to ensure
that d.where will work.

That sounds simple enough.

More work would be to replace gnuplot with d.graph or whatever, like
is done for d.polar.

Is d.linegraph suitable?

> These:
> r.in.wms/r.in.wms
> r.tileset/r.tileset

I think it is ok for the replacement version to depend on GDAL > 1.5.0.
A first step will be adding XML+GDAL option to the bash/sh version
for testing of that method.

It might be simpler to just start from scratch in Python.

> v.in.gpsbabel/v.in.gpsbabel
> v.in.garmin/v.in.garmin

Certainly v.in.gpsbabel should survive in some form. v.in.garmin is
useful as the gpsbabel garmin filter does not pass through all fields.

the XML stuff in v.in.gpsbabel is in desperate need of a python
replacement, otherwise it shouldn't be /too/ different from v.in.mapgen.

Right. v.in.mapgen was another one that I was going to leave, but I
ended up downloading some sample files. That made it much easier to
see what the code was trying to do.

> are too complex to replace using "rote" conversion (i.e. simply
> replacing each chunk of code with equivalent Python code).

> The above scripts really need to be rewritten by someone
> who understands the overall purpose of the script.

For v.in.gps I guess that means me, but my python+xml is rather weak.

The Python documentation has an example at:

http://docs.python.org/library/xml.dom.minidom.html

Or I can write the code to parse the XML if you can explain the
overall structure. It's just that trying to deduce the data structure
from a sequence of grep/head/tail/cut commands is rather mind-numbing.

For a Python version r.in.wms, I defer the larger project to someone
else, but can try to add GDAL support to the existing sh/bash version
to demonstrate/verify the method.

That probably isn't much help. It would probably be more useful to
simply describe the process and provide some example URLs. I find
English much easier to read than bash/grep/head/tail/cut/awk/sed.

[...]
etc/gui/scripts/d.colors.sh
etc/gui/scripts/d.path.sh
etc/gui/scripts/r.colors.rules
etc/gui/scripts/r.reclass.file
etc/gui/scripts/r.reclass.rules
etc/gui/scripts/r.recode.file
etc/gui/scripts/r.recode.rules

these are all there as wrapper code between the GUI and command line
modules, usually WRT piping info from stdin. All should be replaced by
integrated wxGUI wizards not simply python scripts AFAICT.

Right. The various *.rules scripts plus d.colors.sh invoke xterm,
while the *.file versions just redirect stdin from a file (to
compensate for the lack of a file= option).

So, r.reclass and r.recode each need a file= (G_OPT_F_INPUT) option.

d.path.sh combines d.vect and d.path. The d.vect step was necessary
when d.path used the mouse to obtain the coordinates, so that the user
could see what they were supposed to be clicking on. That's no longer
applicable.

etc/gui/scripts/r.support.sh was partially needed because
1) non-interactive version did not in the past support all interactive
functionality, and
2) interactive version weirdly exits prematurely when called directly
from Tcl/Tk.

More generally, is there any intention to fix up gis.m with regard to
the various changes in 7.0, or are we going to abandon it and rely
upon wxgui instead?

--
Glynn Clements <glynn@gclements.plus.com>

Hamish wrote:

semi-related-- does anyone know of a libgis function that will return
the current projection+coordinate's convergence angle WRT true north?
A method to find that with the `proj` program is given here:
http://grass.osgeo.org/wiki/Ps.map_scripts#Creating_a_fancy_North_Arrow

The -V option is handled by vprocess() in proj.c (in PROJ); after
converting to/from lat/lon, it basically just calls pj_factors() (in
pj_factors.c) then prints out all of the values.

The following code (from r.proj) will get you the "struct pj_info" for
the current location.

    struct pj_info oproj;

    /* Get projection info for output mapset */
    if ((out_proj_info = G_get_projinfo()) == NULL)
  G_fatal_error(_("Unable to get projection info of output raster map"));

    if ((out_unit_info = G_get_projunits()) == NULL)
  G_fatal_error(_("Unable to get projection units of output raster map"));

    if (pj_get_kv(&oproj, out_proj_info, out_unit_info) < 0)
  G_fatal_error(_("Unable to get projection key values of output raster map"));

The "pj" field (i.e. oproj.pj) is the projPJ* which the PROJ library
wants. You should then be able to use pj_factors() directly to get the
convergence angle, e.g.:

  LP lp = { <lat, long> };
  struct FACTORS fact;
  pj_factors(lp, oproj.pj, 0.0, &fac);
  convergence = fac.conv;

--
Glynn Clements <glynn@gclements.plus.com>

> > v.in.gpsbabel/v.in.gpsbabel
> > v.in.garmin/v.in.garmin
>
> Certainly v.in.gpsbabel should survive in some form. v.in.garmin is
> useful as the gpsbabel garmin filter does not pass through all fields.
>
> the XML stuff in v.in.gpsbabel is in desperate need of a python
> replacement, otherwise it shouldn't be /too/ different from
> v.in.mapgen.

Right. v.in.mapgen was another one that I was going to leave, but I
ended up downloading some sample files. That made it much easier to
see what the code was trying to do.

[v.in.gpsbabel]
For importing points it uses the style file grass_write_ascii.style to
create a flat "x|y|alt|time_str|label" file which can be fed to v.in.ascii
points mode. Easy enough but throws away some data.

For importing lines it exports to GPX format (XML) and parses that into
v.in.ascii's standard mode format. For each track it also calculates
start/end time+coords, as only the overall line may take an attribute
but each vertex has its own timestamp.

You can create a test GPX file with 'v.out.ogr format=GPX'.
.... which leads to the obvious idea of importing GPX with v.in.ogr instead
of parsing to v.in.ascii. Populating DBs will probably be a minor snag.

It's just that trying to deduce the data structure
from a sequence of grep/head/tail/cut commands is rather
mind-numbing.

right. (& why I've stayed away from the GPX side of v.in.gpsbabel)
I can provide test files when the time comes. wpt,trk,rte:
  (3x GPX, 3x gpstrans, 3x gardump files)

they are all highly simple ascii, the complication comes from v.in.ascii
wanting number of vertices prior to reading a line, and splitting off
attribute columns for DB insertion.

> For a Python version r.in.wms, I defer the larger project to someone
> else, but can try to add GDAL support to the existing sh/bash version
> to demonstrate/verify the method.

That probably isn't much help.

The main goal there is not to create example code but to discover any
gotchas before investing too much effort into the method.

It would probably be more useful to simply describe the process and
provide some example URLs. I find English much easier to read than
bash/grep/head/tail/cut/awk/sed.

Parsing the capabilities.xml file will still be a chore regardless of
v.in.ogr to read the actual data.

Tile merging parts of the code can be probably be replaced by the python
scripts that come with GDAL.

Most of the gratuitous magic can probably be abandoned.

xml header setup:
http://www.gdal.org/frmt_wms.html

example URLs:
http://grass.osgeo.org/grass64/manuals/html64_user/r.in.wms.html

WMS spec in document #06-042:
http://www.opengeospatial.org/standards/wms
http://www.opengeospatial.org/standards/is

So, r.reclass and r.recode each need a file=
(G_OPT_F_INPUT) option.

better to make their existing rules= options interpret "-" to mean from
stdin (if they don't already) and keep the module interfaces that much
simpler.

d.path.sh combines d.vect and d.path. The d.vect step was necessary
when d.path used the mouse to obtain the coordinates, so that the user
could see what they were supposed to be clicking on.
That's no longer applicable.

if needed the full d.vect step could become a flag in d.path.

More generally, is there any intention to fix up gis.m with regard to
the various changes in 7.0, or are we going to abandon it and rely
upon wxgui instead?

My vote for grass7 would be to finally focus our finite energy on
supporting a single (wx) GUI.

Hamish

On Mon, Oct 6, 2008 at 4:16 AM, Hamish <hamish_b@yahoo.com> wrote:

      d.rast.leg (d.frame)

(any thoughts Markus?)

perhaps replaced by some wxGUI "load cartographic template" tool?
(users could contribute their own etc..)

d.rast.leg we use on a daily basis. Something similar would be needed
(cmd line support).

Markus

On Mon, Oct 6, 2008 at 7:34 AM, Glynn Clements <glynn@gclements.plus.com> wrote:

Hamish wrote:

...

etc/gui/scripts/r.support.sh was partially needed because
1) non-interactive version did not in the past support all interactive
functionality, and
2) interactive version weirdly exits prematurely when called directly
from Tcl/Tk.

More generally, is there any intention to fix up gis.m with regard to
the various changes in 7.0, or are we going to abandon it and rely
upon wxgui instead?

I thought we had already decided to remove gis.m, so only wxpython.

Markus

Markus Neteler wrote:

>> etc/gui/scripts/r.support.sh was partially needed because
>> 1) non-interactive version did not in the past support all interactive
>> functionality, and
>> 2) interactive version weirdly exits prematurely when called directly
>> from Tcl/Tk.
>
> More generally, is there any intention to fix up gis.m with regard to
> the various changes in 7.0, or are we going to abandon it and rely
> upon wxgui instead?

I thought we had already decided to remove gis.m, so only wxpython.

Eventually, sure. The question is whether it's worth maintaining gis.m
for a while, or whether wxgui is already sufficient.

--
Glynn Clements <glynn@gclements.plus.com>

On 06/10/08 04:16, Hamish wrote:

Glynn wrote:

  d.vect.thematic (d.mon)

(somewhat replaced d.thematic.area,v.class? any thoughts Moritz/Michael?)

d.thematic.area works for me, but I have not had any testing feedback from anyone else.
I still need to finish "d.thematic.points_and_lines" (still need to find a good name for it, or split it into two d.thematic.points and d.thematic.lines). Probably "just" a 1-2 days job for me (1-2 hours for others :wink: ), but I need a full stretch of time to be able to concentrate on it, and this is a very rare commodity in my life at this time (and has been for a while).

Moritz

On Mon, 6 Oct 2008, Glynn Clements wrote:

Oops. It turns out that Python doesn't understand %PATHEXT% (but it
least it's consistent; it doesn't work with .bat or .cmd files
either).

It looks to me like it doesn't work with .exe either, i.e. it doesn't assume the extension - e.g. trying to run a script on Windows I had to change line 139 of grass.py from
     os.execvp("g.parser", [name] + argv)
to
     os.execvp("g.parser.exe", [name] + argv)
to get it to do anything at all.

It appears that %PATHEXT%, assoc and ftype are features of cmd.exe.
CreateProcess() itself doesn't understand them:

  To run a batch file, you must start the command interpreter;
  set lpApplicationName to cmd.exe and set lpCommandLine to the
  following arguments: /c plus the name of the batch file.

[ http://msdn.microsoft.com/en-us/library/ms682425.aspx ]

It works if you use os.system(), or subprocess.Popen() with
shell=True, which isn't surprising. But none of the os.spawn* or
os.exec* functions work, nor subprocess.Popen() with shell=False.

Having said that, it should be simple enough to create a wrapper
around the Windows version of subprocess.Popen._execute_child() which
searches for the "executable" using PATH and PATHEXT, then
automatically prepends the interpreter where necessary. Or which uses
subprocess.list2cmdline() to convert the command to a string, then use
['cmd', '/c', cmdstring].

Looks like something like that will be necessary.

In any case, it appears that windows_launch.bat isn't an option, as we
would need to jump through the same hoops to be able to run .bat files
directly.

It's neat though in the sense that no registry editing is necessary to launch scripts that way, and it will work from the command-line.

On the other hand, for the PATHEXT method to work does Python have to be
officially "installed" on Windows in some manner? If it does that might
preclude just including the Python interpreter as part of a GRASS binary
distribution, which someone might want to do.

Also, in order to be able to execute a file without explicitly
specifying the interpreter, the extension and type have to be known to
"assoc" and "ftype". AFAICT, the "assoc" information is stored in:

      HKEY_LOCAL_MACHINE\SOFTWARE\Classes\.py

while the "ftype" information is stored in:

      HKEY_LOCAL_MACHINE\SOFTWARE\Classes\python.file\Shell\Open\Command

At least, deleting those keys causes assoc and ftype to immediately
"forget" the association.

However, these keys weren't set by the Python installer, although it
did set the keys under both HKEY_CURRENT_USER\Software\Classes and
HKEY_CLASSES_ROOT. AFAIK, those are the ones which Explorer uses.

If we want to handle this ourselves from within GRASS' Python scripts,
we can use the _winreg module (note: leading underscore), e.g.:

      >>> import _winreg
      >>> h = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE,

r'SOFTWARE\Classes\$

      >>> (val, type) = _winreg.QueryValueEx(h, None)
      >>> print (val, type)
      (u'python.file', 1)
      >>> h = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE,

r'SOFTWARE\Classes\$

      >>> (val, type) = _winreg.QueryValueEx(h, None)
      >>> print (val, type)
      (u'"C:\\Program Files\\Python25\\python.exe" "%1"', 2)

This seems more complicated than a solution analogous to windows_launch.bat with a GRASS_PYTHON environment variable. I'm not sure what's best.

Paul

Paul Kelly wrote:

> Oops. It turns out that Python doesn't understand %PATHEXT% (but it
> least it's consistent; it doesn't work with .bat or .cmd files
> either).

It looks to me like it doesn't work with .exe either, i.e. it doesn't
assume the extension - e.g. trying to run a script on Windows I had to
change line 139 of grass.py from
     os.execvp("g.parser", [name] + argv)
to
     os.execvp("g.parser.exe", [name] + argv)
to get it to do anything at all.

Full marks for consistency, I suppose.

> If we want to handle this ourselves from within GRASS' Python scripts,
> we can use the _winreg module (note: leading underscore), e.g.:

This seems more complicated than a solution analogous to
windows_launch.bat with a GRASS_PYTHON environment variable. I'm not sure
what's best.

Yeah, but we would need to do something similar for .bat files (except
that we don't need a separate variable; %COMSPEC% exists for that).

Essentially, there are two choices:

1. Mimic cmd.exe's handling of PATH, PATHEXT, assoc and ftype, i.e.
identify the full pathname corresponding to the command, then look up
the program used to run such files.

2. Invoke all commands via cmd.exe (well %COMSPEC%). For Popen(), it
should suffice to set shell=True, which prepends "%COMSPEC% /c" to the
argument string; it converts the argument list to a string regardless
of the value of "shell".

Either way, the issue is where to do this. The subprocess.Popen()
class defines a helper method, _execute_child(), which varies between
platforms. The POSIX version uses os.fork and os.execvp[e], while the
Windows version uses CreateProcess().

We can either create and use a subclass that overrides that method, or
we can simply replace the method with a wrapper. It depends on whether
we only need to affect direct use of Popen() from GRASS scripts, or
also affect "internal" use.

If we also need to affect the os.exec* and os.spawn* functions, we
need to override os._execvpe.

--
Glynn Clements <glynn@gclements.plus.com>

I'm not sure if it's relevant, but the wxpython GUI requires the pywin32 package to be installed. Perhaps this includes some workarounds for this sort of thing - I haven't looked into it to see. But I don't like the idea of requiring lots of extra packages - installing wxpython on top of Python is already a lot to ask a Windows user to do.

Paul

was: [GRASS-dev] porting r.in.wms to python

wrt: angle between grid north and true north

Hamish wrote:
> does anyone know of a libgis function that will return
> the current projection+coordinate's convergence angle WRT true north?
> A method to find that with the `proj` program is given here:
http://grass.osgeo.org/wiki/Ps.map_scripts#Creating_a_fancy_North_Arrow

Glynn:

The -V option is handled by vprocess() in proj.c (in PROJ); after
converting to/from lat/lon, it basically just calls pj_factors() (in
pj_factors.c) then prints out all of the values.

The following code (from r.proj) will get you the "struct pj_info" for
the current location.

    struct pj_info oproj;

    /* Get projection info for output mapset */
    if ((out_proj_info = G_get_projinfo()) == NULL)
  G_fatal_error(_("Unable to get projection info of output raster map"));

    if ((out_unit_info = G_get_projunits()) == NULL)
  G_fatal_error(_("Unable to get projection units of output raster map"));

    if (pj_get_kv(&oproj, out_proj_info, out_unit_info) < 0)
  G_fatal_error(_("Unable to get projection key values of output raster map"));

The "pj" field (i.e. oproj.pj) is the projPJ* which the PROJ library
wants. You should then be able to use pj_factors() directly to get the
convergence angle, e.g.:

  LP lp = { <lat, long> };
  struct FACTORS fact;
  pj_factors(lp, oproj.pj, 0.0, &fac);
  convergence = fac.conv;

Hi,

I am trying to add this as "g.region -n" now, some questions:

1) local_proto.h: what's next in this series:
#define PRINT_REG 0x01
#define PRINT_SH 0x02
#define PRINT_LL 0x04
#define PRINT_EXTENT 0x08
#define PRINT_CENTER 0x10
#define PRINT_METERS 0x20
#define PRINT_3D 0x40
#define PRINT_MBBOX 0x80

#define PRINT_NANGLE 0x ???

2) #include <projects.h> to get LP, FACTOR, and pj_factors() ?

3) The compiler complains that arg 2 of pj_factors(, oproj.pj,) is of
an incompatible pointer type. If I make it *oproj.pj it doesn't complain
but the output fact.conv is always 0.00000.

?

thanks,
Hamish