[GRASS-dev] [GRASS GIS] #528: wxGUI cmd: r.mapcalc fails

GDAL · March 13, 2009, 7:21am

#528: wxGUI cmd: r.mapcalc fails
-------------------------+--------------------------------------------------
Reporter: neteler | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: default | Version: unspecified
Keywords: | Platform: Unspecified
Cpu: Unspecified |
-------------------------+--------------------------------------------------
Running in Cmd> of wxGUI the following commands (NC data set):
{{{
r.buffer lakes out=lakes_buff dist=60,120,240,500 --o
r.mapcalc "developed_lake=if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())"
}}}

leads to
{{{
r.mapcalc "developed_lake=if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())"
syntax error, unexpected $end, expecting '='
Parse error
}}}
Maybe the " chars are not protected?

Markus

Unrelated idea: make the > of Cmd > clickable to clear the line?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 7:22am

#528: wxGUI cmd: r.mapcalc fails
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Changes (by neteler):

  * platform: Unspecified => All
  * version: unspecified => svn-develbranch6
  * component: default => wxGUI
  * cpu: Unspecified => All

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 8:03am

#528: wxGUI cmd: r.mapcalc fails
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by martinl):

Replying to [ticket:528 neteler]:
{{{
> r.mapcalc "developed_lake=if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())"
> syntax error, unexpected $end, expecting '='
> Parse error
}}}
> Maybe the " chars are not protected?

yes, anyway it should without quotation marks

{{{
r.mapcalc developed_lake = if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())
}}}

Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 8:24am

#528: wxGUI cmd: r.mapcalc fails
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by martinl):

Replying to [ticket:528 neteler]:
> Unrelated idea: make the > of Cmd > clickable to clear the line?

Done in r36354 (trunk) and r36355 (devbr6).

Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 8:38am

#528: wxGUI cmd: r.mapcalc fails
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by hellik):

Replying to [comment:2 martinl]:
> Replying to [ticket:528 neteler]:
> {{{
> > r.mapcalc "developed_lake=if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())"
> > syntax error, unexpected $end, expecting '='
> > Parse error
> }}}
> > Maybe the " chars are not protected?
>
> yes, anyway it should without quotation marks
>
> {{{
> r.mapcalc developed_lake = if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())
> }}}
>
> Martin

i tried it on the wx-cmd-line (osgeo4w, win vista) without quotation marks
and the calculation is done.

maybe there could be a hint for this in
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/grasstestrast.txt

helmut

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 8:54am

#528: wxGUI cmd: r.mapcalc fails
----------------------+-----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
----------------------+-----------------------------------------------------
Comment (by neteler):

Replying to [comment:2 martinl]:
> Replying to [ticket:528 neteler]:
> {{{
> > r.mapcalc "developed_lake=if(landuse96_28m==1 || landuse96_28m==2,
lakes_buff, null())"
> > syntax error, unexpected $end, expecting '='
> > Parse error
> }}}
> > Maybe the " chars are not protected?
>
> yes, anyway it should without quotation marks

This would break compatibility to the common command line (hence, in all
books, tutorials, courses etc this hint would be required which is rather
cluttering stuff). If technically possible (and time permits) I hope that
quotes can be maintained also in the wx-cmd line.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 11:27am

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Changes (by glynn):

* priority: major => critical

Comment:

Replying to [comment:5 neteler]:

> > > Maybe the " chars are not protected?

More likely, they aren't *interpreted*, e.g. something is splitting the
command line into arguments, treating every spaces as a separator.

> > yes, anyway it should without quotation marks

Not in 7.0 it won't, nor will that work for e.g. title=... or filename
arguments which need to accept strings containing spaces.

> This would break compatibility to the common command line (hence, in all
books, tutorials, courses etc this hint would be required which is rather
cluttering stuff). If technically possible (and time permits) I hope that
quotes can be maintained also in the wx-cmd line.

If the GUI is to provide a command line, it needs to actually support the
basic features of any command-line interface, e.g. the ability to have
spaces, quotes etc within arguments.

If it simply passing the command on to the OS, it should probably just
pass shell=True to Popen() rather than trying to parse it itself. If it
needs to know how the command is parsed, I suggest writing a small helper
program which writes its argv to stdout, separated by NULs, then have
the GUI invoke that program via the shell and parse its output.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 3:18pm

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by cmbarton):

Currently, the command line is passing the command to the OS, with some
simple parsing to 1) check for valid GRASS commands and 2) to get it into
Popen(). A full terminal is possible via Python and wxPython, but it will
take 1) some considerable reprogramming and 2) some space (maybe could
replace the output window). It's probably a very good idea but I would
prioritize it below some other more critical issues to be fixed.

We need more than Martin and I on the GUI team to help explore some of
these good ideas and fix existing bugs (e.g., the attribute manager is now
broken for Mac).

This seems more of an enhancement request than a defect.

Michael

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 3:33pm

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by cmbarton):

Forgot to mention. The ">" is just text, but a clear button could be
added. Another good idea.

Michael

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 7:57pm

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by glynn):

Replying to [comment:7 cmbarton]:

> This seems more of an enhancement request than a defect.

The inability to have spaces within arguments is a defect, and a rather
fundamental one.

If the GUI doesn't need to understand the command(s), the string should be
passed verbatim to Popen(..., shell=True). But this will allow arbitrary
shell commands, including redirection, pipelines, for/if/while, etc.

If you need to parse a string into program and argv, consider using
shlex.split():

http://docs.python.org/library/shlex.html

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:9>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 13, 2009, 9:07pm

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by cmbarton):

I'm not sure we're talking about exactly the same thing.

Currently the very simple command processor does not require quotes around
the main r.mapcalc argument. This is also the case in other settings,
including the TclTk command processor. Quotes are not required around the
main r.mapcalc argument in the terminal on my Mac either. That was the
original issue. I don't see this, in and of itself as a defect--though I
suppose others might disagree.

Spaces, in and of themselves are not a problem in the command processor.
The following works fine, for example

r.mapcalc newmap = 5

Note the lack of quotes and the spaces on either side of the "=".

I think the command processor also allows strings with spaces surrounded
by quotes within arguments if a command requires them, but I would need to
test this on a particular set of cases. I currently can't think of an
appropriate test, but I'm sure there are such cases. If it does not allow
spaces within an argument (as allowed by a GRASS command), this does need
to be fixed if possible so that any GRASS command can be parsed correctly.

Michael

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:10>
GRASS GIS <http://grass.osgeo.org>

neteler · March 13, 2009, 11:39pm

Michael,

...

Currently the very simple command processor does not require quotes around
the main r.mapcalc argument.

But it should accept them. Otherwise folks copy command from
books and tutorials and get an error.

Spaces, in and of themselves are not a problem in the command processor.
The following works fine, for example

r.mapcalc newmap = 5

Note the lack of quotes and the spaces on either side of the "=".

Please try the shell valid example

r.mapcalc "newmap = 5"

You get a parse error in the wxGUI cmd which I find confusing
since it differs from the "normal" cmd line behaviour.

Markus

GDAL · March 14, 2009, 11:25pm

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by glynn):

Replying to [comment:10 cmbarton]:

> I'm not sure we're talking about exactly the same thing.
>
> Currently the very simple command processor does not require quotes
around the main r.mapcalc argument. This is also the case in other
settings, including the TclTk command processor. Quotes are not required
around the main r.mapcalc argument in the terminal on my Mac either. That
was the original issue. I don't see this, in and of itself as a defect--
though I suppose others might disagree.

This isn't true in 7.0, where r.mapcalc uses the parser. The 6.x r.mapcalc
is a special case, as it simply concatenates all of its arguments,
separated by spaces, which eliminates the need for quoting. Any command
which uses the parser requires quoting (or escaping) if an argument
contains spaces.

> Spaces, in and of themselves are not a problem in the command processor.
The following works fine, for example
>
> r.mapcalc newmap = 5
>
> Note the lack of quotes and the spaces on either side of the "=".

r.mapcalc is a poor example, for the reason given above.

> I think the command processor also allows strings with spaces surrounded
by quotes within arguments if a command requires them,

The original report suggests otherwise. That is the real problem here. The
fact that the 6.x r.mapcalc provides a workaround doesn't help in general.

> but I would need to test this on a particular set of cases. I currently
can't think of an appropriate test, but I'm sure there are such cases.

Choose any command which accepts a filename argument, and pass a filename
which contains spaces.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:11>
GRASS GIS <http://grass.osgeo.org>

GDAL · March 15, 2009, 11:22am

#528: wxGUI cmd: r.mapcalc fails
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by hellik):

Replying to [comment:11 glynn]:
> Replying to [comment:10 cmbarton]:
>
>
> The original report suggests otherwise. That is the real problem here.
The fact that the 6.x r.mapcalc provides a workaround doesn't help in
general.
>
> > but I would need to test this on a particular set of cases. I
currently can't think of an appropriate test, but I'm sure there are such
cases.
>
> Choose any command which accepts a filename argument, and pass a
filename which contains spaces.

maybe related?

platform grass64-rc3-3 (osgeo4w) win vista with wx-gui

command taken from the short tutorial in
http://www.grassbook.org/examples3rd_chapter6.php

copied from the website to the wx-gui command-line:
v.extract stopsall out=stops1 where="ROUTES LIKE '%1%' AND ROUTES NOT LIKE
'%11%'"

following error-message:

v.extract stopsall out=stops1 where="ROUTES LIKE '%1%' AND ROUTES NOT LIKE
'%11%'"
Sorry <LIKE> is not a valid option
Sorry <'%1%'> is not a valid option
Sorry <AND> is not a valid option
Sorry <ROUTES> is not a valid option
Sorry <NOT> is not a valid option
Sorry <LIKE> is not a valid option
Sorry <'%11%'"> is not a valid option
Description:
  Selects vector objects from an existing vector map and
creates a new map containing only the selected objects.
Keywords:
  vector, extract
Usage:
  v.extract [-dtr] input=name output=name [...]

the working command in the v.extract-gui looks like:

v.extract input=stopsall@wolfline_lrs output=stops1 type=point
where=ROUTES LIKE '%1%' AND ROUTES NOT LIKE '%11%'

    25 categories loaded from table <stopsall>
    Extracting features...
    Building topology for vector map <stops1>...
    Registering primitives...
    25 primitives registered [...]

but if i copy the working command of the v.extract-gui to the wx-gui-cmd,
the i get again the first error message with

v.extract input=stopsall@wolfline_lrs output=stops1 type=point
where=ROUTES LIKE '%1%' AND ROUTES NOT LIKE '%11%'
Sorry <LIKE> is not a valid option [...]

Helmut

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:12>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 21, 2009, 3:10pm

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Changes (by hamish):

* summary: wxGUI cmd: r.mapcalc fails => wxGUI cmd: r.mapcalc fails
(quoting + args with spaces)

Comment:

see also bug #604 for the GIS.m equivalent.

spearfish test case:
{{{
v.db.select roads where="label ~ 'highway'"
}}}

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/528#comment:13>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 22, 2009, 11:47am

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by hamish):

Hi,

I just added a patch to this ticket (vs devbr6) which roughly uses
`shlex.split(command)` to parse the command string in a shell-like way. It
doesn't actually work but I think it gets close. something like the
`command = ' '.join(command)` from the Xterm fn is happening so it just
tries to execute the first letter. Hopefully a more learned pythoner can
quickly finish the job from here.

Hamish

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/528#comment:14>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 23, 2009, 11:49am

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by martinl):

Replying to [comment:14 hamish]:
> I just added a patch to this ticket (vs devbr6) which roughly uses
`shlex.split(command)` to parse the command string in a shell-like way. It
doesn't actually work but I think it gets close. something like the
`command = ' '.join(command)` from the Xterm fn is happening so it just
tries to execute the first letter. Hopefully a more learned pythoner can
quickly finish the job from here.

Modified path applied in trunk (r37395), devbr6 (r37396) and relbr64
(r37397).

Testing welcomed ...

Martin

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/528#comment:15>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 24, 2009, 5:07am

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by hamish):

Replying to [comment:15 martinl]:
> Modified path applied in trunk (r37395), devbr6 (r37396) and relbr64
(r37397).

thanks.

> Testing welcomed ...

* autoparsing is nice in theory but slows things down and breaks for non-
grass commands. e.g. "ls -l" or "gdalinfo --version"
* left and right arrows don't work to move the cursor

{{{
in the terminal:
ls: unrecognised option `--interface-description'
Try `ls --help' for more information.
...

in the Command output tab:
...
self._module = menuform.GUI().ParseInterface(cmd = cmd)
File "/usr/src/grass/svn/develbranch_6/dist.i686-pc-linux-
gnu/etc/wxpython/gui_modules/menuform.py", line 1809, in
ParseInterface

xml.sax.parseString(getInterfaceDescription(cmd[0]).decode(e
nc).encode("utf-8"),
File "/usr/src/grass/svn/develbranch_6/dist.i686-pc-linux-
gnu/etc/wxpython/gui_modules/menuform.py", line 1769, in
getInterfaceDescription

raise IOError, _("Unable to fetch interface description for
command '%s'.") % cmd
IOError
:
Unable to fetch interface description for command 'ls'.
}}}

gdalinfo --version:
{{{
File "/usr/lib/python2.5/xml/sax/xmlreader.py", line 123,
in parse

self.feed(buffer)
File "/usr/lib/python2.5/xml/sax/expatreader.py", line
211, in feed

self._err_handler.fatalError(exc)
File "/usr/lib/python2.5/xml/sax/handler.py", line 38, in
fatalError

raise exception
xml.sax._exceptions
.
SAXParseException
:
<unknown>:1:0: syntax error
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/528#comment:16>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 24, 2009, 5:17am

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by hamish):

also when typing in
{{{
v.db.select roads where="label ~ ...
}}}

you get an error:
{{{
...
raw = self.read_token()
File "/usr/lib/python2.5/shlex.py", line 172, in
read_token

raise ValueError, "No closing quotation"
ValueError
:
No closing quotation
}}}

can auto-completion be restricted to only run if you press the tab key?

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:17>
GRASS GIS <http://grass.osgeo.org>

GDAL · May 24, 2009, 11:07am

#528: wxGUI cmd: r.mapcalc fails (quoting + args with spaces)
-----------------------+----------------------------------------------------
  Reporter: neteler | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: wxGUI | Version: svn-develbranch6
Resolution: | Keywords:
  Platform: All | Cpu: All
-----------------------+----------------------------------------------------
Comment (by martinl):

Replying to [comment:16 hamish]:
> * autoparsing is nice in theory but slows things down and breaks for
non-grass commands. e.g. "ls -l" or "gdalinfo --version"

> * left and right arrows don't work to move the cursor

Note that autocomplete propmt is in very early stage of development.
Autoparsing should be reduced only to GRASS commands.

Martin

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/528#comment:18>
GRASS GIS <http://grass.osgeo.org>