[GRASS-dev] GRASS 7 scripts overhaul

neteler · September 4, 2011, 10:20am

Hi,

I have tested almost all GRASS 7 scripts and fixed a lot of them (mostly
broken in the parser part).

To all relevant (most) I have added North Carolina examples for easier
testing.

This set of modules I did not manage to fix:

* db.in.ogr
-> it does not, as programmed, delete the empty geometry and the cat column

* d.rast.leg
-> messy output, at least in wx0

* i.in.spotvgt
-> NameError: global name 'string' is not defined

* i.landsat.rgb blue=lsat7_2002_10 green=lsat7_2002_20 red=lsat7_2002_30
Processing <lsat7_2002_30>...
Color table for raster map <lsat7_2002_30> set to 'rules'
Processing <lsat7_2002_20>...
Color table for raster map <lsat7_2002_20> set to 'rules'
Processing <lsat7_2002_10>...
Color table for raster map <lsat7_2002_10> set to 'rules'
WARNING: Unable to write history for <lsat7_2002_30>. Raster map
         <lsat7_2002_30> not found in current mapset.
WARNING: Unable to write history for <lsat7_2002_20>. Raster map
         <lsat7_2002_20> not found in current mapset.
WARNING: Unable to write history for <lsat7_2002_10>. Raster map
         <lsat7_2002_10> not found in current mapset.

* i.spectral
-> no more errors but also no output

* r.mask
-> should show '[Raster MASK present]' as in GRASS 6 in the user prompt

* r.out.xyz (trying to export to elev_lid792_1m.csv)
-> File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/r.out.xyz",
line 42, in main
outf = file(output)
IOError: [Errno 2] No such file or directory: 'elev_lid792_1m.csv'

* GRASS 7.0.svn (latlong_wgs84@neteler):~ > v.in.wfs
url="http://mapserver.gdf-hannover.de/cgi-bin/grassuserwfs?REQUEST=GetFeature&SERVICE=WFS&VERSION=1.0.0"
out=grass_users
Retrieving data...
Importing data...
ERROR: Unable to open data source
</home/neteler/grassdata/latlong_wgs84/neteler/.tmp/north/18964.0.xml>
WFS import failed

* v.out.gps -t input=railroads output=trail.gpx
[nothing happens...]
^CTraceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 278, in <module>
    main()
Traceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 316, in <module>
    main()
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 207, in main
    p2.stdin.write(line)
KeyboardInterrupt

The rest in now ok, at least for the test cases added to the manual pages.

Markus

Hamish · September 4, 2011, 9:59pm

Markus N wrote:
...

This set of modules I did not manage to fix:

...

* i.landsat.rgb blue=lsat7_2002_10 green=lsat7_2002_20
red=lsat7_2002_30
Processing <lsat7_2002_30>...
Color table for raster map <lsat7_2002_30> set to
'rules'
Processing <lsat7_2002_20>...
Color table for raster map <lsat7_2002_20> set to
'rules'
Processing <lsat7_2002_10>...
Color table for raster map <lsat7_2002_10> set to
'rules'
WARNING: Unable to write history for <lsat7_2002_30>.
Raster map

<lsat7_2002_30> not found in current
mapset.
WARNING: Unable to write history for <lsat7_2002_20>.
Raster map

<lsat7_2002_20> not found in current
mapset.
WARNING: Unable to write history for <lsat7_2002_10>.
Raster map

<lsat7_2002_10> not found in current
mapset.

It tries to run grass.raster_history() without first testing
if the map is in the current mapset.

the shell script version has this test to avoid that trouble:

# write cmd history (only if raster maps are located in the current mapset)
CURRENT_MAPSET=`g.gisenv MAPSET`
...
BLUE_MAPSET=`g.findfile -n elem=cell file=$BLUE | grep mapset | cut -d'=' -f2`
...
if [ "$BLUE_MAPSET" = "$CURRENT_MAPSET" ] ; then
r.support ${BLUE} history="${CMDLINE}"
fi

probably we should continue this thread in bug reports.

thanks for the testing,
Hamish

Glynn_Clements1 · September 5, 2011, 2:19am

Hamish wrote:

It tries to run grass.raster_history() without first testing
if the map is in the current mapset.

Given that those maps are listed as inputs, it's debatable whether it
should be modifying the history even if those maps are in the current
mapset.

In any case, the corresponding fix is:

--- scripts/i.landsat.rgb/i.landsat.rgb.py (revision 48120)
+++ scripts/i.landsat.rgb/i.landsat.rgb.py (working copy)
@@ -124,8 +124,10 @@
set_colors(i, v0, v1)

# write cmd history:
+ mapset = grass.gisenv()['MAPSET']
for i in [red, green, blue]:
- grass.raster_history(i)
+ if grass.find_file(i)['mapset'] == mapset:
+ grass.raster_history(i)

if __name__ == "__main__":
options, flags = grass.parser()

--
Glynn Clements <glynn@gclements.plus.com>

Markus_GRASS · September 5, 2011, 6:13am

Glynn Clements wrote:

Hamish wrote:

It tries to run grass.raster_history() without first testing
if the map is in the current mapset.

Given that those maps are listed as inputs, it's debatable whether it
should be modifying the history even if those maps are in the current
mapset.

All that i.landsat.rgb does is creating color rules. They are stored
in /colr2 if the input maps are not in the current mapset. As r.colors
does not modify the raster history, and i.landsat.rgb is just a
front-end for r.colors, one could argue that i.landsta.rgb should not
touch raster history as well.

my2c

Markus M

In any case, the corresponding fix is:

--- scripts/i.landsat.rgb/i.landsat.rgb.py (revision 48120)
+++ scripts/i.landsat.rgb/i.landsat.rgb.py (working copy)
@@ -124,8 +124,10 @@
set_colors(i, v0, v1)
\# write cmd history:
+ mapset = grass.gisenv()['MAPSET']
for i in [red, green, blue]:
- grass.raster_history(i)
+ if grass.find_file(i)['mapset'] == mapset:
+ grass.raster_history(i)

if __name__ == "__main__":
options, flags = grass.parser()

--
Glynn Clements <glynn@gclements.plus.com>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Markus_GRASS · September 5, 2011, 7:27am

Markus Neteler wrote:

Hi,

I have tested almost all GRASS 7 scripts and fixed a lot of them (mostly
broken in the parser part).

To all relevant (most) I have added North Carolina examples for easier
testing.

This set of modules I did not manage to fix:

* db.in.ogr
-> it does not, as programmed, delete the empty geometry and the cat column

fixed

* d.rast.leg
-> messy output, at least in wx0

fixed by Michael Barton

* i.in.spotvgt
-> NameError: global name 'string' is not defined

fixed

* i.spectral
-> no more errors but also no output

worksforme

* r.out.xyz (trying to export to elev_lid792_1m.csv)
-> File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/r.out.xyz",
line 42, in main
outf = file(output)
IOError: [Errno 2] No such file or directory: 'elev_lid792_1m.csv'

fixed. BTW, r.out.xyz is nothing more than r.stats -1gn

Markus M

Markus_GRASS · September 5, 2011, 2:11pm

Markus Neteler wrote:

Hi,

I have tested almost all GRASS 7 scripts and fixed a lot of them (mostly
broken in the parser part).

To all relevant (most) I have added North Carolina examples for easier
testing.

This set of modules I did not manage to fix:

[snip]

* v.out.gps -t input=railroads output=trail.gpx
[nothing happens...]
^CTraceback (most recent call last):
File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 278, in <module>
main()
Traceback (most recent call last):
File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 316, in <module>
main()
File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 207, in main
p2.stdin.write(line)
KeyboardInterrupt

I think the piping from p1 to p2 is broken because the scripts tries
to sneak in between p1.stdout and p2.stdin and p2 is maybe getting
confused.

Considering that OGR supports writing out gpx or anything gpsbabel
supports, v.out.gps tries to do what v.out.ogr can already do? So why
not getting rid of v.out.gps? Removing a module for a change would be
nice;-)

Markus M

neteler · September 5, 2011, 7:57pm

On Mon, Sep 5, 2011 at 4:11 PM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:

* v.out.gps -t input=railroads output=trail.gpx

...

Considering that OGR supports writing out gpx or anything gpsbabel
supports, v.out.gps tries to do what v.out.ogr can already do? So why
not getting rid of v.out.gps? Removing a module for a change would be
nice;-)

Sounds very good to me. There also also some other leftover (?) shell
scripts there.

Still open:

On Sun, Sep 4, 2011 at 12:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

* r.mask
-> should show '[Raster MASK present]' as in GRASS 6 in the user prompt

In GRASS 6, there was lib/init/prompt.sh for this job. Currently one cannot know
from the command line if a MASK is there or not.

* GRASS 7.0.svn (latlong_wgs84@neteler):~ > v.in.wfs
url="http://mapserver.gdf-hannover.de/cgi-bin/grassuserwfs?REQUEST=GetFeature&SERVICE=WFS&VERSION=1.0.0"
out=grass_users
Retrieving data...
Importing data...
ERROR: Unable to open data source
</home/neteler/grassdata/latlong_wgs84/neteler/.tmp/north/18964.0.xml>
WFS import failed

One more:
* when leaving GRASS 7 with d.mon wx0 still open, the polling never finishes.

Markus

Hamish · September 5, 2011, 9:56pm

Markus Metz wrote:

Considering that OGR supports writing out gpx or anything
gpsbabel supports, v.out.gps tries to do what v.out.ogr can
already do? So why not getting rid of v.out.gps?

v.out.ogr can certainly _not_ export to anything that gpsbabel
supports. That's the entire point of v.out.gps. GPX is the
main overlap and thus why it's used as the in-between format
in the script: the script exports as GPX with v.out.ogr internally
and then gpsbabel converts from GPX to whatever.

http://www.gpsbabel.org/capabilities.html
http://gdal.org/ogr/ogr_formats.html

In fact v.out.gps roughly doubles the number of vector formats
GRASS can export to. Being able to load up your handheld GPSs
with wpts/trks/rtes directly from the GIS is a great and useful
feature to have.

keep it.

n.b. If Glynn's python port is working well, there's no reason
to keep the old shell script version of it in trunk any more.

thanks,
Hamish

ps- note I still need to port (aka rewrite) the v.in.gpsbabel
shell script to v.in.gps (python) in trunk. v.in.garmin can be
dropped (it has richer support for older Garmin hardware).
https://trac.osgeo.org/grass/changeset/45975

Glynn_Clements1 · September 6, 2011, 5:31am

Markus Metz wrote:

> I have tested almost all GRASS 7 scripts and fixed a lot of them (mostly
> broken in the parser part).
>
> To all relevant (most) I have added North Carolina examples for easier
> testing.
>
> This set of modules I did not manage to fix:
>
[snip]
>
> * v.out.gps -t input=railroads output=trail.gpx
> [nothing happens...]
> ^CTraceback (most recent call last):
> File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
> line 278, in <module>
> main()
> Traceback (most recent call last):
> File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
> line 316, in <module>
> main()
> File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
> line 207, in main
> p2.stdin.write(line)
> KeyboardInterrupt

I think the piping from p1 to p2 is broken because the scripts tries
to sneak in between p1.stdout and p2.stdin and p2 is maybe getting
confused.

It's m.proj which is broken:

p = grass.Popen(cmd, stdin = grass.PIPE, stdout = grass.PIPE)

cs2cs' stdout is redirected to a pipe, but the script never reads it.

It needs:

--- scripts/m.proj/m.proj.py (revision 48154)
+++ scripts/m.proj/m.proj.py (working copy)
@@ -235,7 +235,7 @@
# cs2cs | sed -e 's/d/:/g' -e "s/'/:/g" -e 's/"//g'

cmd = ['cs2cs'] + copyinp + outfmt + in_proj.split() + ['+to'] + out_proj.split()
- p = grass.Popen(cmd, stdin = grass.PIPE, stdout = grass.PIPE)
+ p = grass.Popen(cmd, stdin = grass.PIPE)

while True:
line = inf.readline()

--
Glynn Clements <glynn@gclements.plus.com>

Markus_GRASS · September 6, 2011, 6:11am

On Mon, Sep 5, 2011 at 11:56 PM, Hamish <hamish_b@yahoo.com> wrote:

Markus Metz wrote:

Considering that OGR supports writing out gpx or anything
gpsbabel supports, v.out.gps tries to do what v.out.ogr can
already do? So why not getting rid of v.out.gps?

v.out.ogr can certainly _not_ export to anything that gpsbabel
supports.

But see

http://gdal.org/ogr/drv_gpsbabel.html

Markus M

neteler · September 6, 2011, 7:24am

On Tue, Sep 6, 2011 at 7:31 AM, Glynn Clements <glynn@gclements.plus.com> wrote:

It's m.proj which is broken:

p = grass.Popen(cmd, stdin = grass.PIPE, stdout = grass.PIPE)

cs2cs' stdout is redirected to a pipe, but the script never reads it.

It needs:

--- scripts/m.proj/m.proj.py (revision 48154)
+++ scripts/m.proj/m.proj.py (working copy)
@@ -235,7 +235,7 @@
# cs2cs | sed -e 's/d/:/g' -e "s/'/:/g" -e 's/"//g'
cmd = \[&#39;cs2cs&#39;\] \+ copyinp \+ outfmt \+ in\_proj\.split \+ \[&#39;\+to&#39;\] \+ out\_proj\.split
- p = grass.Popen(cmd, stdin = grass.PIPE, stdout = grass.PIPE)
+ p = grass.Popen(cmd, stdin = grass.PIPE)
while True:
   line = inf\.readline

Fixe in r48160.
This leads now to:

v.out.gps -t input=railroads output=trail.gpx
Traceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 278, in <module>
    main()
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 260, in main
    for line in p.communicate()[0].splitlines():
AttributeError: 'NoneType' object has no attribute 'splitlines'
ERROR: Error reprojecting data

Markus

Hamish · September 6, 2011, 10:11pm

Hamish:

> v.out.ogr can certainly _not_ export to anything that
> gpsbabel supports.

Markus M:

But see
http://gdal.org/ogr/drv_gpsbabel.html

hmmm, a most interesting and welcome development.

none the less, wrt v.out.gps I'd still argue for a separate,
specialized, and easy to use front end (script) for interfacing
with field work GPSs. I was not sure for v.in.gps if we should
use the v.in.ogr GPX method directly or stay with the earlier
gpsbabel grass vector ascii format (grass_write_ascii.style)
definition + csv->db.in.ogr method. In the interest of strength
through diversity I'm now leaning towards the gpsbabel style-
file method. That makes reprojection to/from WGS84 slightly
easier than fun with ogr2ogr, and I'd suspect make the operation
a bit less lossy, but I suspect we'll only know the answer to the
lossiness question after trying both ways.

In the past the various v.in.garmin, v.in.gpsbabel, and
v.in.ogr(GPX)-only methods all ended with very different levels
of metadata/attribute loss, with the oldest of the 3 (v.in.garmin)
giving the least loss, in situations where the gps was an old
enough garmin for it to work.

Hamish

Glynn_Clements1 · September 8, 2011, 8:26am

Markus Neteler wrote:

> It's m.proj which is broken:

Fixe in r48160.
This leads now to:

line 260, in main
for line in p.communicate()[0].splitlines():
AttributeError: 'NoneType' object has no attribute 'splitlines'

Oops. I was looking for references to p.stdout, and missed that. The
actual problem was calling p.wait() before reading the output.

Try r48203.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · September 8, 2011, 8:00pm

On Thu, Sep 8, 2011 at 10:26 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:
...

Oops. I was looking for references to p.stdout, and missed that. The
actual problem was calling p.wait() before reading the output.

Try r48203.

Now it is back to deadlock (?)

v.out.gps -t input=railroads output=trail.gpx
^CTraceback (most recent call last):
Traceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 274, in <module>
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 316, in <module>
    main()
    main()
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/v.out.gps",
line 207, in main
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 245, in main
    p2.stdin.write(line)
    p.stdin.write(line)
KeyboardInterrupt
KeyboardInterrupt

Markus

Glynn_Clements1 · September 9, 2011, 10:32pm

Markus Neteler wrote:

> Oops. I was looking for references to p.stdout, and missed that. The
> actual problem was calling p.wait() before reading the output.
>
> Try r48203.

Now it is back to deadlock (?)

Try r48225.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · September 11, 2011, 2:08pm

On Sat, Sep 10, 2011 at 12:32 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

...

Try r48225.

Now a different one:

GRASS 7.0.svn (nc_spm_08@neteler):~ > v.out.gps -t input=railroads
output=trail.gpx
Traceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 287, in <module>
    main()
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 267, in main
    y, z = yz.split(' ')
ValueError: too many values to unpack

Markus

Glynn_Clements1 · September 12, 2011, 6:24pm

Markus Neteler wrote:

> Try r48225.

Now a different one:

line 267, in main
y, z = yz.split(' ')
ValueError: too many values to unpack

I've improved m.proj as much as I can.

The main problem is that, when cs2cs is run with the -E flag, it
appears to copy the input semi-verbatim, meaning that the output
format varies according to the input format. This makes it impossible
to reliably parse the output from cs2cs.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · September 12, 2011, 8:23pm

On Mon, Sep 12, 2011 at 8:24 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

> Try r48225.

Now a different one:

line 267, in main
y, z = yz.split(' ')
ValueError: too many values to unpack

I've improved m.proj as much as I can.

Thanks for your efforts!

Still...

v.out.gps -t input=railroads output=trail.gpx
Traceback (most recent call last):
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 288, in <module>
    main()
  File "/home/neteler/grass70/dist.x86_64-unknown-linux-gnu/scripts/m.proj",
line 267, in main
    x, y = xy.split('\t')
ValueError: need more than 1 value to unpack

The main problem is that, when cs2cs is run with the -E flag, it
appears to copy the input semi-verbatim, meaning that the output
format varies according to the input format. This makes it impossible
to reliably parse the output from cs2cs.

I believe that cs2cs should get the equivalent to the -g flag in GRASS which
makes the output truely machine-readable.

Perhaps we concentrate on more important issues than v.out.gps and
let it rest for now. Thanks, Glynn, I am sure that m.proj is way better now.

Markus

neteler · September 12, 2011, 8:25pm

On Mon, Sep 5, 2011 at 9:57 PM, Markus Neteler <neteler@osgeo.org> wrote:
...

Still open:

On Sun, Sep 4, 2011 at 12:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

* r.mask
-> should show '[Raster MASK present]' as in GRASS 6 in the user prompt

In GRASS 6, there was lib/init/prompt.sh for this job. Currently one cannot know
from the command line if a MASK is there or not.

This has been kindly fixed by Martin now.

* GRASS 7.0.svn (latlong_wgs84@neteler):~ > v.in.wfs
url="http://mapserver.gdf-hannover.de/cgi-bin/grassuserwfs?REQUEST=GetFeature&SERVICE=WFS&VERSION=1.0.0"
out=grass_users
Retrieving data...
Importing data...
ERROR: Unable to open data source
</home/neteler/grassdata/latlong_wgs84/neteler/.tmp/north/18964.0.xml>
WFS import failed

Yet open (but not important for me).

One more:
* when leaving GRASS 7 with d.mon wx0 still open, the polling never finishes.

d.mon will need some more work to make is especially faster. Perhaps
stream compression could be used? But I didn't study the internals (yet).

Markus

Hamish · September 12, 2011, 10:34pm

Markus wrote:

d.mon will need some more work to make is especially faster.
Perhaps stream compression could be used? But I didn't study
the internals (yet).

re. 'd.mon wx0' also it would be wonderful if you could turn off
the top menu bar* and the bottom status line to get 100% map real
estate. Functionality of those icons could go into a right-click
context menu with tick-mark menu entries to display them or not.
Just looking at how xfce4-terminal does it, besides their
Show _Menubar
menu item there's a
_Fullscreen
menu item too. ooh, lets one of those too please.

[*] I could rip the menu bar off into a floating window, but not
close it without taking down the rest of the window with it.

thanks a bunch,
Hamish