[GRASS-dev] 6.4.0 blocker bugs

GRASS GRASS developer
1

everyone wrote:

> >>> time to get out 6.4.0final :slight_smile:

MN:

> I learned that we should await the ctypes port to get
> rid of SWIG?

Glynn:

SWIG is only used within GRASS for the wxGUI vdigit and
nviz modules. It's also used to generate wrappers for
programmers who wish to access the libraries directly, but
these aren't used by any part of GRASS.

I suggest disabling all of this in the final release. The
vdigit and nviz modules don't work on Windows, and aren't
particularly robust on other platforms (and being loaded
in-process means that any problems affect the GUI as a whole,
not just the vdigit/nviz modules).

The SWIG wrappers for the libraries are barely usable and
are planned to be replaced, so we shouldn't be encouraging
their use.

IOW:

1. The "swig" directory should be removed from DIRS in the
top-level Makefile, so it isn't built (unless the user builds
it manually).

fyi I've just commented out the SUBDIRS+=python in the
grass 6.5 swig/Makefile, assuming ctypes will take its place
there. no objection to it happening in the root Makefile.

I agree that we shouldn't release 6.4.0 with swig enabled, as
it will only encourage folks to invest in a dead-end.

2. Official binaries shouldn't use --with-python; this will
prevent the vdigit and nviz modules from being built.

3. Optionally back-port the ctypes wrappers (lib/python/
ctypes). Even if this doesn't work (fails to build or
generates broken wrappers), it shouldn't break anything which
would otherwise work.

4. Optionally back-port the ctypes-based nviz module
(wxnviz.py). This has most of the same issues as nviz/vdigit
(i.e. the GRASS libraries are being accessed directly from the
GUI process, so a segfault or G_fatal_error() will kill the
GUI), but not all of them.

4b. Alternatively, back-port the changes but not wxnviz.py
itself. The result is equivalent to just building
--without-python (i.e. the GUI will try to import the wxnviz
module, fail, and warn that it's disabled), except that the
user can drop in wxnviz.py from SVN if they want to try it out.

I've no big opinion on if wxVdigit and wxNviz using swig should
remain enabled in 6.4.0 for now, or not. Works for me.

I suggest to continue to stabilize ctypes in 6.5svn and see where
it ends up. If it works cleanly there, backport to the 6.4 branch
for 6.4.1. (or if it's a quick job, for 6.4.0, but then as Martin
suggests we'd need one last RC. If it means a better end product
I don't mind that much)

Some folks on the list (eg Kim) report working Python interfaces,
I'm not sure if that has anything to with SWIG or just lib/python/
and init.* magic. Anyway, we should announce intentions once we
have some so they can adapt as needed. :slight_smile:

Python programming hooks are a big selling point these days (our
univ even offers a new course in geospatial programming using
python) so it would be a pity to remove that from the 6.4.0
release announcement, but not the end of the world if it will be
released mature in 6.4.1. Personally I'd say that the stuff
offered by lib/python/ and our solid collection of low-level C
modules let us claim support for python programmers with a
straight face, even if it isn't every single libgis C lib fn.

--

RC bugs according to the trac'er:
  https://trac.osgeo.org/grass/report/13

of those, weak-blockers IMO are-

seems solvable with slight effort AFAICT:
#1006 r.terraflow fails to stat() stream file on Windows
#994 v.buffer creating wrong buffer around polygon edges

I'm still meaning to spend a little time on this:
#1051 wxgui: SEARCH_PATH corruption

and verify that g.proj's memory bug is /really/ fixed:
https://trac.osgeo.org/grass/ticket/1032
https://trac.osgeo.org/grass/ticket/827
https://trac.osgeo.org/grass/ticket/820
https://trac.osgeo.org/grass/ticket/555

v.in.gpsbabel on WinGrass + a GPX file as in #555 is a quick test:

why does g.proj consistently fail in one particular script
with exit code 5, "ERROR_ACCESS_DENIED", and yet works fine
from the MSys command line and in other scripts?

could someone with MS-Windows please (re)test that?

--

Helena, fyi the FOSS4G 2010 Live osgeo demo DVD/usb stick will
ship with 6.4.0rc6; but MacOSX and MS Windows installers can be
updated at the last minute if newer versions become available.
The live version ships with the version UbuntuGIS has at build
time (which in turn is often derived from what Debian's Testing
has). We've got to prep, build, test, and send the master to
press a long time before the actual conference.

when it's ready,
Hamish

2

Hamish wrote:

[snip]

RC bugs according to the trac'er:
https://trac.osgeo.org/grass/report/13

The one blocker applies to Windows 7 only. My point was that for Linux
and IIUC also MAC 6.4 is stable. GRASS is not part of a Windows7
installation, but available through Linux repositories, e.g. yum
install grass should give you 6.4 without enabling particular
repositories, GDAL and PROJ4, also QGIS and SAGA are available through
standard repositories. Nothing but man power keeps us from making
available new wingrass installers on a regular basis.

of those, weak-blockers IMO are-

of those, the only blocker is #1020, and it applies apparently to Windows7 only.

I'm still meaning to spend a little time on this:
#1051 wxgui: SEARCH_PATH corruption

Great!

and verify that g.proj's memory bug is /really/ fixed:
https://trac.osgeo.org/grass/ticket/1032

A mystery because not reproducible.

https://trac.osgeo.org/grass/ticket/827

Windows only, out of reach for us, unless we make a plan on how to
tell libgdal to ignore any Windows/System32/*.dll

https://trac.osgeo.org/grass/ticket/820

Proposed fix for OSGEO4W: add msys bash because from within the
OSGEO4W msys (the one you get when starting grass with msys console),
there is no /bin/bash

https://trac.osgeo.org/grass/ticket/555

v.in.gpsbabel on WinGrass + a GPX file as in #555 is a quick test:

why does g.proj consistently fail in one particular script
with exit code 5, "ERROR_ACCESS_DENIED", and yet works fine
from the MSys command line and in other scripts?

could someone with MS-Windows please (re)test that?

Done so on XP.

when it's ready,

No blockers left for Linux, it's ready, IMHO.

Umh, we could wait a bit more to discover new blockers caused by e.g.
new incompatibilities between wxPython 2.8.11 and 2.8.12 or changes in
OSGEO4W...

Just nagging,

Markus M

3

Hi,

proposal for action plan:

- release 6.4.0 now as-is
- increment SVN release branch to 6.4.1
- start immediately to backport relevant patches into 6.4.1
    - winGRASS binaries will become available through josef
    - Linux snapshots as well
- Get out first 6.4.1RC1 by August
- Have final 6.4.1 ready in September for FOSS4G 2010

Markus

4

Hi again,

On Thu, Jul 8, 2010 at 11:09 AM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

proposal for action plan:

I am getting a bit tired of the "oh let's yet wait another weekend" emails...
Tonight in Europe could be a nice release preparation evening!

We need to complete the release material and write a press release.

As before I suggest:

- release 6.4.0 now as-is
- increment SVN release branch to 6.4.1
- start immediately to backport relevant patches into 6.4.1
- winGRASS binaries will become available through josef
- Linux snapshots as well
- Get out first 6.4.1RC1 by August
- Have final 6.4.1 ready in September for FOSS4G 2010

Markus

5

Markus:

I am getting a bit tired of the "oh let's yet wait another
weekend" emails...
Tonight in Europe could be a nice release preparation evening!

updated wingrass binaries which start to use 'make distclean' have only
been posted in the last _hour_. (and so r.terraflow fix may be tested)
Testing the 6.4 binary will have to wait until tomorrow's job to be
included as only just backported.

someone posted a failure of v.db.addcol(?) on wingrass in the last days,
we should check on that too.

We need to complete the release material and write a press release.

I will work on the stuff in grass-web/announce/ today. (task #677)
any help with updating the twiki side of things would be appreciated.

regards,
Hamish

ps - anyone? #994 v.buffer creating wrong buffer around polygon edges.

https://trac.osgeo.org/grass/attachment/ticket/994/buffer_bug.png
https://trac.osgeo.org/grass/ticket/994#comment:2

it's fails to join up closed polygons so it does not include the
contribution from the buffer around the node where the area is closed
(the green "x" in v.digit).

bad results in a core module, seems not too deep to fix..

6

- start immediately to backport relevant patches into 6.4.1

just a logistics note for all, to help with that please don't start bulk
moving random tickets from 6.4.0->6.4.1 in trac until all tickets
currently marked for 6.4.1 are dealt with (mostly backports waiting for
the branch). otherwise that stuff which should be done asap gets lost in
the noise.

thanks,
Hamish

7

sympathies about requesting yet more server transition work, but fwiw we
should really get the rsync mirrors back up and running before announcing a
new release.

http://grass.ibiblio.org et. al are currently frozen at July 13 when the
xblade went off-line.

happy to scrape out a little time to help with that as needed,
Hamish

ps- I notice the weekly translation stats page is broken at the ibiblio
mirror (& has been for a while).
?

8

On Sun, Jul 18, 2010 at 11:32 AM, Hamish <hamish_b@yahoo.com> wrote:

sympathies about requesting yet more server transition work, but fwiw we
should really get the rsync mirrors back up and running before announcing a
new release.

I am working on all this for many hours :slight_smile: please stay tuned...

http://grass.ibiblio.org et. al are currently frozen at July 13 when the
xblade went off-line.

of course. Will change asap.

happy to scrape out a little time to help with that as needed,
Hamish

ps- I notice the weekly translation stats page is broken at the ibiblio
mirror (& has been for a while).
?

will check that once the other mess is done.

more soon,
Markus

9

On Tue, Jul 6, 2010 at 10:07 PM, Markus Metz
<markus.metz.giswork@googlemail.com> wrote:

Hamish wrote:

[snip]

RC bugs according to the trac'er:
https://trac.osgeo.org/grass/report/13

The one blocker applies to Windows 7 only. My point was that for Linux
and IIUC also MAC 6.4 is stable. GRASS is not part of a Windows7
installation, but available through Linux repositories, e.g. yum
install grass should give you 6.4 without enabling particular
repositories, GDAL and PROJ4, also QGIS and SAGA are available through
standard repositories. Nothing but man power keeps us from making
available new wingrass installers on a regular basis.

the two (!) remaining blocker bugs are:

* https://trac.osgeo.org/grass/ticket/1115
   -> Mac OSX only? Seems to be a wxpython 2.8.10.1 problem (wasn't there
     something similar recently in the new i.points?)

* https://trac.osgeo.org/grass/ticket/1020
  -> Windows. AFAIK solved and spammed with another problem.

...

I'm still meaning to spend a little time on this:
#1051 wxgui: SEARCH_PATH corruption

Great!

Please do or postpone - we are holding 9000 or so changes
after the last stable release. And 6.4.1 will come out way
faster.

and verify that g.proj's memory bug is /really/ fixed:
https://trac.osgeo.org/grass/ticket/1032

A mystery because not reproducible.

-> was closed.

https://trac.osgeo.org/grass/ticket/827

Windows only, out of reach for us, unless we make a plan on how to
tell libgdal to ignore any Windows/System32/*.dll

-> workaround in the ticket

https://trac.osgeo.org/grass/ticket/820

Proposed fix for OSGEO4W: add msys bash because from within the
OSGEO4W msys (the one you get when starting grass with msys console),
there is no /bin/bash

-> r.in.wms should not hold a release.

https://trac.osgeo.org/grass/ticket/555
v.in.gpsbabel on WinGrass + a GPX file as in #555 is a quick test:

-> closed, too.

No blockers left for Linux, it's ready, IMHO.

Good. So only two blockers left (see above).

Markus

10

Markus Neteler wrote:

the two (!) remaining blocker bugs are:

* https://trac.osgeo.org/grass/ticket/1115
-> Mac OSX only? Seems to be a wxpython 2.8.10.1 problem (wasn't there
something similar recently in the new i.points?)

Not related to the problem the new i.points had with wxpython 2.8.7,
unfortunately, otherwise I could have had an idea. I looked into #1115
but have no idea why it fails, same wx version but different python
version, so it's apparently not wx but python.

* https://trac.osgeo.org/grass/ticket/1020
-> Windows. AFAIK solved and spammed with another problem.

Apparently fixed. Is there a new ticket for the problem mentioned in
comment::25?

Markus M

11

On Jul 26, 2010, at 2:39 PM, Markus Metz wrote:

Markus Neteler wrote:

the two (!) remaining blocker bugs are:

* https://trac.osgeo.org/grass/ticket/1115
  -> Mac OSX only? Seems to be a wxpython 2.8.10.1 problem (wasn't there
    something similar recently in the new i.points?)

Not related to the problem the new i.points had with wxpython 2.8.7,
unfortunately, otherwise I could have had an idea. I looked into #1115
but have no idea why it fails, same wx version but different python
version, so it's apparently not wx but python.

Cleared up, bad wx build. Sorry for the holdup.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

First Pogril: Why is life like sticking your head in a bucket filled with hyena offal?
Second Pogril: I don't know. Why IS life like sticking your head in a bucket filled with hyena offal?
First Pogril: I don't know either. Wretched, isn't it?

-HitchHiker's Guide to the Galaxy

12

Markus Neteler wrote:

-> r.in.wms should not hold a release.

fwiw, r.in.wms and (presumably) i.oif on WinGrass do not work right now
because they use helper shell scripts installed to $GISBASE/etc/$module/
and those shell scripts do not currently have associated .bat files to
launch them so they aren't executable and so not found in the %PATH%.
should be an easy fix to generate those batch files in the two modules'
Makefiles, but so far it remains TODO.

earlier problems with g.* + r.in.wms seem to be unrelated and fixed now.

regards,
Hamish

ps- everyone please review:
https://trac.osgeo.org/grass/browser/grass-web/trunk/announces/abstract_grass640.txt
https://trac.osgeo.org/grass/browser/grass-web/trunk/announces/announce_grass640.html

13

Hamish wrote:

Markus Neteler wrote:
> -> r.in.wms should not hold a release.

fwiw, r.in.wms and (presumably) i.oif on WinGrass do not work right now
because they use helper shell scripts installed to $GISBASE/etc/$module/
and those shell scripts do not currently have associated .bat files to
launch them so they aren't executable and so not found in the %PATH%.
should be an easy fix to generate those batch files in the two modules'
Makefiles, but so far it remains TODO.

I'm not sure that they should need .bat files, given that they are
only run from within a shell scripts. I thought that MSys could handle
this itself.

Does it work if the main scripts are changed to run helper scripts via
an absolute path?

--
Glynn Clements <glynn@gclements.plus.com>

14

> Markus Neteler wrote:
> > -> r.in.wms should not hold a release.

Hamish:

> fwiw, r.in.wms and (presumably) i.oif on WinGrass do not work right now
> because they use helper shell scripts installed to
> $GISBASE/etc/$module/ and those shell scripts do not currently have
> associated .bat files

Glynn:

I'm not sure that they should need .bat files, given that they are
only run from within a shell scripts. I thought that MSys could handle
this itself.
Does it work if the main scripts are changed to run helper scripts via
an absolute path?

You are correct, by adding the full path to the helper script it can
now find them. fix applied in 6.5svn.
i.oif already had that.

next problem: r.in.wms calls wms.request which calls r.tileset.
r.tileset calls cs2cs*, with +proj4 terms taken from SRS=`g.proj -j`.

for spearfish those terms include the conus grid file, which is in
$GISBASE/etc/nad/ ... and $GISBASE of course contains a space, and
then cs2cs dies a horrible death.

so how to quote the right side of the individual +nadgrids= line? and
will cs2cs accept that?

I expect it may be a bad idea to ask g.proj to output that, but maybe
that's not so evil after all. ?? perhaps just do that for MINGW32?
I guess that would mean putting ugliness into g.proj's print_proj4().

[*] so m.proj and r.in.gdalwarp probably also has this problem

I'm also a bit concerned about what happens when it hits r.tileset's
bashisms alittle later in, but we'll jump off that bridge when we get to
it.

Hamish

15

Hamish wrote:

next problem: r.in.wms calls wms.request which calls r.tileset.
r.tileset calls cs2cs*, with +proj4 terms taken from SRS=`g.proj -j`.

for spearfish those terms include the conus grid file, which is in
$GISBASE/etc/nad/ ... and $GISBASE of course contains a space, and
then cs2cs dies a horrible death.

so how to quote the right side of the individual +nadgrids= line? and
will cs2cs accept that?

You need to construct the cs2cs command line "the hard way"; simply
inserting the "g.proj -j" output won't work.

Something like:

  opts=`g.proj -j | (
      opts=
      while read line ; do
          opts="$opts '$line'"
      done
      echo "$opts"
  )`

BTW: the Python versions of m.proj and r.in.aster will have problems
with this, as they use "g.proj -jf ...".

--
Glynn Clements <glynn@gclements.plus.com>

16

Hamish wrote:
> next problem: r.in.wms calls wms.request which calls r.tileset.
> r.tileset calls cs2cs*, with +proj4 terms taken from SRS=`g.proj -j`.
>
> for spearfish those terms include the conus grid file, which is in
> $GISBASE/etc/nad/ ... and $GISBASE of course contains a space, and
> then cs2cs dies a horrible death.
>
> so how to quote the right side of the individual +nadgrids= line? and
> will cs2cs accept that?

Glynn wrote:

You need to construct the cs2cs command line "the hard way"; simply
inserting the "g.proj -j" output won't work.

Something like:

    opts=`g.proj -j | (
        opts=
        while read line ; do
            opts="$opts '$line'"
        done
        echo "$opts"
    )`

thanks, applied in 6.5svn in r.in.wms/wms.request and r.tileset.

... still no-go. at the 6.4.0svn msys prompt:

GRASS 6.4> g.proj -j
+proj=utm
+zone=13
+a=6378206.4
+rf=294.9786982
+no_defs
+nadgrids=c:/Program
Files/GRASS-64-SVN/etc/nad/conus
+to_meter=1.0

.. so g.proj is broken ..

g.proj/output.c print_proj4():

    for (i = proj4mod; *i; i++) {
        /* Don't print the first space */
        if (i == proj4mod && *i == ' ')
            continue;

! --> if (*i == ' ' && !(dontprettify))
            fputc('\n', stdout);
        else
            fputc(*i, stdout);
    }
    fputc('\n', stdout);

BTW: the Python versions of m.proj and r.in.aster will have problems
with this, as they use "g.proj -jf ...".

ok

Hamish

17

On Thu, 29 Jul 2010, Hamish wrote:

... still no-go. at the 6.4.0svn msys prompt:

GRASS 6.4> g.proj -j
+proj=utm
+zone=13
+a=6378206.4
+rf=294.9786982
+no_defs
+nadgrids=c:/Program
Files/GRASS-64-SVN/etc/nad/conus
+to_meter=1.0

.. so g.proj is broken ..

Oops - well I've just committed a change to 6.5 in r42943 that should only split at a space character if there is a + character following it - hopefully that works?

Paul

18

Paul Kelly wrote:

> ... still no-go. at the 6.4.0svn msys prompt:
>
> GRASS 6.4> g.proj -j
> +proj=utm
> +zone=13
> +a=6378206.4
> +rf=294.9786982
> +no_defs
> +nadgrids=c:/Program
> Files/GRASS-64-SVN/etc/nad/conus
> +to_meter=1.0
>
>
> .. so g.proj is broken ..

Oops - well I've just committed a change to 6.5 in r42943 that should only
split at a space character if there is a + character following it -
hopefully that works?

Ah; so pj_get_def() returns a definition using spaces both as an
argument separator and within arguments?

In which case, we need a more robust alternative to pj_get_def().

--
Glynn Clements <glynn@gclements.plus.com>

19

Glynn:

Ah; so pj_get_def() returns a definition using spaces both
as an argument separator and within arguments?

it seems so.

with the help of y'alls fixes all should be working in 6.5svn now
except for r.in.aster.

cs2cs accepts individual argv's, but r.in.aster uses gdalwarp
which takes all +proj terms in a single (quoted) argv, and if
you use multiple 'single quotes' for each term within the -t_srs
"outer quote", gdalwarp chokes with a parse error. So I think
that's a bug in gdalwarp as it seems impossible to quote
+nadgrids there.

In which case, we need a more robust alternative to
pj_get_def().

only if you can't live with Paul's fix, otherwise I think we're
ok. Are the NTv2 grid files likely to ever be installed to a
dir with a " +" in the name? famous last words, but it seems
unlikely to me. or at least defer worrying about it until we
get an actual bug report.

Hamish

20

On Fri, 30 Jul 2010, Hamish wrote:

Glynn:

Ah; so pj_get_def() returns a definition using spaces both
as an argument separator and within arguments?

In which case, we need a more robust alternative to
pj_get_def().

I guess the alternative is to remove the step whereby (in g.proj) the set of PROJ parameters that has been constructed by pj_get_kv() (a GRASS function, despite the pj_ prefix) is passed to pj_get_def() (a PROJ.4 function) for conversion into a string. Instead, part of pj_get_kv() could be separated out into a separate function that GRASS modules could use to get access to the PROJ parameters as an array of separate parameters rather than as a concatentated string. That would be a little bit of work but not too hard to do.

only if you can't live with Paul's fix, otherwise I think we're
ok. Are the NTv2 grid files likely to ever be installed to a
dir with a " +" in the name? famous last words, but it seems
unlikely to me. or at least defer worrying about it until we
get an actual bug report.

Interesting sidenote here is that it is (AFAIK) an undocumented feature of PROJ.4 that it accepts a full pathname as the value of the +nadgrids parameter. Years ago when we were putting the datum support into GRASS, the recommended method from Frank was just to put an unqualified filename as the value of +nadgrids, and then call the pj_set_finder() function to specify a "finder function" that PROJ.4 can call back to, to find the directory where the gridfiles are stored. But this approach falls down when exporting a PROJ.4 string for use by another application, as there is no way of telling that application where the gridfiles are stored. So we had to change it to use a fully qualified path to the nadgrids file - which has now thrown up this problem.

Paul