[GRASS-dev] grass 6.x g.extension install path layout simplification

Hi,

3 questions for your consideration--

For grass 6.x g.extension.sh installs into GRASS_ADDON_PATH, are there
any objections to installing the addon man pages to e.g.
~/.grass6/addons/docs/man/ instead of ~/.grass6/addons/man/ ? I don't
think anything is depending on that directory layout structure, and it
is an opportunity to make the addons base dir a little cleaner. The html docs
stay in the same place; I don't see anything in wxPy that cares about man
pages; and nothing which references $MANPATH. (although we could have
g.manual try $GRASS_ADDON_PATH/man/man1/ if we agree to use that)

system-wide installations (as root) into the real $GISBASE would
still go to their customary destination.

Also in 6.x's GRASS_ADDON_PATH, as discussed earlier I'd like to get rid
of the symlinks and move the executables from bin/ and script/ into the
main $PATH'd dir. In devbr6 r49343 I've committed some (deactivated) code to
show how that would be done. comments? criticisms!?

system-wide installations (as root) into the real $GISBASE would
still go to their customary destinations.

Finally, g.extension.sh's central `make` command also adds
-I$TMPDIR/$DIST_DIR/include and -L$TMPDIR/$DIST_DIR/lib. Are those
actually needed by the build? I see the addon module(s) i.pr has its
own include/, so maybe it is. r.rast4d has its own lib/ but that just
contains some shell scripts.
Keeping the extra search paths does no harm, just adds a bit of
clutter.

If there are no objections for the man page moving and symlink replacement
I'd like to backport that before 6.4.2rc3, aka ASAP.

thanks,
Hamish

Hi,

2011/11/24 Hamish <hamish_b@yahoo.com>:

For grass 6.x g.extension.sh installs into GRASS_ADDON_PATH, are there
any objections to installing the addon man pages to e.g.
~/.grass6/addons/docs/man/ instead of ~/.grass6/addons/man/ ? I don't

I would keep manual pages where they are. At least the current
directory layout is synchronized with $GISBASE. Moving manual pages
somewhere else makes things less clear from my POV. Same with creating
extra symlinks for the binaries which eg. on Windows means to create a
copy of the file.

In GRASS7 we could move in $GISBASE `man` do `docs` directory. Don't
spend with GRASS 6 too much energy with minimal improvement.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

Hamish wrote:

If there are no objections for the man page moving and symlink replacement
I'd like to backport that before 6.4.2rc3, aka ASAP.

The add-ons directory should follow exactly the same structure as the
main $GISBASE directory.

Ultimately, I'd like to see $GISBASE replaced by a path, so that all
"GRASS" files are located via the path. That will only work if the
relative filename is independent of whether the actual file resides in
the main installation or the add-ons directory.

Finally, g.extension.sh's central `make` command also adds
-I$TMPDIR/$DIST_DIR/include and -L$TMPDIR/$DIST_DIR/lib. Are those
actually needed by the build?

In 7.0, add-ons should be built with $(ARCH_DISTDIR) referring to the
staging directory. Any libraries/headers generated during the build
will end up there, and any subsequent libraries and modules will find
them there.

But really, there shouldn't be any need for g.extension in 7.0. It
should be sufficient to simply run e.g.:

  make MODULE_TOPDIR=$GISBASE ARCH_DISTDIR=$HOME/.grass7/addons

--
Glynn Clements <glynn@gclements.plus.com>

Hamish:

> For grass 6.x g.extension.sh installs into
> GRASS_ADDON_PATH, are there any objections to
> installing the addon man pages to e.g.
> ~/.grass6/addons/docs/man/ instead of
> ~/.grass6/addons/man/ ?

Martin:

I would keep manual pages where they are. At
least the current directory layout is
synchronized with $GISBASE.

but there is no dependency that we try to do that.
basically this is finishing the job of installing
the addon module from the build dir. I hate to
repeat this, but GRASS_ADDON_PATH was never
designed to be a shadow GISBASE, and there is no
reason or great benefit for it to try to become one
now.

Moving manual pages somewhere else makes things
less clear from my POV.

moving manuals into docs/ is not clear? (from the
end-user's POV) keeping them outside seems weird
and messy to me, and here we have a short window of
opportunity to fix that, or at least to not
propagate the weirdness.

Same with creating extra symlinks for the
binaries which eg. on Windows means to create a
copy of the file.

er, in the original post, the paragraph directly
following the one you quoted above explained the
plan to get rid of those; and (deactivated) code in
devbr6 is ready to do that, just waiting for review.

bin/ and script/ would go away resulting in a dir
structure looking like:

GRASS_ADDON_PATH /
                 / all executables
                 / docs
                       / html
                       / man/man1

very simple and clean for the end-user.
we get to get rid of the cruft and preserve
historical addon usage at the same time.

In GRASS7 we could move in $GISBASE `man` do
`docs` directory.

I don't mind that, but it's another matter.

Don't spend with GRASS 6 too much energy with
minimal improvement.

Since 6.4.2 will be the first release with a
functional g.extension for end users without the
source code installed (ie most), I think it is
important that we nail down the design decisions
before release, rather than be left with something
suboptimal for the remainder of the 6.4.x usage
lifetime.

best,
Hamish

Glynn wrote:

The add-ons directory should follow exactly the
same structure as the main $GISBASE directory.

perhaps for grass7, but 10 years worth of backwards
compatibility with the implementations in grass5 and
grass6 says otherwise for those.

Hamish

Hamish wrote:

perhaps for grass7, but 10 years worth of backwards
compatibility with the implementations in grass5 and
grass6 says otherwise for those.

I think perhaps I should say something about why GRASS_ADDON_PATH
came to be. This isn't THE history, but it is a history; you
were there & I don't mean to be condescending or anything, just
to explain where I'm coming from since we seem to be looking
at this from different perspectives.

[If you are too busy to read this little story, skip down to
the "Back to the issue at hand" paragraph below.]

once upon a time if you wanted to compile a non-standard module
obtained from elsewhere or self-written you had to copy it into
the source tree (e.g. into raster/r.module/), or symlink it
there, and compile with `gmake5`. it then (optionally) got
installed into your GISBASE. and all was good until you did a
'make clean'. worse, if you installed a new version of GRASS and
uninstalled the old one, you lost access to your self-compiled
module, so it was nice to copy it somewhere outside of the
install dir which got replaced every few months.
so far, so obvious, and fwiw I maintain a `gmake6` alias which
runs `make MODULE_TOPDIR=...` which is generally what I use
instead of g.extension, but then again I've got the source code
installed which is a special case.

--it's probably worth noting here that if the GRASS ABI changed
you had to recompile, but the same is true for things compiled
with g.extension today, although luckily we have the GIS_H
version test these days.

At the same time, and I'd expect much more common but _invisible
to developers_, people were writing their own scripts to do their
own little jobs connecting the building blocks "do one thing
well" modules together in a similar fashion to the GUI graphical
modeler. For example, my ADDON_PATH is littered with dozens of
these work related scripts (reusable macros if you prefer) and a
few personal preference shortcuts (e.g. see d.stations in addons
svn; it's not really a module, just a glorified alias).

mainly these are the cache of scripts to run some set of analyses
or generate some reusable plot template. they are not modules,
but are the personal repository of workflow scripts. That is
not to say there are no personal modules mixed in there, far
from it, only to say that there are both modules and user-scripts
to consider and to accommodate.

anyway, back to the story:
So eventually with grass6 we abandoned contrib/ and the list
of external addons maintained on the website, and started the
grass wiki and its list of addon modules. And then as some of
those external websites started to go 404, about the same time
as we moved to trac svn, and the addons svn was born as a more
permanent home. Finally some of the things in the addons svn
matured and become rather interesting to others, and non-
developer users had some trouble compiling them, and so
Makefiles started to pop up in the addon module dirs,
g.extension came to be, a bit later Makefiles were added to
compile collections of addons, and most recently the osgeo4w
addon collection. (as `make` is not happy to run due to spaces
in path names, and most people will not install to
C:\GRASS-6.4.1\) With about 20% of all public GRASS modules
being addons, and easier access to them, progress has been good
and we seem to be on the right track!

A small digression:
With automatic compiles of everything with a Makefile in addons
svn there is the conceptual conflict in addons being a module
incubator, or is it a home for finished, release quality modules?
A svn sandbox had been added, but I'm still not fully convinced
that addons svn should be abandoned as a place for still-in-
development works, and what the role of the sandbox should be.
Mainly I think where things grow organically is a better guide
than any planned rule, so I'll wait and see where that goes with
an open mind.

Back to the issue at hand:

I get the point about addon modules in theory being drop-in
additions to the normal source tree, and so experiencing the
same fate and expectations (in a relative dir sense) as an
official module, but my main point here is that GRASS_ADDON_PATH
over the years has been as much, if not more so, useful as a
place to put your end-user scripts, not "modules" at all. And
GISBASE, or a GISBASE-alike, makes no sense in the context of
a script which is not a module.

In practice mine has held both self-built modules and non-module
grass user-scripts to varying degrees, what I object to is
imposing refugee status on the personal work-scripts at this
late stage in a stable 6.x release series by redefining
GRASS_ADDON_PATH into a GRASS_ADDON_BASE.

For backwards compatibility we must keep GRASS_ADDON_PATH in
grass6 in the $PATH, and so this is just silly, for no gain:

PATH="$PATH:$GRASS_ADDON_PATH:$GRASS_ADDON_PATH/bin:$GRASS_ADDON_PATH/scripts"

vs

PATH="$PATH:$GRASS_ADDON_PATH"

symlinks are one way to do that, but not ideal for Windows.
(but g.extension doesn't work there anyway, so I wonder if
in practice that actually matters?) Anyway simply combining
$ADDONS/bin/ and $ADDONS/scripts/ avoids the symlinks and
makes avoids the unnecessary dir structure ta boot.

If i.pr or some advanced addon wants some other part of the
GISBASE structure, still no problem, it still gets installed.
Since g.manual(.sh) and the guis know nothing of GRASS_ADDON_PATH
(yet), there is no backwards compatibility to break if we move
$ADDONS_6/man/man1 into docs/. No code is depending on that, and
it isn't making the man pages difficult to discover when
exploring by hand.

Ah well, enough rambling..

gobble gobble,
Hamish