[GRASS5] GRASS extension manager proposal

GRASS GIS GRASS GIS developer
1

Hi everyone,

as I am making good progress porting my belief modelling tools
to 5.7, I realized they would make an excellent example for
a new 5.7 extension manager.
My tools consist of quite a few separate modules plus helper libs
and instead of merging all of that into the already quite full
"raster", "vector" etc. dirs, I would like to package them
as an external extension which everyone interested could download,
compile and install from inside their GRASS installations,
without having to merge the code into the source tree first.
I would then try and code a little user frontend program 'g.install' which
would take care of all the installation issues.

Michael has told me that some of you have already spent some
thought on it so could you fill me in on how far you got?

My ideas are:

For each extension:
- put all source code and binaries for selected systems into
  a tarball
- also add some standardised ASCII description files to the
  tarball:
  - version information
  - binary and dependencies information
  - source URL and author info
  - Tcl/Tk menu code for the GIS manager

For GRASS 5.7:
- the g.install module could be used to
  list, install, update and remove extension
        binaries and documentation

- installation could be system-wide into any GRASS bin tree location
  the user has write access to or user-only into a directory
  into a dir under the user's current location or mapset.

- a new menu entry 'Extensions' in the GIS manager where extensions
  can register their modules in a submenu and hooks for g.install
  to query, update or remove these extensions are provided
  
For a future perspective, I think the R approach of putting all
functionality into packages would work quite well.
R includes the most important packages in the main distro and
has a simple install tool for users to add whatever they need.
GRASS already has very many modules and browsing the documentation
list to find the right module has always been
a bit of a challenge, since their is nothing except the program
name to hint at a module's purpose.
So, I would think it a useful thing to group modules by functionality
such as 'base', 'spatial statistics', 'imagery',
which would also be reflected in the source tree organisation and
the HTML documentation structure.

What do you think?

Cheers,

Benjamin

2

as I am making good progress porting my belief modelling tools
to 5.7,

great

What do you think?

why bother with g.install and just put it all in GRASS CVS?

Hamish

3

On Thu, 16 Dec 2004, Hamish wrote:

> as I am making good progress porting my belief modelling tools
> to 5.7,

great

Do you (Benjamin) have any reports beyond the useful comments you've
already given about interface writing using the 5.7 API?

> What do you think?

why bother with g.install and just put it all in GRASS CVS?

Because, as Benjamin and others (especially Paolo Cavallini) have
suggested, a development model with add-in modules is vastly more robust
that one in which everything is in the same tarball. Central servicies
need to be in the core distribution, but everything else should be
modularised. They can be stored on a central CVS repository, but do not
need to be in the same project, just to follow the same installation
rules.

In particular, it isn't obvious that the original choices made in GRASS to
carry around large amounts of model interface, and also unmaintained model
code, were sensible. If the models are maintained standalone anyway, the
interface can be built and installed by people who need the functionality
and have both GRASS and the model.

How to do a clean g.install isn't so clear, though - exposing the
dependencies of the local GRASS installation, its headers and libraries,
and to do it portably across OSX, Unix/Linux, and Cygwin. A lot of the R
CMD * family are stubs, scripts, and Perl programs, and that feels
flexible and robust.

Roger

Hamish

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

--
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
e-mail: Roger.Bivand@nhh.no

4

Roger Bivand wrote:

why bother with g.install and just put it all in GRASS CVS?

Because, as Benjamin and others (especially Paolo Cavallini) have
suggested, a development model with add-in modules is vastly more robust
that one in which everything is in the same tarball. Central servicies
need to be in the core distribution, but everything else should be
modularised. They can be stored on a central CVS repository, but do not
need to be in the same project, just to follow the same installation
rules.

I agree. I prefer an archive where everybody can add packages. It is not realistic, to create CVS account for every student who writes a script.

This is certainly good starting point:
http://www.public.asu.edu/~cmbarton/grass_scripts.htm

Radim

5

+1. This is exactly the kind of architecture that will take GRASS to
the next level. Very exciting to see!

On Thu, 2004-12-16 at 03:35, Roger Bivand wrote:

On Thu, 16 Dec 2004, Hamish wrote:
Because, as Benjamin and others (especially Paolo Cavallini) have
suggested, a development model with add-in modules is vastly more robust
that one in which everything is in the same tarball. Central servicies
need to be in the core distribution, but everything else should be
modularised. They can be stored on a central CVS repository, but do not
need to be in the same project, just to follow the same installation
rules.

In particular, it isn't obvious that the original choices made in GRASS to
carry around large amounts of model interface, and also unmaintained model
code, were sensible. If the models are maintained standalone anyway, the
interface can be built and installed by people who need the functionality
and have both GRASS and the model.

How to do a clean g.install isn't so clear, though - exposing the
dependencies of the local GRASS installation, its headers and libraries,
and to do it portably across OSX, Unix/Linux, and Cygwin. A lot of the R
CMD * family are stubs, scripts, and Perl programs, and that feels
flexible and robust.

Roger

>
>
> Hamish
>
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
>

6

I want to echo the positive other replies to this proposal. This is very
much along the lines of what Paolo, I, and others have suggested. It would
be good if the 'g.install' routine could also insert shell scripts into the
extension menu hierarchy.

Another longer-term idea (i.e., for the 6.x series) I tossed into the fray a
few months back is to make modules self-organizing in the GUI unless
overridden by a users. We already have a start in how that could be achieved
by the letter-dot prefix of GRASS modules. r.* modules are raster, and so
on. We could build on that so that each module and script written would have
a short header string that identified where it belonged in the menu
hierarchy. Then, when someone installed a module, it would go to a logical
local rather than into a long extension menu.

The ultimate objective is to make a GUI that is more or less
self-generating. If I remember correctly, some time back, Radim was working
on a QT version of the GUI that would use XML to manage the menu hierarchy
with this kind of goal in mind. In such a case, a self-generating menu
system (or alternative better GUI) would be considerably more feasible.

Michael

On 12/15/04 3:49 AM, "Benjamin Ducke" <benducke@compuserve.de> wrote:

My ideas are:

For each extension:
- put all source code and binaries for selected systems into
  a tarball
- also add some standardised ASCII description files to the
  tarball:
- version information
- binary and dependencies information
- source URL and author info
- Tcl/Tk menu code for the GIS manager

For GRASS 5.7:
- the g.install module could be used to
list, install, update and remove extension
        binaries and documentation

- installation could be system-wide into any GRASS bin tree location
  the user has write access to or user-only into a directory
  into a dir under the user's current location or mapset.

- a new menu entry 'Extensions' in the GIS manager where extensions
  can register their modules in a submenu and hooks for g.install
  to query, update or remove these extensions are provided

______________________________
Michael Barton, Professor of Anthropology
School of Human, Evolution and Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262; fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

7

On 12/16/04 2:11 AM, "Radim Blazek" <blazek@itc.it> wrote:

Roger Bivand wrote:

why bother with g.install and just put it all in GRASS CVS?

Because, as Benjamin and others (especially Paolo Cavallini) have
suggested, a development model with add-in modules is vastly more robust
that one in which everything is in the same tarball. Central servicies
need to be in the core distribution, but everything else should be
modularised. They can be stored on a central CVS repository, but do not
need to be in the same project, just to follow the same installation
rules.

I agree. I prefer an archive where everybody can add packages. It is not
realistic, to create CVS account for every student who writes a script.

This is certainly good starting point:
http://www.public.asu.edu/~cmbarton/grass_scripts.htm

If this kind of repository ever catches on, we might want to think about
moving it to the Wiki server. I mentioned this to Markus and he was
favorable.

The problem with the ASU server is that all submissions currently need to be
sent to me personally (i.e., via email or my office Mac) and then put on the
ftp/html download server by me manually. I don't mind managing or helping to
manage a GRASS applications repository, but using the ASU server for this is
a pretty klunky way to do it at present. It would be better to have a
submissions area accessible via some kind of broad but controled access (the
new Wiki server seems good at this). New submissions could wait there for
testing. After testing, they could then be moved into a subdirectory
hierarchy that matched their function and place in the GUI hierarchy.

Michael
______________________________
Michael Barton, Professor of Anthropology
School of Human, Evolution and Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262; fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

8

OK, it seems everybody agrees that it is a good idea.
I will try and write a small g.install prototype
(that would also include the management of binary
and script installations)
as soon as I have my belief modelling stuff ready.
Let's see hw smooth it goes ...

Cheers,

Benjamin

On Thu, 16 Dec 2004 09:13:45 -0700
Michael Barton <michael.barton@asu.edu> wrote:

I want to echo the positive other replies to this proposal. This is very
much along the lines of what Paolo, I, and others have suggested. It would
be good if the 'g.install' routine could also insert shell scripts into the
extension menu hierarchy.

Another longer-term idea (i.e., for the 6.x series) I tossed into the fray a
few months back is to make modules self-organizing in the GUI unless
overridden by a users. We already have a start in how that could be achieved
by the letter-dot prefix of GRASS modules. r.* modules are raster, and so
on. We could build on that so that each module and script written would have
a short header string that identified where it belonged in the menu
hierarchy. Then, when someone installed a module, it would go to a logical
local rather than into a long extension menu.

The ultimate objective is to make a GUI that is more or less
self-generating. If I remember correctly, some time back, Radim was working
on a QT version of the GUI that would use XML to manage the menu hierarchy
with this kind of goal in mind. In such a case, a self-generating menu
system (or alternative better GUI) would be considerably more feasible.

Michael

On 12/15/04 3:49 AM, "Benjamin Ducke" <benducke@compuserve.de> wrote:

> My ideas are:
>
> For each extension:
> - put all source code and binaries for selected systems into
> a tarball
> - also add some standardised ASCII description files to the
> tarball:
> - version information
> - binary and dependencies information
> - source URL and author info
> - Tcl/Tk menu code for the GIS manager
>
> For GRASS 5.7:
> - the g.install module could be used to
> list, install, update and remove extension
> binaries and documentation
>
> - installation could be system-wide into any GRASS bin tree location
> the user has write access to or user-only into a directory
> into a dir under the user's current location or mapset.
>
> - a new menu entry 'Extensions' in the GIS manager where extensions
> can register their modules in a submenu and hooks for g.install
> to query, update or remove these extensions are provided
>

______________________________
Michael Barton, Professor of Anthropology
School of Human, Evolution and Social Change
Arizona State University
Tempe, AZ 85287-2402
USA

voice: 480-965-6262; fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

9

On Thu, 16 Dec 2004, Benjamin Ducke wrote:

OK, it seems everybody agrees that it is a good idea.
I will try and write a small g.install prototype
(that would also include the management of binary
and script installations)

I think rather than having pre-compiled binaries for lots of different platforms sitting around somewhere, g.install should just download the source code from somewhere and compile it on the target machine. With the shared libraries now included with GRASS, I believe the only extra thing needed for this to be possible is to include all the GRASS header files in the binary distributions (so people can compile extra modules without having to download the whole GRASS source distribution).

Paul

10

> why bother with g.install and just put it all in GRASS CVS?

Because, as Benjamin and others (especially Paolo Cavallini) have
suggested, a development model with add-in modules is vastly more
robust that one in which everything is in the same tarball. Central
servicies need to be in the core distribution, but everything else
should be modularised. They can be stored on a central CVS repository,
but do not need to be in the same project, just to follow the same
installation rules.

What I would like to avoid is a situation like r.cva -- a very useful
module which never becomes an official part of GRASS and is pretty much
lost to us once the author moves on to other projects.

I can see an advantage in the R-style modularity for places which want
to add their own proprietary modules for in house use. But that doesn't
help the rest of us any :wink: .. this could easily be done for GRASS
scripts, and GRASS has enough low level modules that those can be very
useful. As far as multiplatform binary distribution though, Debian
provides the only working model I know of.. rather complex solution..

best,
Hamish

11

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Agreed fully. This is the kind of mechanism R uses on GNU/Linux, AFAIK.
BTW: could this be a way of simplifying the installation of the (much needed)
v.in.dwg (once one has got access to the non-free libraries)?
All the best.
pc

At 21:56, giovedì 16 dicembre 2004, Paul Kelly has probably written:

I think rather than having pre-compiled binaries for lots of different
platforms sitting around somewhere, g.install should just download the
source code from somewhere and compile it on the target machine. With the
shared libraries now included with GRASS, I believe the only extra thing
needed for this to be possible is to include all the GRASS header files in
the binary distributions (so people can compile extra modules without
having to download the whole GRASS source distribution).

Paul

- --
Paolo Cavallini
cavallini@faunalia.it www.faunalia.it www.faunalia.com
Piazza Garibaldi 5 - 56025 Pontedera (PI), Italy Tel: (+39)348-3801953
http://pkg-grass.alioth.debian.org/cgi-bin/wiki.pl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBwple/NedwLUzIr4RAq7tAJ9I5raF4RbdCisUO0YqU062yuvrSwCgoa0h
QjwzTZX/nHEXCJpO5kIOCjw=
=VyMr
-----END PGP SIGNATURE-----

12

Paul Kelly wrote:

On Thu, 16 Dec 2004, Benjamin Ducke wrote:

OK, it seems everybody agrees that it is a good idea.
I will try and write a small g.install prototype
(that would also include the management of binary
and script installations)

I think rather than having pre-compiled binaries for lots of different platforms sitting around somewhere, g.install should just download the source code from somewhere and compile it on the target machine. With the shared libraries now included with GRASS, I believe the only extra thing needed for this to be possible is to include all the GRASS header files in the binary distributions (so people can compile extra modules without having to download the whole GRASS source distribution).

I agree, binary distribution doesn't make sense.

We have to add include/Make to binary distribution. GRASS_HOME in Platform.make must be changed when binary is installed.

Radim